Spencer was born in Payson, Utah, but spent most of his young life growing up in Homer, Alaska, living with his father. The home in which he grew up did not have indoor plumbing, but a simple outhouse instead. He and his father sometimes earned a living by singing in bars and taverns. It was from these experiences he learned to yodel, a quality demonstrated in many of his songs. His father was a Mormon, but they stopped attending the church shortly before he turned eight.