Born in 1942, Swanson grew up in Denver, Colorado. He met his future wife Sandee there as a teenager. Swanson attended Colorado State University where he was a member of the Army's Reserve Officers' Training Corps. After graduating in 1965 and becoming a commissioned officer in the Army, he continued to correspond with Sandee, who was serving with the Peace Corps in the Philippines. The two were married in 1967 in Honolulu, Hawaii, while Swanson was on leave from his first tour in Vietnam. They settled in Boulder, Colorado, and had two daughters, Brigid and Holly.