Rice grew up in Hastings-on-Hudson, New York and attended Hastings High School, graduating in 1957. He went on to play football at Ithaca College from 1957 to 1960, and then earned a master's degree from New York University. Currently, he is married to his wife, Jeanne Taylor, who is the former assistant athletic director at the University of Mississippi. They reside in Marco Island, Florida.