Born and raised in Guin, Alabama, Riley was a four-sport athlete in high school and played football, basketball, and baseball at Howard College (now Samford University). He entered the U.S. Navy in 1942 and was stationed at NAS Pensacola in the nearby Florida panhandle as a physical trainer. Following his discharge from the military in 1946, he returned to college at the University of Alabama in Tuscaloosa, where he lettered in basketball and baseball. He earned a bachelor's degree in physical education in August 1948 and earned a master's degree in 1953.