Dooley was born in 1934, in Mobile, Alabama. There, he attended the McGill Institute, administered by the Brothers of the Sacred Heart. Dooley then attended Perkinston Junior College in Perkinston, Mississippi from 1952 to 1953. In 1953, he moved on to Mississippi State University and graduated in 1956, where he was an all-SEC lineman for the Maroons/Bulldogs. Dooley's brother is former University of Georgia head football coach Vince Dooley. His nephew, Derek Dooley is the former head football coach at the University of Tennessee. Today, Dooley lives in Wilmington, North Carolina. He is married to Marie Dooley, and has four sons: Jim, Billy, Sean, and Ashton.