Dooley grew up in Mobile, Alabama and attended the McGill Institute, administered by the Brothers of the Sacred Heart. Dooley competed on behalf of McGill's athletic teams, known as the Yellow Jackets, and for a few years considered basketball to be his best sport. Dooley is a graduate of Auburn University (bachelor's degree 1954, Master's in history 1963) where he played college football and later coached under Ralph "Shug" Jordan. Dooley was a member of the Phi Kappa Theta fraternity as an undergraduate at Auburn.