Born in Wichita Falls, Texas in 1940, Holder attended school in his home town and moved to Goodwell, Oklahoma in 1958 to enter Panhandle A&M College. A three-sport performer, Holder played golf and football for the Aggies, while setting track and field records in the long jump and 100 yard dash. After his freshman year, Holder returned to Wichita Falls to attend Midwestern State University, then re-entered Panhandle State in 1961 to complete his college athletic career.