Harbaugh was born in Crestline, Crawford County, Ohio, the son of Marie Evelyn (née Fisher) and William Avon Harbaugh. His ancestry includes German and Irish. Harbaugh played college football for the Bowling Green State University Falcons from 1957–1960, where he was a three-time letterman. In his junior year, the Falcons finished the season 9–0–0 and were named the small college division national champions. Harbaugh played professionally for one season, 1961, in the American Football League for the Titans of New York.