A native of Cookeville, Tennessee, Brown was one of the top-rated quarterbacks in the nation coming out of high school. He was also recruited to play basketball and was drafted by the Pittsburgh Pirates as a shortstop. He chose to stay in state and was a quarterback at Vanderbilt University from 1969–1972. He started all four years at Vandy and led the Commodores to their best seasons in terms of wins since 1960. One of his victories was a 14–10 upset over the #13 Alabama Crimson Tide in 1969. It was Vanderbilt’s first victory over Alabama in 13 seasons.