A native of Columbus, Mississippi, Brewer first came to Ole Miss as a player for the legendary Rebel teams of Johnny Vaught. He played quarterback, defensive back, punted and held for placekickers from 1957 to 1960, and Ole Miss named him to its "Team of the Century" in 1993.