Walker was born in Mocksville, North Carolina, and attended college at the University of Oklahoma. He played for the Oklahoma Sooners football team as a tackle from 1946 to 1949, and he served as the team captain for two seasons. Walker was part of the 1946 recruiting class—brought to Norman by one-year head coach Jim Tatum—that included nine future All-Americans. In 1947, Tatum's former assistant Bud Wilkinson had taken over as head coach but by midseason the team was struggling with a 2–2 record. In an effort to evoke player leadership, he told the team to elect two co-captains to serve the remainder of the season, and they chose Walker and the previous season's honor captain, Jim Tyree. In 1948, the International News Service named Walker as the right tackle on its All-Midlands team.