Oliver attended the University of Alabama, where he played for Bear Bryant from 1959 to 1961 as a defensive back. After coaching for high schools in Alabama, Oliver joined Auburn as a secondary coach in 1966 under coach Ralph Jordan. Oliver remained at Auburn until 1970, when he left for the same position at Alabama. In 1980, Oliver became head coach at the University of Tennessee at Chattanooga, a position he held until 1983. Oliver was the defensive coordinator for the Memphis Showboats of the United States Football League in 1984 and 1985, and then coached the secondary at Clemson University. In 1990, Gene Stallings hired Oliver as a secondary coach at Alabama and promoted him to defensive coordinator in 1993. Alabama's athletic director, Hootie Ingram promised that Oliver would succeed