McCarver was born in Memphis, Tennessee. He began his playing career after being signed by the St. Louis Cardinals from Christian Brothers High School in Memphis in 1959. He hit .359 that year while splitting time between the Cardinals' minor league teams in Keokuk and Rochester and, though just 17 years old, was briefly called up to the Cardinals. He spent the 1960, 1961, and 1962 seasons shuttling between St. Louis and the minor leagues in places like Memphis, Charleston, West Virginia and Atlanta. In 1963, he was called up to the majors for good.