Davidson was born in Los Angeles, California, the son of Avis (née Wheat) and Benjamin Earl Franklin, Senior. He attended Woodrow Wilson High School in the El Sereno neighborhood of Los Angeles, but did not play football in high school; because of his height (6 feet 8 inches), basketball and track were more to his liking. While attending East Los Angeles Community College, he was spotted by the football coach and asked to join the team. He was subsequently recruited to play at the University of Washington where he flourished as a member of two Rose Bowl winning teams and gained entry into professional football.