Charles L. Scott was born January 23, 1827 in Richmond, Virginia. His father was Robert G. Scott, well known attorney and politician of Richmond, who was born in McIntosh Co. GA and died in Alabama. His mother was Susan Randolph Madison, daughter of Rt.Rev. Bishop James Madison of VA. His grandfather was Col. William Scott, born 1752 in VA and lived in Camden Co. GA. Charles Lewis Scott attended the public schools and Richmond Academy. In 1846 he graduated from the College of William and Mary, Williamsburg, Virginia. He then studied law and was admitted to the bar in 1847, practicing in Richmond. During the 1849 California Gold Rush he went to California and mined gold. In 1851, he resumed practicing law in Sonora.