Born in Paris, France, Colston was the adopted son of Maria Theresa, 2nd Duchess of Valmy (c. 1775-1845) and Dr. Raleigh Edward Colston (1796–1881). His mother had divorced her husband François Étienne de Kellermann, a famous cavalry general under Napoleon Bonaparte. Dr. Colston's marriage to a Catholic and a divorcee scandalized the Colston family of Virginia. In 1842, young Colston was sent to study in the United States, living with an uncle in Berkeley County, Virginia, now West Virginia. His "rigidly pious" uncle repeatedly tried to get Colston to enter the Presbyterian ministry, but the young man preferred a military career.