Martin was born in Caroline County, Virginia, and later lived at Albemarle County and then at Henry County, Virginia, at his plantation, "Belmont," on Leatherwood Creek in Martinsville, not far from the plantation of his friend Governor Patrick Henry, "Leatherwood."