Thomas, Jr. was the grandson of Thomas "Scotch Tom" Nelson, an immigrant from Cumberland, England who was an early pioneer at Yorktown. His parents were Elizabeth (Burwell) and William Nelson, who was also a leader of the colony, and briefly served as governor. Thomas was born at Yorktown, and like many Virginians of his time, was educated in England. He attended Newcome's School for six years and was at Eton in 1754, before entering Christ's College at Cambridge University in 1758. He graduated in 1760 and returned home the following year. He helped his father with his farm at this time.