Born at Westminster, London, Dartmouth was the eldest son of William Legge, 5th Earl of Dartmouth, and Lady Augusta, daughter of Heneage Finch, 5th Earl of Aylesford. The Hon. Sir Henry Legge was his younger brother. He was educated at Eton and Christ Church, Oxford. He played first class cricket for Marylebone Cricket Club in 1877.