Contee was the son of Col. Thomas Contee (1729–1811) and Sarah Fendall (1732–1793). He was born at "Brookefield", near Nottingham, Prince Georges County, Maryland, the home of his father, and original home of his ancestor, Maj. Thomas Brooke, Sr., Esq. (1632–1676).