Peter Gansevoort was born into the Dutch aristocracy of Albany, New York. His father Harman (1712–1801) represented the third generation in America, while his mother, Magdalena Douw Gansevoort (1718–1796) was connected with the Van Rensselaer family {Her mother was a granddaughter of Jeremias Van Rensselaer}. His family had been in Albany since 1660, when it was the Dutch colony of Fort Orange, and Harman owned a brewery and farms. His younger brother Leonard was more active politically, serving in the state assembly and senate, as well as the Continental Congress. His son Herman Gansevoort (1779–1862) built the Gansevoort Mansion in 1813 on his father's tract at Gansevoort in Saratoga County, New York. The house was added to the National Register of Historic Places in 1976.