Oliver was the son of Major General Oliver De Lancey (1718–1785) and Phila Franks, the oldest daughter of an early New York Jewish family. He was the nephew of James De Lancey, a celebrated New York lawyer, who was chief-justice of that colony from 1733 to 1760, and lieutenant-governor from 1753 to 1760. These two brothers were the sons of a wealthy Huguenot of Caen in Normandy, who emigrated to America on the revocation of the edict of Nantes, and bought large estates in the colony of New York, where they ranked among the wealthiest and most powerful citizens.