Whitehouse was born November 5, 1921 in Paris, France, the son of American parents Mary Crocker (née Alexander) and Edwin Sheldon Whitehouse (1883–1965). His father was a Foreign Service officer, and served as U.S. Minister to Guatemala, 1930–33, and to Colombia, 1933-34. Charles Whitehouse was a great-grandson of railroad executive Charles Crocker, and a grandson of Charles Beatty Alexander and Harriet Crocker. He was also a great-grandson of Henry John Whitehouse, Episcopal bishop of Illinois. He was raised in Europe and South America.