Anthony W. White was the son of Anthony White. His great grandfather was Anthony White I, a royalist, who, after the execution of Charles I, emigrated to Bermuda and became connected with the government of the islands, of which his son, Anthony White II, and grandson, Leonard White, were chief justices. Anthony White III moved to the United States from Bermuda and married Elizabeth Morris, the daughter of Governor Lewis Morris of New Jersey.