William was born in Bagshot, the eldest son of Joseph Fairfax of the Horse Guards and Mary Anne. He married firstly Hannah Spears (d.1770) in 1767, and secondly in 1772, Janet Margaret Charters. William was the father of eight children. A daughter, Mary Somerville, became a noted science writer and polymath.