Phipps-Hornby was born in Lordington, Hampshire on 31 December 1857, the son of the Admiral of the Fleet Sir Geoffrey Thomas Phipps Hornby GCB.