He was born in Ugbrooke, the son of Hugh Clifford of Chudleigh, Devon, and his wife Mary, daughter of Sir George Chudleigh, Bt. He was baptised 4 August 1630 at Ugbrooke, and matriculated at Exeter College (Oxford), receiving his B.A. there in 1650.