Buller was married to Brunanter sex icon Caroline Koch. They had a very stormy relationship, with both having many affairs, but this did not affect their marriage. They had one daughter, Maxine in 1961. Buller died in 1978 in a car accident.