The castle is known as the site where the plot was hatched to assassinate Thomas Becket (1118–1170). More recently, it became the home of the art historian Lord Clark of Saltwood (1903–1983), then his son Alan Clark (1928–1999), a minister in Margaret Thatcher's government.