The castle was built as a manor house by Sir John de Broughton in 1300 at a location where the confluence of three streams created a natural site for a moated manor. The house was sold in 1377 to William of Wykeham, Bishop of Winchester, and has remained in the same family since that time. The original house was crenellated by Sir Thomas Wykeham in 1406. In 1451 it passed by inheritance to the Fiennes family, Barons Saye and Sele. Beginning in 1550 Richard Fiennes transformed the medieval manor into a house in the Tudor style, significantly expanding the building.