The first significant building to occupy the site of the castle was a timber-framed dwelling, inhabited from the early 16th century by Richard Streatfeild, an iron master and wool merchant. Little remains of this first structure as, in 1679, Henry Streatfeild (1639-1719) had the house rebuilt in red brick in the Restoration style. The building was known as High Street House or High Street Mansion since it fronted the village high street. Remodelling of the house’s grounds in the 19th century resulted in the current diversion of the road through the village.