The early motte and bailey castle was built on a cliff overlooking a ford on the River Wye in 1070 by William Fitzpond. The castle was intended to provide protection for a planned Norman settlement above the River Wye of quite a substantial size, with c. 200 plots being intended, running uphill for half a mile towards Llanfair, where the church was sited on the hilltop. St Mary's church remains today, much altered in the Victorian era. It has a wooden 13th century effigy of what appears to be a clerical figure. It has been said to resemble another effigy, of Bishop Peter of Aigueblanche, in Hereford Cathedral.