This site was originally an Iron Age, 600 BC to 50 AD, hill fort. A fortified Norman castle with stone walls was built on the site possibly by William Fitz Osbern or his son Roger de Breteuil, Earl of Hereford in the period 1070 to 1075. The castle was eclipsed with the building of Brecon castle before April 1093. The fortress seems to have been constructed in stone from the first with a hall-keep surrounded with curtain walls and square towers. Historically the castle remained a part of Brecon or Brycheiniog barony until 1207 when King John of England granted it to Peter fitz Herbert. It then became caput of what was to become the Talgarth or Blaenllyfni barony.