Originally known as Philorth Castle it was built in the early 14th century, destroyed in winter of 1308-1309 in the Wars of Independence, and re-built by the Fraser family in 1380. Subsequently, a courtyard and outbuildings were added to the main tower. It is now open to the public by appointment only. Flora Fraser, 21st Lady Saltoun and her husband Captain Alexander Ramsay of Mar, who was one of Queen Victoria's great grandchildren, lived there for a while but nowadays their daughter Katharine Fraser, Mistress of Saltoun, lives there with her family.