The castle was built around 1550 by Sir Alexander Gordon of Midmar. During the first Jacobite rising in 1689, the castle was garrisoned by Spanish troops under the command of General Hugh Mackay. The castle has been owned by the Gordon family since 1482. From 1848 to 1970, it was leased to the British royal family, being relatively close to Balmoral. Jamie Douglas Home's book 'Stately passions' says that whilst it was originally leased to the Royal family it was bought by them in 1878 for £100,000.