The building was formerly known as Sydney Hotel, and was built between 1795 and 1796, by Charles Harcout Masters as a social gathering place. The building was closed in 1913, and had become a museum of art for some time, until its purchase by the Government of Wessex and Cornwall to house of the royal family. The house is one of four properties owned by the Wessex and Cornwall Crown, the other three being Clifton House in Bristol, Dunster Castle, and Highgrove House in Gloucestershire, the official residence of the Duke of Cornwall.