The Old House dates back to as early as 1802, based on a painting in the entrance hall. It remained the primary residence of its owners until Jack Baker oversaw construction of the Main House as a new home for his family. The old house was badly damaged in October 2014 following a hurricane, and repairs were still not completed in 2017 due to the Bakers' loss of control to Eveline.