Originally a medieval castle, Chazeron was altered in the 17th century by the architect François Mansart. He built a staircase on the site of the keep and added an arcaded gallery. The moat was filled in, three of the external walls were demolished and two wings were added. During the Second World War, Léon Blum, Georges Mandel, Édouard Daladier, Paul Reynaud and Maurice Gamelin were imprisoned in the castle in 1942 before their appearance at the Riom Trial.