The castle had seven towers connected by a high wall. The three largest towers, measuring , protected the most vulnerable southwestern flank. A wide moat separated the structure from the town. The castle was attacked in 1382 and 1383 (during the civil war of 1381–1384) and in 1390 (during the civil war of 1389–1392). After the 1422 Treaty of Melno, the castle lost its significance as a defensive structure. It is known that Grand Dukes used it as a residence. Sigismund Kęstutaitis was murdered in the castle on 20 March 1440.