A fortress was first built on the site of the present castle in 1230, intended to defend the city of Ghent and to fortify the river Leie. This fortress was owned by Nikolaas van Hoendunc, lord of Nevele. After it was destroyed by Louis de Male, count of Flanders, Jean de Fosseux rebuilt and enlarged the fortress from 1381 onwards, adding moats, so that the castle could only be reached by drawbridge, and strengthening the defences with four towers.