The kingdom only had a population of less than 22 people at the time of the Goblin kidnappings (Graham comments there are more mattresses than there are people in Daventry, and there are only about 22 mattresses). Assuming that the trolls, the two goats, and Triumph don't count as humans, they are still considered by Graham to be citizens, as well as counted among the population that joined later. Not much is known about Tree Dorman but we can assume he was probably there behind the scenes since Chapter 1. The town fluctuates between 17-23 over the various chapters.