Nothing bad will really happen (u will actually have a larger gold income without the upkeep of the troops) but if someone attacks you, then they will defeat you without question as they will outnumber you without a doubt. They will then pillage as much resources as they can out of your town ( each cargo ship they send will allow them to pillage another 500 resources)