It's the way it was designed. The only real reason to take a neutral city is to access the ogre camp nearby.
or for the kingdom factions its the dragon like humanoids (think they are called drath).. if they have ogre camp and you kingdoms you cant use them so raze city and build own if you want more then a lvl 1 city (which i am sure most do
)