Given that your cities immediately convert the land into lush greenery or a Mordor hellscape, AND the fact the your factions are useless without natural resources like Lost Libraries Fertile Land, etc, I think that the idea of terrain actually having an effect like that is flawed. Swamps are no issue.
Actually, please someone slap me if i'm wrong, the terrain dosent' actually change. The adjective just changes from "barren" to "fallen" in Empire's case. So the terrain type remains the same, it just looks different.
What I'm looking for is a way to make terrain a little more important. Like Alixenius pointed out, it kinda makes no sense that you can as easily build a building on swamps, then on grasslands. Maybe increasing the cost of a building depending on what tile it is built on.
For example:
(Take note that these value are purely arbitrary.)
Construction Cost penalty per terrain:
Grassland: 0%
Swamp & Hills: 50%
Desert & Artic: 75%
So if you have to build a building that costs 10 Material and 10 Gold, it would end up costing you 15 Materials & 15 Gold if you built it on a Swamp tile for example. I think this would be a good way to reflect the ramifications of building in odd terrain.
I'll go post this in the suggestion forum 
Regards,
V.