Towns are the #1 priority for building (gildar income), but I find having 1-2 fortress can make your military better by being the place where your units are trained while the other towns provide the money/research. This work best when:
1. Your fortress has very high production.
2. The fortress has 2-3 essence. This is for enchanted hammers, and Heart of Fire. Heart of Fire adds 1 fire attack per essence. While HoF is expensive at 100 mana, if you produce lots of units from that city, 2-3 extra fire damage per unit can really start to add up.
Archers, in particular, can really put that extra 3 fire damage to good use.
3. You have armorer, barracks, forge, training yard, etc. Combine high productivity with these bonuses means you're churning out solid troops quickly, at reduced resource cost.
4. You want to build a lot of henchmen. Building them from a fortress starts them ahead a level, and gives them all those wonderful attack/defense bonuses as well.
(The more towns you have behind the fortresses, the more you can pay to rush production out of your fortresses, boosting their productivity further. )
I would guess the optimum ratio is something like 1 fortress for every 5 towns?