Archers are pretty awful overall, imo. Look at the tech tree and compare any bow to melee weapons at the same tech level. Bows have:
> Less damage than any melee weapon
> Less initiative than any melee weapon
> No passive or active ability (unless you count simply having range as their passive)
> Two handed, so no shield either
> No swarm bonus
Crossbows have a (weak) special ability and allow a shield, but their damage and initiative is even worse than bows. Now, staves are amazing and arguably overpowered, but that's another discussion, I'm assuming that one could not mistake bows for staves.
The only thing they have going for them is range, and that's not too much of an advantage given how small battlefields are - all it gets them is an extra couple attacks as your troops charge to melee range (and likely only one attack if you have cavalry). That could be devastating against low level or unarmored troops, but against decent troops it just won't make enough difference; as long as you make it to melee range in decent shape you'll win, you have more damage and more initiative no matter what melee weapon you choose, likely more armor and dodge from a shield (or armor piercing from a spear), and you get attack and accuracy bonus from swarming. In the battle OP describes, it sounds like the enemy's troops simply outclassed you, if their archers had been mace+shield instead you would've lost even worse.
.. Setting aside the relative merit of archers, however, the OP's afterthought is very true - there's rather little specialization among weapons. I've never thought "Aha they're using swords, I'm sure glad I brought my spearmen to counter them" - even though spears are a little better against swords than other weapons, it's not that big a difference, spears are just generally good weapons. Same for mace, the special ability might be slightly better against low armor enemies that you can nearly wipe out with that first hit, but then again the higher damage penetrates heavy armor better as well, there's no situation where you don't want it - just all around good.
Swords and bows are the closest thing we have to specialized weapons at the moment, but unfortunately they're not great weapons overall, so in this case "specialized" means "only decent in certain situations, otherwise weak" - they can be good, but the spotlight is stolen by weapons that are pretty much always good.