The brigands actually do something? I've seen some brigands loitering around towns, but they won't actually attack. Actually, nothing attacks my towns. Nor units, really
i got attacked many times even if i had a few time s when they just stood still
i think the ai is made so they attack only when towns are with very few defenses (they attacked me only when i had fewscouts/peasants)
maybe they should be more courageos, they are pretty strong after all