My thought:

Is it possible for roads to grow one sqare at a time, built by a defenceless caravan unit? You could still grow new roads but not instantly, nor through hostile territory unless you protect the builder.

It is an issue, to some people, myself included. I have used it to easily win wars and supercharge my expansion but I felt dirty doing it, it feels close to an exploit, not merely to grab shards miles and miles away but instamagic all the infrastructure between them too. One game I had a giant road running along the hilly and swampy edge of a massive mountain range along the coast, in one cast I turned a 20+ turn trip dividing the entire known world into a 2 turn shortcut across the map (on horseback with tireless march +1 moves per champ).

I wonder if the problim is roads/movement in general. So, slower roads? Slower movement? One idea for balance, addressing the key bits:

1 tile base growth per turn of road connections,
Road 2x speed outside of ZOC,
Road 4x speed within ZOC,
No stacking of tireless march,
5 turn cooldown on Cloudwalk.
Horse/warg nerf*

*Good weapons have an initiative penalty, horses need something negative. (Lower defence/no fortification bonus) Either way they are part of the collective easy movement problem and worth adding here. But probably not in as much depth as I want to get into...so I know it's unrelated but their prone avoidance bugs me, horses aren't invulnerable to magic and knocking over a knight on horseback should be possible, plus it takes far longer to get back upright and in the saddle than to stand up again... especially if you're a ninja in light robes being knocked over or a strong heavy giant who weighs as much as a horse anyway. There... rant over. Also, horse armour could negate the defence penalty but weigh the same as the added weight capacity (or most of it), ditto for other mounts... except those that are naturally tough skinned.