It's pretty much how Heavenfall said, though I can list a couple exceptions.
City-String Exception:
In my experience, if you're playing a Kingdom, you only ever get one Caravan. Because of this, if you ever end up starting on a peninsula, or a canyon, or anything at all causes you to string out cities, it can be a good idea to caravan string from City 2 to 3, 3 to 4, etc etc. Conceivably, it's always better to send Caravans 2 through X to City 1, but the problem arises where you lose efficiency in creating roads, and it can begin to take tens of turns to get a caravan back, as I've experienced before. This is especially apparent on the larger map sizes, where, if you end up on one of these terrain-formations that causes you to string cities and your expansion goes unhindered, it's not uncommon to string together 5-15 cities before you hit the mainland. This is especially apparent in the case of forests, where all units move incredibly slow, requiring twice as much movement per tile to get around, and if you find yourself in one of these situations, you are almost certainly going to have to deal with that. After the road(s) is(are) created, if you find yourself low on food, then it becomes prudent to dismiss the caravans and send them back to City 1, which will be much quicker with the roads already established from Cities X, Y, and Z, to City 1.
Mega-Farm Exception:
Drachonus alluded to this a little bit. In most games, you'll find that your Capital is going to be your main source of food production, but some times you end up a finding a city with the potential for both a Farm and Apiary, or something like that. When this happens, this is usually when I end up actually using the Home City Caravan, which I normally never build, and sending it to that city, as well as re-routing all my other caravans if I can be bothered. If you're playing a Kingdom, the addendum to this exception is, "Unless you haven't researched Lost Bounty." The Lost Bounty technology creates new food resources within your territory, making it very profitable to wait until you have a large empire to use it. More often than not, the tech will spawn a Twilight Bees resource near your capital, making any other food-producing city still moot as far as caravans go, unless it also happens to get a new food resource at the 'Mega-Farm' city, though you don't see it very often.
The Empire Exception:
And now the biggest exception of all, the Empire's. With only a few points in their Diplomacy tech tree, you can have two caravans per city. The second Caravan makes the City-String Exception moot, as you can just use the first of the two to get a road, then the second one to send back to City 1. If you can micro it well, it can also make the Mega-Farm Exception moot, especially if you haven't used both caravans already, since you can send a Caravan both to your Home City and to the Mega-Farm city. If you have used the extra caravan, finding the one traveling between City X and City Y can be a little tricky, but if you can find it, you can just dismiss it and then send the extra caravan to the Mega-Farm city. It's also worth keeping in mind that the Mega-Farm Exceptions Addendum, as stated above, doesn't apply to Empire's since they don't have a Food-Resource-Creating tech like the Kingdoms do, so go ahead and shoot those caravans straight to both Farming Cities.