The answer is it depends on the hero.
For heroes that state you gain x/turn, where x is gilder, tech knowledge, ect, they do NOT need to be in a city.
For heroes that add a % improvement (+20% boost to farming) they DO need to be in a city. They will only add that production to the city they are stationed in and the bonuses DO stack.
My only queston is for heroes that have a bonus % to production. I am not sure what "production" refers to. Is that everything? Food? Metal? Research?... Not sure, anyone know?