Specifically regarding tourism: I suspect that it is not worth building tourism improvements. Last I checked, it appeared as though base tourism income was proportional to the average population on all colonies in the galaxy, and that the constant used was such that the base tourism income was always less than the base standard income that the colony would generate at relatively balanced slider settings. On top of that, most factions can only build one tourism improvement per colony (not counting wonders) and the tourism improvements generally don't receive adjacency bonuses from anything else you can build. Additionally, bonuses to the same output stack additively, which means that a Port of Call or other tourism improvement is no better on a world with a +X% tourism bonus than on a world with no bonus at all to tourism. The effect of this is that if you wanted to generate income, you're typically better off building another market or farm than a tourism improvement.
Regarding the general question: I would say that in general the tile and planetary bonuses are fairly negligible overall, especially if you're making heavy use of starbases or have artifacts turned on. Build what's best for your empire; if that happens to line up with what the planet and its tiles have bonuses for, great, but if not it's only a minor loss of efficiency overall, especially if you're stacking up starbase bonuses, have a few artifacts being studied, or pick up the techs which provide empire-wide output modifiers, or get anomalies/events which give empire-wide output bonuses.
One final comment: It isn't always the case that you should only build farms, but the larger the bonuses to planetary outputs you can get from things other than planetary improvements are (i.e. from economic starbases, precursor artifacts being studied by starbase labs, anomalies, events, technologies, and the occasional wonder), the more likely it is that you're better off just building farms and other improvements which directly modify the planet's base production than building improvements which modify specific outputs, especially if you're playing with coercion at the default levels and so have an enormous efficiency penalty for allocating spending at something other than the default 33/33/33 manufacturing/research/income split.
