In a recent game a city of mine was able to build an Earth crystal-harnessing building that was four squares from the town center on the world map, but once the structure was completed, I did not have access to the Earth element, according to my spell caster and the spell purchasing page.
Building a new town adjacent to the building immediately fixed this, although the building remained a part of the previous town. Surely this isn't the intended order of things.
edit: reposted to correct forum