@Wylaryzel - Yeah, we were at war. But even when you're at war with someone, I don't think you should be able to set up new cities in their area of influence. I expected a 'raze city' option somewhere in the city details screen, but I couldn't find one.
@hermes33 - I know I'll still get the resources, but I won't get the extra bonuses from the fancy buildings in my big city if the resources are closest to this new outpost will I? Cheers for the info about razing needing a technology. I assume that's a warfare tech. I sure hope you can raze your own cities, or I'll have to do something odd like wait for the outpost to be captured before recapturing it again and razing it this time.