The units were probably standing on top of the city but not in it.
This can happen when you use your garrisoned army to attack something, they will end their turn "ontop of the city" -- which I guess is suppose to be mean they left the city gates. I put this problem up as a bug myself, but had it pointed out the mechanic could be purposeful.