I found out that world difficulty on anything above challenging makes the monsters ignore the AI players apparently
I don't think this is accurate.
There are AI bugs that make the monsters less likely to attack AI units and cities in general. This is based on Brad's feedback. The forum link you left has someone say they think it's based on difficulty level, and that 'Brad said something like this.' Pretty sure he said the monsters don't attack AI units reliably because of simultaneous turns being taken by AI players and monsters.
Brad paraphrase from memory: "They are literally chasing each other."
Monsters attacking cities is a bug where monsters will approach cities fully intending to attack, but a bug comes up when a monster wants to attack a city that is ADJACENT, leading the monster to approach for attack but then usually fail to execute the attack action. This is again based on what Brad has said on these forums.
Could be more going on, but I don't believe difficulty makes a difference other than increasing monster levels and making them more 'confident' in their attack odds.
So in other words, monsters are generally less likely to attack the AI, but will. Enemy AI units do get killed, I've seen it, and enemy AI cities can get razed (seen it many times). If you have the monsters way tougher than the AI, you will likely notice the AI suddenly having much more monster trouble. But not as much as you will.