Unfortunately that level of difficulty is something that only really appeals to *maybe* 5% of the market.  I've seen games (mostly old games) where you aren't guaranteed to win on normal and many people have a fit over that.  The old "I play games to win, if I can't win... (insert more crying and whining)."

I agree there should be the top difficulty that will stomp all over you for those who want it.  But easy should not be a 50/50 chance because some people just want to win every time no matter how badly they suck/screw up/don't try.

So long as the AI is beatable (i.e there are strategies that can be used to beat the AI on the lower difficulties) then personally I'm happy.  I'm not asking to wins to be handed to me, but as a casual player of these games, I am looking for a challenge without being overwhelmed.  The original Gal Civ was great in that regard.  The AI put up a fight, but it still could be beaten if you were prepared to put in the time to learn the game and the strategies involved to play the game.  Winning every time doesn't equate to fun, winning after putting in some effort to defeat the AI is a lot of fun IMO.

That will require either a significant redesign of several core game mechanics to make taking advantage of them more friendly towards a brute-force approach to problem solving, at which computers excel when compared to humans, or ramping up the AI cheats to such a level that it is unlikely that the top 1% of players will consider it fun playing against.

I know I have written disparagingly about the current state of the Elemental AI previously because, dammit, it is utterly incapable of putting up a decent challenge for any accomplished strategy gamer unless he intentionally crimps his playing style, but to be fair a lot of the AIs problems are caused not because of a lack of long term planning but by a design that has too many options available that are damn hard to predict the value of, unlike say the design in GalCiv in which the vast majority of game elements seemed, at least to me, to be be designed with the clear goal of making them easy for an AI to deal with [which is not the same as saying that writing an AI that plays well is easy given such a design, for it isn't, but the problems to be solved are considerably less when you have a good idea of the value of different choices and the more precise evaluations you can make the more efficient a brute-force approach will tend to be with regards to providing a challenge].

That's definitely cool stuff but what happens if modders create new game features?
Obviously the AI has to be updated to use their new mechanics but I don't think every modder automatically has the skills to work with a multi-threaded AI without crashing the game.
As it happens, that includes me. =P   I have written quite a few realtime scripts that work asynchronously but that's not quite the same as using multiple low level threads.

Hier Hier, I agree 100%.

The #1 factor in game design has to be a challenging AI for me. I could care less about graphics. I don't normally play a game once I can beat the AI on the hardest setting, there's just no challenge anymore. Once there's no challenge, there's no longer any enjoyment for me. I've been playing this game even though I can consistently beat the AI mainly because they've been patching the game very quickly (stardock has an incredible support team, much more than most other PC game companies in my opinion) increasing AI capability a small bit each time, and because it takes an incredibly long time to finish a game on large map, at least for me. I actually haven't been able to finish a game as fast as they've added patches. That's ok, I like long games just as much as I like short ones.

A top level difficulty which is unbeatable would keep players like me interested in the game indefinately.

I'm not a "cheater" or "hacker," I'm just looking for a personally challenging game to play: challenging = fun, non-challenging = boring. 

First of all I would remove the word "ridiculous" from the list of options: it is confusing (for a long time I thought it was the easiest setting, since being the last setting available in a rotating list it also is the one that comes BEFORE the easiest setting).

Besides, unfortunately, it is indeed ridiculous at the moment and not in the way it is intended!

Everyone I heard making an argument about chess being so "hard" to program ended up agreeing that programming a chess game is much easier than programming a wargame, this because there are less options involved and because, let's face it, chess is mostly a mathematical game: player that can anticipate and calculate more moves/developments have an advantage rather than being those with more fantasy. I am not a big fan of chess because of that. Also chess has no randomness, while in games like this randomness is basic. In the old game "Warlords" solving randomness issues was addressed by having the game calculate 20 times the results of a battle before deciding whether to engage. That can only work if the auto-fight option is well implemented (very hard to do in this game, I am sure, but Age of Wonders Shadow Magic does a very good job at that, therefor it is possible.)

The AI at the moment still lacks basic concepts:the AI still attacks stacks with its sovreigns, stack that are officially stronger (the AI calculates stacks strengs in an arguable way, but should at least NEVER attack in enemy territory with its sovreign a stack considered stronger.)

Why that is not the VERY FIRST problem addressed is beyond me!

The AI should ALWAYS keep its sovreign in its main city with tons of troops. 

Other than that, I am not a programmer: I can only wish good luck to the devs of this great but greatly unbalanced (for now) game.

Agree totally, and that is usually addressed in modern games with multiplayer options.

In Age of Wonders the longevity of the game was not created by improving the AI (which is decent but in the end, solves its main problem, the sovreign's vulnerability, by just keeping him in the city with tons of troops.)

The longevity of AOWSM was acquired by developing an effective multiplayer system. The devs of that game soon realized that people don't sit through a game for hours when they start losing, because as time passes the winner's turn takes more and more time, while the loser's side is reduced to move a couple of units and then press the end turn button. I wrote a long post a while ago, explaining why Play By EMail is the only option that is ever going to work for multiplayer:

Players get their turn by email and send it to the next players: no long boring sessions, no need for players to all find the same time frame to play, no need to wait for the other to end their turn (I like to watch my empire and take my time: that's part of the fun) -

An Epic game can only be carried through with PBEM. I can state that for sure because I have played MANY games one hour a day for a year without anyone complaining. Besides that way the loser, while a bit annoyed doesn't take long to  end his turn and doesn't have to wait for me to end mine. (Having more than one game started at the tme gives an extra motivation for losers to start their PBEM session)... And then surprises DO come: sometimes a side that looks like it is going to lose suddenly recovers and amazing things can happen! All that wouldn't happen with regular multiplayer because people are used to singleplayer games where you mostly progress straight to victory, and they get "bored" as soon as something goes wrong.

Looks promising, frogboy. I am very confident to witness a new masterpiece in the making.

Looking forward for the demo!

The problem is, it's not possible to create a challenging AI without buffering it (increasing its handicaps and bonuses). An AI is simply not capable of competing with a human brain on a 1:1 level. Even on ridiculous setting in the current game the AI is buffered a lot (800 hp catapult squads and 1200 hp recrutable monsters) although it doesn't seem so because it's so easily stompable even on ridiculous (just beat it before it can get this far ahead of you).

I think just about every computer game I've ever played will buffer the AI on higher difficulty (whether stonger units, more units, more production, gold, resources, knowledge, science, food etc). And this is a very good thing, otherwise every computer strategy game would sux beyond belief (might as well not even have a computer opponent). The only computer game I can think of that doesn't buffer AI is chess, and even that took decades to create a truly impossible & challenging AI: 1968 to 1996 beating Kasperov. But, chess is completely different animal with fixed # units & moves on a one dimensional 8x8 grid that never changes. Games of this simplicity would be much easier to program (relatively speaking) a challenging AI than games like EWOM which have essentially infinite combinations of units & moves (in comparison to chess).

I have no doubt that the AI can be "improved" without buffering, but I don't want to play an AI that I can still easily whip after it's been "improved". This is not an improvement to me. If some players want to play a very simple game that they can win all the time & offers no challenge the "easy" setting should provide this as a sort of OJT training tutorial, but there should be a gradient of difficulty settings that increase the amount of buffering, say 5 { easy(5% buff), average(20%), challenging(40%), hard(75%), ridiculous(150%) }, that ramp up to "ridiculous" where "ridiculous" means just that: impossible to win except with exreme luck (maybe 1 in 100 games). If there is a strategy that can be used (that does not involve "cheating" or "hacking") to consitently beat "ridiculous" then something is wrong with the AI and it needs to be "fixed". 

Isn't this pretty much the same as choosing a difficulty level, where difficulty ranges from easy (5% buff) to impossible (150% buff) ?

I suppose a slider would give slightly more results: say you can choose 6%, but 6% is not that much different from 5% (easy). 

Not for me, regardless of how challenging the game is, if the game is balantly cheating then it ceases to be fun. Personally I would like an entire dialog filled with options to customize each opponent and how they will play (intelligence level, military buffs, econ buffs, personallity, cheating on/off, ect.) This can be done with mods right now to some degree, but I think it would be good if it could be done ingame and the 'difficulty level' (each level would be defined in the resources being used) just sets the base levels and settings.

If you can get that to work in a practical way in a strategy game & create an excellent, challenging AI that functions, then I'm 100% behind you.

True, some players probably do cheat. I have my own personal "game-code-of-chivalry," so to speak. I never cheat (restart & save), I never use hacks or cheat codes. If I were to ever cheat, I wouldn't want to continue playing because if I won, I'd know it was only because I cheated. Cheating pretty much sucks the fun out of a game in a similar way that un-challenging AIs do.

... but that's just me.