Yeah, there's a lot of very good mechanical justifications for not allowing fleets to retreat. While I suspect most of those can be overcome (Director's tactical speed approach, with significant elaboration, would address a fair number), you'd need more reasons as to why it's worth it.
Some things that would need to change before retreat is even worth discussing:
1) Ship speeds need to be nerfed to hell and back. Retreat cannot work in a game where some units can move 50+ tiles in a turn. Personally, I think that a speed limit of 12 tiles/turn would be a good move for the game anyway, and would make the possibility of retreat worth talking about; as it stands, the very concept of allowing it would be game-breaking.
2) Some means to intercept when it's not your turn would need to be included. Otherwise, Retreat will simply make the same battle happen over and over again until someone runs out of moves. There will be no actual change in the result; I'm just being forced to attack you a dozen times to achieve it.
3) Possibly implementing halt-on-attack. I'll expand on this one a bit more, since I know most players will absolutely hate the idea, but actually, I think this would be a good thing for the game anyway.
Presently, the AI does this anyway, so moving after attacking is actually just a massive advantage for the player. With the insane movement rates players are able to achieve on their ships, it's not uncommon for a mid-to-late fleet to be able to clear out a dozen or more enemy fleets on the first turn of a war, and to withdraw to safety afterwards. This, quite simply, breaks the game.
Halt-on-attack would give us several key improvements:
1) it would make huge engine stacking much, much less valuable. Why bother building a warship with 50 moves when half of the time, half those engines are doing nothing? I'm just making a less-useful warship compared to the lumbering 5-move-per-turn monster with twice my firepower.
2) It would slow down warfare and make it actually take time to best even a small enemy. A turn is 1 week, ffs. It takes me less time to cripple a multi-star system empire than it takes for the average French surrender. I have been in pub brawls that have lasted longer than interstellar conflicts do in GC3 right now.
3) Being able to control space would begin to matter. If I have a fleet that gets half-wrecked in combat, I presently just use the remaining 12+ moves to run away back to reinforcements. My supply lines are basically safe, because I just fly round anyone getting in the way with no penalties. But if my front lines were forced to actually remain on the front lines, being able to bring reinforcements right up to them would matter, and being able to keep a reasonable force in place to protect damaged units would become a thing.
4) It makes more sense anyway. We're travelling vast distances at multiple-times light speed. Again, every turn is 1 week. It takes time to stop and fight a massive naval engagement. In real life, engagements often takes days, weeks or even months. If we even just say that it takes 1 day to fight and win a space ship battle, that still amounts of 1/7th of your ship's movement; with a 50+ move ship that should be 7 or 8 tiles. This is without taking into account regrouping the fleet, recovering fighters, patching up minor damage etc. Hell, just losing 1 turn to
These all combine to severely reduce the massive advantage that attackers presently enjoy and make attacking require some actual tactical thought. And it might make retreating actually have some point rather than just being an invitation to micro-hell. 1 attempted retreat per turn permitted, success based on some mechanic (I like the tactical speed idea), if it fails then combat happens, if it succeeds then the attacker loses their turn and the defender moves away 1 random hex .
This might give us something resembling a strategic war game, rather than the present system, which more resembles a brawl in a playground.