There is a 2 gig ram usage limit hardcoded into the Sins engine. Malestorm is a massive mod it is possible that after a time the RAM usage hits this limit and the game dumps. Try reducing your settings and see if you can last longer.

Edit: Just to expand on what i wrote above. It doesn't matter how much RAM you have (as long as the free RAM is greater than or equal to 2 gig) it is when the usage by Sins gets over 2 gig the game crashes. This is because the game was started long ago when 2 gig looked like  a HUGE limit.