If we're doing an after evaluation report on the Elemental beta process, I concur with many of the comments here.
1. Most of the beta versions weren't stable enough to test properly.
2. With-holding crucial content felt like the whole beta process was a sham and that, as a tester, any feedback was inherently going to be largely irrelevant.
3. If content should be held back from the beta testers then the beta process should move to a closed beta at that point where select testers have access to all the content.
4. The standard response during testing was 'The real game is much better - why are you wasting our time with this irrelevant feedback when we only want you to focus on xyz?' which became extremely confusing as a beta tester.
5. The many warnings from the testers that the game wasn't ready for launch were treated as personal attacks - which as a tester was intimidating and would have stopped a lot of honest feedback.
6. Often feedback would be given without any sort of response. This same feedback would be given time after time and as a tester the assumption is that no-one read or cared about the feedback which left you thinking 'Why bother.'. I still believe the 'to-hit' percentage in the tactical battle unit card is incorrect but there has been no confirmation or denial that this is the case.
7. Not having the easy ability to upload error reports meant that many testers would not have bothered responding.
8. Perhaps a more structured approach would be better such as the way bugs are now handled in many web applications with a dedicated bug list.
9. The amount of access to the development team via the forums was very positive in general.
I honestly didn't enjoy the 1.0 beta process at all. I became way too emotionally attached to the game which left me feeling extremely frustrated when it didn't live up to expectations. On top of the frustration it ended up chewing way too much of my time. Unless the process is changed I think I'll still try the beta games but only as a preview rather than to give much feedback. The 1.1 beta process was heaps better because of Kael's changelogs which at least gave you concrete context of what was happening.