XNA Peer Review Process
After going through, and failing the peer review system for XNA twice now I have built up some complaints I have about the system that gets independent games from the XNA community site, to the XBox Live Service. I would like to preface this post with the fact that I never thought, going into this process, that I would look back on it and say that it was harder than an actual Microsoft Certification... but at this point in time, I am.
A basic overview of the process:
You submit your games to the community portal and peers can then download it and play it on their XBox. There is a forum post called the “Evil Checklist” that you are suppose to go by for checking games... A FORUM POST!!! All of these “rules” are written as guidelines, and there is no actual checklist to determine if anyone actually looked at a single one of them. Not to mention, the forum itself says that the rules are subject to your interpretation. Nevertheless, people then review your game and pass or fail it. If enough people pass it before it fails you go to XBox Live... seems simple right.
Now, just a few of the flaws I see in the current system:
- The people “reviewing” your game are the ones that are going to be sharing XBox Live marketing space with you... HUGE RED FLAG TO ME.
- The fact that anyone can interpret any of the rules however they want opens the door for anyone to simply fail a game cause they do not enjoy it.
- It takes one fail to be rejected... One. Even if you are one pass short of approval.
- There is no rebuttal process for a fail. You simply are taken down from the list and cannot resubmit for 7 days.
- If you have a group of 8 or so friends with Community Games accounts, they can approve the game in the first hour it goes up for review and you basically bypass the peer review process and get anything you want on Live (although it would obviously be pulled right back down if it contained material against the terms of use)
Now some food for thought on how to solve the situation:
- Make a game require two fails before rejection. This prevents one angry jackass from keeping what is a bug free, great game off Live.
- Make a cut and dry set of rules (Like Microsoft TCR’s), so that there is no grey area as to whether or not something passes
- If you have failed a game before, you cannot fail it again. This prevents what I call “sniping”. Which is basically a single person finding your game every submit and failing it because he does not like it.
I feel that any of the implementations above would greatly improve the system, yet not require a huge overhaul. Each game has a forum thread created for the reviewers, and to fail the game you need to give a reason in that forum. If you find a fail issue, you post it, and the future reviewers can look at the forum and know what issue to look for in a new release. If they read the forum, and feel you failed the game for an unjust reason, it is their call on whether they feel the issue is worth a fail or not. The idea that you can continually fail your competition, for issues that your own game exhibits seems absolutely insane to me.  It defeats the spirit of independent game development and goes against the feeling of “community” that Microsoft says they are trying to create. It actually creates little networks of 8-12 creators that go around passing their own games and failing the competition. Not a very friendly environment.
Just wanted to give my thoughts on the system, and how I think it could be greatly improved. This would allow better games to get to the consumers, which really should be the goal.
Thursday, March 18, 2010