Feb 232012

So we’re all on the same page here, I want to assign some reading. Here’s two recent articles:

Sensitive Opinions

A lot of what Jenn has to say is spot-on; a lot of the games submitted to the IGF are, indeed, pretty crappy or just absolutely broken.  And overall this is drama, and it’s kind of annoying to see it cropping up year after year. However, Jenn targets her ire at Rotting Cartridge in what I think is an unfair manner, and uses some straw-man arguments as well (RC’s game didn’t appear to crash on install, for example).

At the same time, I agree with what Rotting Cartridge has to say: everyone deserves a fair shake. If you paid the entry fee, you deserve to at least get some eyeballs on your game. I also partially disagree with their passionate post; at least they got some amount of play from the IGF judges.

I have no idea how to judge this particular case; I don’t know the game in question, I don’t know if I trust the game’s metric sources, and I don’t know if the bit of judging they did get was indeed fair. Reading those two articles, though, will give you the opinions of the two camps: Either you think the IGF “does the best it can,” or you have been “personally slighted” by the IGF.

Some praise for the IGF

Before I go on, I want to mention that I have the utmost respect for the IGF in general, and Brandon Boyer (and Simon Carless’s) contributions to the whole thing, and how everyone – developers, judges, organizers – wants to make it better. I don’t think anyone should say the IGF is a bad thing.

I would hate to live in a world where the IGF did not exist. I’m proud to have been a part of the IGF, and I continue to submit games for judgement there.

But the IGF is not perfect

I have my own story about the IGF judgement process that I’ve tried to hash out semi-publicly before (mostly on Twitter). It was over a year ago, and I’m over it now, but here’s the super-short version:

  • I submitted SteamBirds: Survival to the IGF last year, before it was launched to the public
  • The game contained input-recording (replay) functionality to aid in my playtesting pre-release.
  • The game would not run if it could not connect to the replay storing functionality
  • Exactly 1 person played the game for approximately 30 seconds. I am not convinced it was a judge (could have been a test-open or something; it’s hard to even get to gameplay within 30s)

After learning of this, Brandon Boyer was very pro-active at getting to the bottom of this. I think everyone agreed this was not the way the IGF is supposed to work. Unfortunately, judging had already ended, and my contribution to the gaming world would not get it’s “fair shake.”

In the end, it turned out a few other judges did play the game – they already owned it on their iPad and decided to play that version, instead of playing the version submitted for judgement. Of course, it never crossed the judges minds that the submitted version was in fact a sequel to the mobile editions they owned. For that I got heart-felt apologies, which I guess is better than nothing.

After telling this story a few times, I happen to know several other game developers that are recording similar metrics or replays in this year’s IGF games, to ensure they also get a fair shake.

So, yes, the IGF isn’t perfect. There are some problems.  As Team2bit says on twitter, “I never expect perfection for $95″ – but maybe we can improve things a bit.

The solution?

Let’s talk about the issues. Let’s figure out what’s wrong, and let’s work on solutions. I don’t think the appropriate response to legitimate complaints is to “STFU and deal”, as Jenn Frank implies.  Boycotting the IGF is probably an over-reaction too. The knee-jerk response to someone getting frustrated shouldn’t be “try making your own festival then.”

Me discussing my problems with the IGF organizers (hopefully) turned into new policies or procedures that will prevent the problem  from happening to other developers.

Ignoring the issues will not make the IGF better.

Talking about them will.

  10 Responses to “IGF is awesome, and not perfect”

Comments (8) Pingbacks (2)
  1. Yeah I don’t get the overreaction on either side, really. The conspiracy theories on one end, the dismissive, downright mean and mocking attacks from the other side.

    The IGF is amazing, but it will obviously make mistakes, and there’s nothing wrong with talking about it when that happens. Fortunately the IGF officials have been really responsive and care a lot about improving the process, so I’m optimistic that it will keep getting better.

  2. The problem is there’s a huge amount of games and probably a lot less judges. Just getting more judges wouldn’t fix the problem as I believe they do their best to get people they can rely on so quantity would be a risk. Or maybe they tried to fix this problem by getting more judges and ended up with just more problems. I don’t know.

    The only thing I really know is that the tone of BOTH these posts you linked just doesn’t help (well it brought people to talk about it but I wish it could have done the same thing without having to rely on that tone). So fix #1 might be to just lower the heat a bit. All competitions have their problems and shouting at each others just slow down the process to find solutions.

    So yeah I guess I just said the same thing you did. Oh well I tried.

  3. Would it not be possible to crowd-source the initial selection process? Cull the entries down to a Top 100 before real judging even begins?

    Some sort of intelligent list could make this work, ie items with the least reviews are at the top of the list, the big ‘hype’ games would be all the way at the bottom, this way all games are getting exposure, and a fairly even number of ratings.

    That would make it pretty easy to just cut off everything that is rated <80% or some other cutoff….

  4. Culling games before the real judging is exactly what the first-round judges are doing, and it’s those first-round judges that were being critiqued by the rotting cartridge.

    The IGF has its flaws; I just don’t think their particular complaint is a terribly valid one.

    In some ways the IGF tries too hard; it accepts games that it should reject on the grounds of being completely unable to judge them, and it allows developers to keep submitting new versions of their games rather than forcing them to stick with a deadline. That in turn contributes to judges delaying to the last minute and then rushing through games.

    Until and unless the IGF actally has a framework to provide adequate judging for longform and multiplayer-only titles, it shouldn’t accept them into the competition. Same for way-out-there weird stuff like installation games or incredibly obscure platforms. The standard judging system cannot cope with them.

  5. Thanks for this post Andy.

    It’s kind of weird, I get the feeling that people subconsciously think that the IGF is both a lottery where anyone has a small chance of winning and that it’s also an in-depth judging process where your game will be played thoroughly and critiqued/reviewed by experts.

    It is neither. It’s a professional competition with a judging process to meet its needs- which is to select great games to be awarded at the festival. They don’t need to find the “best.” They’d like to just as much as you would but in that time frame with that many entries [and judges] it’s simply not possible to be sure.

  6. I went a bit further and read Brandon’s comment on RC’s original post, then Adam Saltzman’s comment on Mike Meyer’s GPlus post. Out of the whole mass of literature I’ve read this morning, those two probably have the most value.

    Brandon is a pro here; he has to carefully deal with what essentially amounts to a personally attack (after all RC publicly rejected the IGF’s gracious attempts at remedying the situation) without providing ammunition for further vitriol. And Adam’s second point of the human cost in using this kind of drama to promote the game is spot on: I’m sure some people are already downloading Kale from the AppStore as we speak, to “stick it to the man”.
    While I disagree with the tone of Jenn’s rebuttal, her point is spot on. This is grandstanding of the worst sort.
    Disclaimer of sorts, I have not been doing this long enough to submit my work to the IGF, so I can’t speak to the veracity of anyone’s claims in this situation, I would like to think that were there any problems or issues with my submission, I would not slap away a gesture from the administration in such a crass and offensive way. This is not WikiLeaks, there is no need to be blowing any whistles about a judging competition for games.

  7. @Patrick: Yeah, one of my hesitations about posting so publicly about my issues previously was I thought it would be taken to be grandstanding/whining about my game/etc…

    And even privately, speaking to other developers, such was said so to my face. “Maybe my game sucked” “surely you don’t think your game should have won” etc etc…

    I feel it’s safe for me to talk about it now because I’m not emotionally wrapped up in it, I don’t own the SB franchise anymore, and it’s last year’s ordeal. :)

  8. @Andy that right there is probably the most wise thing anyone has said yet! Cool down, let the emotions fade and think about it rationally for a bit.
    At the very least what you’ve put up is not a rant per se, rather a description of the issues you had and how they were resolved. But then again, that doesn’t make for a gripping post with hundreds of comments and trackbacks. On the Internet, no one care hear you appeal for common sense…

Sorry, the comment form is closed at this time.