When you make a game, you want people to enjoy it. But how do you know if they do? Most of your friends will tell you it’s great, even if they got stuck on level 2 and decided to go do something more fun, like cleaning the oven.
Another problem is that of numbers. If you don’t have hundreds of friends, and the ones you do have are busy sanitizing their kitchen appliances, you’re not going to get enough data to base decisions on. In other words, you need a lot of people who are actually playing your game, and some way of figuring out what parts they’re enjoying, and what parts they’d rather do without.
Enter metrics. You publish a beta version of your game, post the link on some forums, and make sure the game captures data on how it’s being played. For our beta versions, we (anonymously) recorded several key player behaviour statistics, such as:
- what levels you play
- whether you completed a level and with how many stars
- how long it took you to complete
- how many undos you used
- what levels you skipped
Then we crunch the numbers. We calculate averages for each level, which tell us something about difficulty. We also determine the dropoff rate: the percentage of players who quit while playing a given level. Levels with a high dropoff rate are candidates for tweaking or removal, and it’s instructive to see what such levels have in common.
It’s not always easy to come up with the story behind the statistics; there’s always a bit of intelligent guesswork involved. But here’s some things we think we can conclude:
- The first higher dropoff rates (>20%) occur at level 5 and 6. This is most likely people who check out the game and decide it’s not for them. We could try to introduce new features of the game earlier to keep these people’s interest, but that would risk alienating the people who prefer the gentle buildup in features and difficulty.
- Complex-looking levels tend to turn people off, even if they are not necessarily the hardest type of level. We’ve tweaked levels to remove unnecessary cruft while keeping the essential nature of the puzzle intact.
- Levels that cannot easily be completed with only 1 star tend to turn a group of people off. The ideal level is fairly easy to complete on 1 star and more difficult on 2 and 3 stars.
- There’s a button in the game to skip any level. But levels with significantly higher dropoff rates don’t have much higher skip rates. So, people don’t seem to notice the skip button. Calling more attention to this feature might help keep more people playing.
- After a high-dropoff level, the dropoff almost always goes way down. So completing a tough level motivates people to keep playing. Or, tough levels get rid of the quitters. ;-)
- An easy level at an advanced point in the game also seems to lose us players. They probably figure that because they breeze through it, they’ve mastered the game and need not continue.
- The last few levels (27-30) of the demo have significantly lower dropoff. This is likely because in level 27, the story starts to build toward the chapter 1 cliffhanger, with the antagonist beleaguering one of the main characters.
Metrics should never drive your game design, but it can help you to figure out what rough edges need to be smoothed over. If you get it right, that means a lot more happy players. And a lot more dirty ovens.
Trackback from your site.