With so many ways of collecting data, deciding what to do with it can be a nightmare.
Designing a game can be like making porridge for bears: some people will think it’s too hot, some will think it’s too cold, and some will think that the boss at the end of level three is “cheap.” But in in Issue 274 of The Escapist, Robert Yang says there’s no sure-fire way to know who to listen to and who to ignore.
If you collect data from highly competitive clan servers, or perhaps from someone who’s never played a videogame before in their life, are those sets of data valid for balancing the game for everyone else? (It depends.) Should we instead test on some sort of “average player” and if so, then who is that player? (It depends.) Is that really the best way to achieve accessibility, or do we end up pleasing no one by trying for everyone? (It depends.) … And then how do you go about interpreting that data you’ve just collected?
Imagine in Team Fortress 2 that data indicates fewer players are playing as spies – does that mean Pyros are overpowered or that Engineers are too difficult to kill or something else entirely? (We would need more data.) And if Engineers are too difficult to kill as a Spy, is it actually a level design problem with specific overpowered build sites on popular maps, or is it a sound-related bug where the Spy’s cloak sound is too loud, or is it a balancing issue with how the Spy’s cloak doesn’t last long enough to get past the front line? (We would need more data.) Or is this a good thing, to have so few players playing as Spies? (It depends.)
The data can only ever be a guide and any decision a developer makes will ultimately be a compromise, and a short-term one at that. You can read more about it in the second part of Yang’s four-part series, “The Philosophy of Game Design.”
