User Research in the Games Industry: Opportunities for Education

Them and Us
Game developers love the thought of getting feedback from actual gamers, but they typically do not possess the expertise to collect valid data about users’ opinions, perceptions, and behavior. Developers and their design teams may occasionally bring in consumers to test their games, but concepts such as response bias and control are foreign to them. In our group, we take a multifaceted approach to testing, and we are often inundated with data. (Nearly
all of us have some graduate training in behavioral research, so this keeps us happy.) We employ converging measures to answer research questions and incorporate surveys, usability and RITE studies, playtests, experimental playtests, and so on.

Let’s say, for example, a game developer wants to know how much fun the game is. Left to their own devices, developers would likely set up a session in which users play part of the game, then are asked to respond to a questionnaire that includes questions like “How fun is this game?” “What is fun (or not fun) about it?” and “Why is it fun?”

At first glance, this may seem a reasonable set of questions to ask users when they interact with your game. But “fun” is an enormously complicated thing to measure. There are dozens of variables that affect a user’s fun rating, and many of these interact in complicated ways. A user may rate a game as “not fun” because she has not played many games within the current game’s genre;
for instance, games in the shooter genre typically employ several consistent features that games from a role-playing genre may not. This leads her to interpret the gun-aiming system incorrectly, which in turn causes her to perform poorly in the “shoot the things shooting you” level of the game.