Playtesting During A Pandemic

The Playtesting Guide To Make Your Games Look and Feel Good When The World Is on Fire.

Written by Maria Mishurenko

When “unprecedented times” arrived this Spring, I was a teacher assistant for the Games Studio 2 course at the NYU Game Center. When the university closed its doors in the middle of March to comply with state executive orders, we had to quickly adapt to the new remote, zoomified reality. That meant, first and foremost, applying a new process for the playtesting, getting feedback, and iterative game improvements. 

It was a hard and interesting challenge for me not only as an educator but also as a designer because my game Bizarre Barber launched just two weeks before the pandemic. My team was planning to address some post-launch feedback and implement new features into the game. At the same time, I’ve been doing some game-related contract work (oh, the never-ending hustle of the NY game dev scene!). And my clients have been clueless regarding how to make sure the design iterations are going in the right direction without regular playtesting routines.

Pre-Pandemic Playtesting Routine

Playtesting at VRBar before the pandemic.
My last in-person playtest at VRBar before the pandemic.

Before the pandemic, my regular routine included playtesting with the target audience every time we’d added a new feature, usually one or two 2-hour long sessions per week. For my personal projects, I’ve usually playtested either on XR-related meetups or at the NYU Game Center Playtest Thursday. For client projects, the internal team members or focus groups usually conducted playtesting and UX user testing in the office.

Once the offices, meetups, and university campuses closed, it felt like game developers lost the connection with the players and access to the most powerful tool for improving the game. In-person conferences where I regularly used for collecting the feedback were gone. It was time to quickly adapt.

After the sad realization that we all would be in this upsetting reality for a while, I started my own process of remote playtesting research and adaptation. Some ideas were from my wonderful and creative students. Some came after the collaboration with clients. Peer knowledge exchange was crucial for my process as well. Below are some advice and thoughts on playtesting philosophy and organization that I refined over eight months of being bound to the laptop in my living room. I have also asked fellow game developers to share some playtesting tips and tricks and included their quotes in this post.

Why Playtesting?

The pandemic horrors and the uncertainty of the future take its toll on mental health and productivity. Remote playtest organization is hard and time-consuming, especially with no prior experience. So why bother? The answer is simple: without the regular playtesting routine, the developer is working in the dark. Sometimes, with incredible luck, people manage to release great untested or undertested games. Don’t count on it. With proper playtesting routines, you will save tons of valuable development time and will always have a valid action plan.

Where To Find Playtesters?

Even before the pandemic, many game communities had a large online presence on Discord, Steam forums, Twitch, Slack, and Facebook. After the pandemic, the presence only increased. If you are not sure how to find those communities, you can always consult your local IGDA chapter or any other community organizations. 

Other game developers are your first responders when it comes to saving your games from feature creep and wrong design strategies. You can start playtesting with peers much earlier, even when the game is not ready to face the general audience. You can playtest gray-box levels and get decent feedback because a colleague has enough experience to imagine your vision. Start with the devs who are working in a similar genre and medium. Many online communities have dedicated show-and-tell days and ongoing mutual playtesting. 

Once you’re ready to test with target audience players, you can start looking for them by defining their personas. This research will eventually help you with game marketing. You can join their communities and offer to participate in the “beta-test to own” program. That means you ask your players to test and provide feedback (sometimes on an ongoing basis) in exchange for a game key and a game credit. If you get the target audience right, people are usually enthusiastic and happy to participate in game creation by providing their feedback and thoughts. However, respect your audience and know the boundaries. If you expect your players to spend more than an hour for the feedback, make sure they are also compensated for their time. 

Playtesting For AR/VR Games

Playtesting for VR/AR games.
Illustration by Olga Ivanova

Testing AR and VR games present an entirely new level of challenge. I’ve asked Olga Ivanova, product designer, community builder, and game creator about her thoughts on VR/AR playtesting during the pandemic. 

Due to the pandemic, it’s not safe sharing headsets anymore, so it is harder to find testers for traditional in-house user testing during this time.

The challenge is: where to find all these people who want to playtest your game? One of the options can be recruiting people in Facebook groups. VR AR Testing specifically aims to help connect VR and AR creators with the community. Members can make a post about an app that they want to test and list all criteria including:

  • What are you testing (e.g., VR game, 360 video).
  • Incentive (monetary compensation or any other option valuable for your audience).
  • Any important details (devices requested, demographics, the timeframe to complete the test, and any other qualities you are looking for).
  • Link to your build or keys if the app is already launched.

Interested members will comment and share their feedback right in the comments. Participation is voluntary and results purely rely on your expertise with user testing. 

The downside is that you do not always know what the response rate will be and if you will be able to complete testing on time. In addition, it may not be consistent among all testers since there won’t be any methodology involved. All the members may be absolutely brilliant candidates in various fields, but talking to the wrong target audience may bring the wrong results.

Remote playtesting session for Masterpiece VR.
Remote playtesting session for Masterpiece VR. Photo from VR AR testing Facebook group.

Practical How-Tos For Effective Playtesting Sessions

The playtesting process itself is highly individual depending on the game specifics and the developer with the details depending on your genre and medium. I usually follow these simple rules:

  • Have internal goals for each playtesting session. That could be, for instance, validation of some design hypothesis, feedback on a new feature, making sure that the previous fix works well in the current build, etc.
  • Don’t give up too much information on what the game is about or how to play it.
  • Encourage your playtester to talk out loud during the playtest session. 
  • Don’t interrupt or help during the session. Only respond if the player explicitly asks for help after being stuck for a while. 
  • Be polite and appreciative of the person who takes time to play your unfinished game during a pandemic.
  • After the session, have your interview questions ready, but be quick. I usually plan my interviews so that they take no more than 10 minutes (3-4 minutes for VR). 
  • If you playtest asynchronously, make sure your player has step-by-step instructions on how to properly run the game and a well-designed short survey to fill in after. 
  • Be very clear about how you use player data (email, game data, survey answers, playtest recordings). Let players know that you care about their data and privacy.  

An Effective Bug Report During Playtesting

I asked Manuela Malasaña, owner of Team Dogpit, former automation engineer and solo developer behind The Cherry Orchard game about her favorite playtesting process: 

The most useful thing a playtester can provide to a developer is a well-written bug report, which is a report that helps the developer fix the bug as fast as possible and has at a minimum these three things:

1. What went wrong (this is the most obvious one!)

2. How to make it happen again. Before you send a bug report, you need to see if you can make the bug happen again! Retrace your steps and do everything you did up to where you found the bug. Make sure to write down if you can make the bug happen every time by following the steps or if you can’t make it happen again predictably, or if you can’t even make it happen again at all. This is important for helping the developer find what is actually causing the bug.

3. What you expected to happen. Most people forget this step, but it’s very important. It’s possible that something you thought was a bug was actually supposed to happen. Or it’s possible that the design documents are outdated or wrong! 

It’s also very helpful to include screenshots and video if possible, as well as the specs of the device(s) you’re using to playtest. It might seem like a lot of work, but the clearer your bug report is, the faster the bug can be fixed. Here are the screenshots of some fun bugs found by my playtesters, alongside the bug report excerpts:

Title: Mask is not applied during glow effect

Area: Shader

Steps:

  1. Start the game.
  2. Open the debug HUD by pressing F1.
  3. Enter 77 into the line field, select Act 3 from the dropdown and press Go.

Expected result: when the glow effect is activated, only the parts of the mesh where the glow mask is white will glow.

Actual result: the entire submesh glows.

Bug report during playtesting.

Bug report during playtesting.
Title: Act 3 Trofimov materials are set in the wrong order

Area: Material

Steps:

  1. Start the game.
  2. Open the debug HUD by pressing F1.
  3. Enter 30 into the line field, select Act 3 from the dropdown and press Go.

Expected result: on-camera changes, materials will be swapped in the order face, body, outline.

Actual result: on-camera changes, materials are swapped in order body, face, outline.

Less Friction Is Always Better! 

You can playtest most PC/ Mac games over Zoom, which allows you to draw on your playtester’s screen and even control their computer with mouse and keyboard. These are very handy features if you need to explain something quickly. Also, everyone uses Zoom nowadays, so you can be sure that you won’t need to explain the UI of the meeting software in addition to your game. 

To minimize the friction even more and make a game suitable for asynchronous playtesting, make sure you add as many data collection methods in the game itself as you need. Unity built-in analytics can give you a pretty comprehensive overview of the player’s behavior (heatmaps are especially neat!). But if the Unity solution seems too complex, you can always write your own data collection to save to JSON and ask the player to send the data file back to you after the playthrough. After that, you will be able to open the data in the spreadsheet editor and analyze them.

Data collected during playtesting for Bizarre Barber.
I used data collected from Play NYC 2019 Bizarre Barber playtest to prove the hypothesis that tall players get better scores. All the data were collected automatically through in-game hooks.

If you are near the beta stage of your game, using the platform beta-testing tools may be convenient for internal housekeeping. Steam, Itch.io, Oculus store, Apple, and Google Play have internal closed testing systems with Itch.io being the most universal and convenient for indies. Using their Butler client, your testers can even download incremental build changes/patches instead of the whole builds!

Sometimes, when you playtest in real-time, you can ask the player to leave their PC camera on and not share the screen at all. Seeing emotions on a human face in real-time will help you to solve design problems faster than playthrough recordings, especially if your game has complex narratives. 

Consider Multiplayer As A Playtesting Method

Online multiplayer games are especially well-suited for pandemic with their capability to enable connection and facilitate fun time with friends. Implementing an online multiplayer to simulate local multiplayer functionality is the most common case. Keep in mind that adding an online multiplayer may present many unexpected challenges for the remote development team. 

Infamous Duck Ball started as a simple local multiplayer competitive game. It ended up being an entirely different, fun, complex, and engaging physics-based action game about naughty geese on vacation for four online players.

I asked one of the developers behind the game, NYU Game Center student Williams Xue, to share his experiences with developing an online multiplayer game with a fully remote team:

There are some decent remote broadcasting solutions like Parsec out there for cases when you only need to conduct remote playtesting, and not the whole online multiplayer game. 

However, it is always great to add an online multiplayer game to your portfolio! Here are the pros and cons, based on my experience:

Pros:

1. You may potentially get plenty of players in the future if the game is fun.

2. The game will be ready to play without much hassle and extra downloads. Not all playtesters are willing to download third-party software like Parsec just for the playtesting.

3. You will seem cool and professional.

Cons:

1. It’s hard to reach enough players with a limited budget and no ads.

2. It’s hard to get enough players for proper matching (this is a real problem!)

3. The game should always include a playable local-multiplayer mode.

4. The development is much more time-consuming than a local multiplayer game.

5. Many network tech providers like Photon only allow players in the same region to be matched. Cross-regional playtests are hard. 

Conclusion

You should carefully consider all your circumstances before making an online multiplayer indie game. For us, it took three times as much time and effort!

Adding Multiplayer Even If Your Game Doesn’t Support Multiple Users

You may consider adding multiplayer even if your game does not support multiple users. It can help a great deal when playtesting VR games and experiences. Being together in the world, being able to point your fingers at things, and touch everything makes iteration incredibly quick and efficient. Implementing an online multiplayer requires additional development skills, but luckily there are plenty of accessible software solutions that can simplify this herculean task. 

  • Photon is one of the oldest solutions on the market, well-documented, and stable.
  • Mirror is open source and free.
  • Normcore 2 offers an entirely new development paradigm and saves tons of implementation time with its clever design and architecture.

If you wanted to get into online multiplayer development, now is a great time. The demand for fun multiplayer games is high. Isolated people are hungry for connections and escape that games can provide. 

Playtesting For Tabletop Games

During the pandemic, some tabletop game designers had to learn an entirely new skill: digitizing their creations to be able to playtest over the Internet. Luckily, there always have been special tools out there to make the process seamless and painless. 

I’ve asked Alina Constantin, game designer & artist, about her favorite tools for playtesting non-digital games remotely:

I’ve used Tabletopia and Playingcards, which were accessible for me – a very non-tech human – along with a researcher from Tandon School of Engineering whom I’ve had the honor of bringing into the games field in the last months.

Tabletop Simulator

I got myself Tabletop Simulator but never got much into it. I didn’t want her or her students to have to buy an account. Although you can’t run too many sessions with a free account, I’ve still managed to get a lot of mileage and it took very little time to set up. Onboarding, which is usually a huge deal for me, was not a problem here.

PlayingCards.io

Playingcards.io is even quicker. However, in my limited experience, it’s best to use it for paper prototyping on a card deck or existing board game system. I’ve used it primarily for cards, but it has templates for the standards like chess, checkers, and backgammon. It is very quick to add/remove the number of game objects on a board for basic game object balance or “paper prototyping.” I would definitely recommend the platform. I have two projects that I started in the last months that I’m working, on both.

Tabletopia

Tabletopia has a nicer feel, bare bones of 3D physics, moving things around, flipping, and a broader range of existing objects as well as customizing.

Playtesting for tabletop games.
Yuxin Gao playtesting an early prototype of the Extraction tabletop game, a collaboration between sociologist Mona Sloane and Alina Constantin.

Conclusion

The reality is challenging for many of us. Every opportunity for authentic human connection should be cherished. I became friends with many of my pandemic playtesters and found wonderful emotional support amongst my players. My outcome from this little research project was not only that remote playtesting will make your game better and save you some development costs & time, but it will also make you feel better by providing validation to your craft and introducing you to new wonderful friends around the world.



Maria Mishurenko is a Brooklyn-based game designer, developer and CEO of Synesthetic Echo XR studio. You can find her on Twitter.