Vital Questions to Consider When Conducting Remote Testing During COVID-19 and Social Distancing

Why Remote?

In these uncertain times, individuals, businesses and entire industries have had to learn to adapt – sometimes in dramatic and unprecedented ways. Certainly, anyone involved in product development has noticed that in-person user research has been impacted and requires a greater degree of creativity to achieve results. An experienced product development firm with a strong toolkit for executing remote user research can quickly pivot and adapt to provide greater flexibility and client benefits in this “new normal.”

Kaleidoscope Innovation is one such product development firm with extensive experience employing remote testing methodologies. Our standards for delivering quality-based and accurate deliverables remain high while we keep safety for all involved as our primary concern.

More Than Just Remote

Yes, this is about remote testing. However, this profoundly unique situation requires additional measures beyond those associated with traditional remote testing. This is especially true if a physical test product or prototype needs to be shipped. Software programs, for instance, can be transmitted and displayed on the web or a smartphone without personal handling. However, additional safety measures around shipping physical objects should be both implemented and communicated to help keep participants safe and to put their mind at ease while testing.

Research teams need to exercise both flexibility and wisdom as they adapt their remote research to this unique time.

Challenges and Questions to Consider

There are always going to be challenges inherent to any research approach, especially when technologies are used. Telephones, laptops and web cams of some sort must often be used in remote testing, and this may be difficult to execute seamlessly with participants who live a more low-tech lifestyle. Even with remote technology, observing and studying human behavior and product interaction becomes more challenging without the inherent benefits of in-person observations.

Here are a few challenges and questions that our team considers with remote testing, as well as some of their tips.

Challenge: Knowing Where to Start

Questions to Ask Yourself As You Design Your Study:
- – What do you need to learn from participants?
- – What tasks do you need participants to perform?
- – If adapting from what was originally intended to be an in-person study, what needs to change and what can stay the same?
- – How do you best step participants through the study components and sections to avoid participant frustration, errors and moving ahead too quickly?
- – Should you use research with or without real-time moderation (or a combination, such as the homework and interview method)?
- – How else might you best minimize issues and maximize benefits associated with remote testing?

Our Team’s Tips:
- – Determine what you need to learn from participants, what tasks need to be performed, what data you need to capture, and other critical questions before deciding what platform or technology to use for your study. Use a technology best tailored to your study, not the other way around.

Challenge: Keeping Participants Safe – and Feeling Safe

Questions to Ask Yourself As You Design Your Study:
- – What can you do to help participants remain safe during your remote study – especially if they have to interact with a physical product delivered to them?
- – How can you prepare products for delivery in the safest possible way?
- – How can you best communicate to participants the safety precautions you are taking?
- – What concerns, real or perceived, might participants in the remote study have related to health and safety, and how can you address their concerns?
- – How can you build trust with participants in general, and particularly around the issue of health and safety?
Our Team’s Tips:
- – If shipping devices or products for remote testing, consider taking extra precautions and including convenient sanitizing solutions for participants. Sanitize the products themselves before sending and also give participants clear sanitizing instructions for when the testing materials arrive. Make sure to communicate those extra precautions you have taken as well – before and during the remote testing. Remember: you are designing the whole experience, not just the test.
- – While COVID-19 is a primary health and safety concern right now, remote testing might bring up real or perceived safety concerns in other areas as well, including cybersecurity. Keep in mind all aspects of safety associated with remote testing.

Challenge: Avoiding Biased Results

Questions to Ask Yourself As You Design Your Study:
- – How skilled are your participants/representative users in using technologies relevant to your test?
- – How accessible are remote testing technologies (i.e. high-speed internet) to participants?
- – How can you make the study as easy as possible for your representative users to complete, especially those who are less able or willing to participate in remote testing?
- – How can you design the test and testing experience in a way that keeps all users engaged / interested in starting and completing the study?
- – How can you design the test and test experience to best accommodate any cultural differences among representative users?
- – What other barriers might your participants experience to starting or completing the study beyond technological acumen and/or access?
- – How else might you minimize possible skewed results from people unable or unwilling to engage in remote testing?
Our Team’s Tips:
- – Don’t expect your participants to learn a new communication tool/technology for your study – particularly one that is far beyond their regular level of usage. Work within the parameters of what is familiar and comfortable to them.

Challenge: Collecting Quality Data

Questions to Ask Yourself As You Design Your Study:
- – How will you capture the results? What technologies will you employ for the study? (i.e. video conferencing software and webcam recording technologies, clickstream data software, telephone, email, mobile phone recording, online research platforms, etc.)
- – How do you ensure the content / data that you capture is high-quality and you are able to observe what is needed and beneficial for the study?
- – How can you make yourself available to participants throughout the study for help (both for the content itself and any technology), if engaging in testing without real-time moderation?
- – What barriers might participants encounter when trying to capture data, including if using video? How can you design the experience to minimize errors?
Our Team’s Tips:
- – Think about how what is being asked of participants can impact their ability to best capture the experience. For instance, if a participant’s task involves using their hands to test out a product, they will likely be focusing on that more than camera angle. By the nature of having a product in their hands, they also physically may not be able to adjust and optimize the camera angle. Thinking through challenges like these can impact how you design your study. For instance, to help overcome this challenge in the example above, it might be better for your team to use testing with real-time moderation so the moderator can ask for camera adjustments as needed.
- – Clear instructions are critical, especially when the moderator and participant are not physically in the same room. Make the instructions both legible and understandable. Consider font size and typeface for legibility, and be direct but comprehensive in your instructions to participants.

Benefits of remote testing

We have outlined a few of the remote testing challenges and questions to consider while addressing them. But there are some important benefits of remote testing that are important to capitalize on as well.

To start, right now remote testing helps ensure the safety of all involved during these uncertain times. This is a central benefit as research teams should be focusing now on safety of their participants and team members as their primary concern.

Additionally, remote testing in general enables research teams to reach a broader group of people across a wider geographic range, including internationally. This can allow for a greater degree of confidence from the sample size; especially for studies without real-time moderation it can be easier to obtain a larger sample size and find statistically significant differences. It might also enable teams to reach participants whom in-person access would have otherwise been difficult to gain, and to capture moments-based insights and longitudinal data over a period of time.

Further, the execution of the study (data collection) often costs less and takes less time. However, it is important to keep in mind that remote studies can be more complex, requiring more time for preparation and analysis. Expenses involved in remote studies – for example shipment of testing materials – tend to be overlooked or underestimated.

Finally, depending on how the study is organized, remote testing can often serve as a more flexible option for participants to take on their own time. (NOTE: The research team needs to be more flexible too!)

Moving Forward

In conclusion, it is important to think through all of the challenges, risks and benefits associated with remote testing and create a test and testing environment that work best for your participants and your project objectives. At Kaleidoscope Innovation, we consider all of these questions and more for remote studies. We are continuing to engage in Insights and Human Factors work and have been adjusting to the current climate by leaning on our remote testing and other expertise to serve clients in these uncertain times. If you’re looking for Insights and Human Factors support for your project, fill out our contact form or reach out to a member of our team.

Valerie Fenster
Director of Insights & Human Factors
vfenster@kascope.com

Matt Suits
Business Development Manager
msuits@kascope.com