There’s a lot of thought that goes into conducting user testing. You have to strategize, plan, prep, and execute. We’ve already discussed how to determine a best fit test type and when to conduct throughout the process, but one question that remains is: Do I need to be there for it to be successful?
This post is the fifth in a series on user experience. We'll be covering each facet of user testing, with each discipline giving their take:
The answer is yes … and no.
If prepping for user testing wasn’t overwhelming enough, we’re going to introduce a few more cards into the mix. The question, “Do I need to be there?” is two-fold. There meaning both physically and (for lack of a better word) emotionally. We’re talking about remote vs. in-person tests (physically), and moderated vs. un-moderated tests (emotionally).
And, if that’s not complicated enough, you can combine approaches—moderated, remote testing; moderated, in-person testing; un-moderated, remote testing and un-moderated, in-person testing. For simplicity and sanity’s sake, let’s look at each individually.
Remote vs. In-Person Testing
Do you want to be in the same room as your participants or are you okay with conducting tests virtually? The terms are pretty self-explanatory, but for clarity: remote testing is virtual and in-person testing is just that, in-person.
Remote testing is done via the internet. There are several viable tools for conducting remote testing. We’ve grown to love UserTesting.com (though it’s best for remote, un-moderated tests, but I digress). As with everything in life, there are benefits and drawbacks to conducting remote testing.
Benefits: It’s cost-effective and you have great access to your target audience. You don’t have to rely on participants’ location to qualify, and you don’t have to find a location for conduction. There’s a lot more flexibility from a scheduling stand-point, as well.
Drawbacks: The only drawback we’ve found to remote testing is that you don’t get to analyze body language. Which, believe it or not, is incredibly telling. Most tools will record screen sessions and some include and record video chat if you’re moderating remotely, but there is a sense of a barrier when you put a screen between two people.
In-person testing is when you are physically in the same room as your participants. While there are several factors that come into play (moderated, un-moderated, location, etc.), it’s pretty straightforward—you’re there with your participants.
Benefits: The biggest upside to in-person testing is that you get to analyze and experience participants’ body language. You can observe how their facial expressions correlate with their actions and words; i.e. if they tense up in frustration but don’t voice it. There really is immeasurable value in witnessing the nonverbal communication.
Drawbacks: It can get expensive. There’s time involved to either moderate or observe each individual test, you need to find a location (which can mean renting space), and recruiting your target audience can become laborious with location limitations.
Do you want to guide users through the process or just observe? Conducting a user test can be as simple as setting up the test, drafting the tasks and letting users take the reigns. Or it can be a bit more involved where you guide them through the test, asking clarifying questions, having conversations around their confusion, etc. These approaches are called un-moderated and moderated tests, respectively. And both have their benefits and drawbacks.
Benefits: Moderated testing allows for more flexibility. You can plan and prep all you want but, if we’re being honest, there’s really no way to account for all possible variances. Did the participant interpret the question/task appropriately? Did the participant explain their thought process clearly? If a participant gets off-task, you’re able to help guide them back. If they’re experiencing frustration, you’re able to ask questions to get to the root of the problem.
Moderated tests allow for conversation, which can lead to invaluable insights that wouldn’t have been found otherwise.
Drawbacks: There are some drawbacks, though. One being introduction of bias. Conducting moderated tests can lead to skewed results if you don’t have someone who is able to conduct your tests without asking leading questions or introducing bias. Documentation is also an issue. Taking thorough notes while carrying on a conversation is nearly impossible. And without thorough notes, it’s difficult to paint the whole picture. A remedy would be to record the session—but that’s dependant on access to proper equipment and permission of the participant.
Benefits: Un-moderated testing typically allows for quicker, more cost-effective results. You set up your test and let participants complete it; there’s no need to hire a moderator to guide participants through the tasks. You’re also able to guarantee omission of bias and leading questions because you can vet your questions/tasks prior to testing.
Drawbacks: Lack of guidance for the participant and you are reliant on them being clear with their feedback and thoughts. You aren’t able to answer their questions and aren’t able to ask any of your own.
There’s really no right answer when it comes to choosing between moderated or un-moderated tests, and whether to do it in-person or remotely. All options have their benefits and drawbacks—you just have to prioritize your client’s needs, account for time and budget, and go from there.