Blog

Designing a Mobile-Centric User Research Lab

Why we ditched the mirrored wall

With Punchkick’s impending move to a new office space, one important thing that needed to be considered was the re-design of our UX research lab. In the early stages of planning, it was considered a given that the traditional 1-way mirror style lab would be retained in the new space. Upon further research, feedback, and advice it was decided that the mirror would be replaced by a more technologically advanced option consisting of a series of digital cameras that will feed video and audio to a dedicated observation room specifically designed to facilitate creative collaboration. Moreover, utilizing a system of cameras as opposed to the 1-way mirror allows Punchkick to transmit high-quality footage to remote locations in real time. The remainder of this article will explain why our decision ultimately led to a more user and observer-centric approach to UX research, and also why V2 of the lab provides the kind of rich and mobile-medium-specific data that drives Punchkick’s user-centered approach to design, development, and UX.

The research

When one closes his or her eyes and thinks of a market research lab what do they see? Most likely it’s something to the effect of two adjoined rooms that are separated by a large mirrored-glass wall. Ironically, the image that manifests is also eerily akin to a police interrogation room. You know, the rooms where investigators make suspects sweat it out for hours in an effort to gain a psychological advantage? Although the intended use of these rooms is extremely different, the feeling that is instilled in their inhabitants is extremely similar. Some words that come to mind are: intimidating, distracting, nerve-racking, and uncomfortable.

Unfortunately, when participants are invited in to provide the valuable end-user feedback that drives design and development iterations, the abovementioned descriptors are precisely what researchers are hoping to avoid. With this information in mind, why is it that the mirrored wall is still an industry standard? Why is it that during an era of high-definition recording and viewing capabilities does tech-centered research continue to rely on archaic, and bias-inducing methods of observation?

Why is it that during an era of high-definition recording and viewing capabilities does tech-centered research continue to rely on archaic, and bias-inducing methods of observation?

The answer: because it’s way things have always been done. The aforementioned response is an example of a logical fallacy known as an appeal to tradition. This type of fallacious reasoning goes as follows: “We have been doing X for decades, therefore it’s incorrect to do anything other than X.” In numerous scenarios this kind of reasoning is problematic, but even more so in the tech industry where failure to evolve ultimately leads to extinction. People loved their VHS players for a long time, but does that mean they made a horrible mistake switching over to DVD? And now that streaming has become an efficient method of media consumption, was it equally erroneous to ditch the DVD player in favor of a Netflix account?

Before moving on, lets take a step back in time to better understand why the 1-way mirror lab became an industry standard in the first place. In the mid 1980s, when observational labs began to gain traction in the market research industry, video cameras were expensive and produced low quality renderings of the images that were captured. The 1-way observation mirror, at that time, was a lower priced option that facilitated higher quality results. Rather then squint at a small, low resolution, square monitor, the 1-way mirror was simply a better and less expensive method of observation. Fast-forward to 2016 and the disadvantages listed above are no longer an issue. Video cameras are now able to record and stream high-definition footage, monitors now produce images at up to 4k resolution (a level of clarity over 6x superior to monitors from the 1980s). Moreover, in the current world of UX research, video and audio recordings are an expected deliverable, therefore the majority of research labs not only have some form of large 1-way mirror wall, but also contain AV capture capabilities. In sum, contemporary research labs often include observational redundancies, in some cases, not for utilitarian purposes but rather as an unnecessary nod to tradition. In a landscape of constantly evolving and shifting paradigms, why is it that one of the most important tools in the tech-advancement arsenal (the user research lab) still looks like it belongs in an episode of Mad Men?

When redesigning the Punchkick research lab we took a similar approach to the way we design and develop products: by allowing the user to inform our decisions. Similar to the way Airbnb has to consider multiple types of users, all with disparate motivations and goals (i.e. the renter and rentee), it quickly became apparent that, for Punchkick’s lab, two very different types users had to be considered: the participant, and the internal team.

When redesigning the Punchkick research lab we took a similar approach to the way we design and develop products: by allowing the user to inform our decisions.

The participant

Although the level of impact varies depending on the individual, the presence of a large mirrored wall has some level of influence on nearly every participant that walks into a lab. The 1-way mirror, in two ways, affects the emotional states and behaviors of individuals; the first has to do with the prospect of being observed and the second with observing their own reflection in the mirror itself.

The potential presence of unknown spectators creates something known as the observer effect. In short, when people know that they are being watched their emotional state and behavior may change. One of the main criticisms against lab research in general is that the lab represents the antithesis of a naturalistic setting. A giant mirrored wall, obviously intended to hide unknown observers, adds an additional level of sterility to the setting and increases feelings of scrutiny within the participant. Often, participants become nervous, fixated on the mirror, frustrated, overly positive, or extremely embarrassed if they can’t quite figure out how to get through a given flow. Rather than focusing on the task at hand, participants begin to focus on the fact that they are being watched.

Secondly, the presence of a reflective surface influences participants’ internal feelings of self-awareness. Rather than giving uninhibited feedback, participants become overly focused on how they look and how they may be perceived if the feedback they give is negative. As a result of cues in the room (e.g. an entire wall made of mirrors) participants’ attention shifts to focus on internal elements rather than directing their attention to external elements, say for example interacting with a digital product.

By and large, individuals are somewhat nervous simply from being in a place they have never been before. When they enter the lab and the first thing they see is an entire wall composed of 1-way observation mirrors, that feeling of nervousness is often amplified. Although moderators do their best to build rapport with the participant, it’s impossible for that participant to build rapport with the unknown voyeurs in the next room. This kind of uncertainty has the potential to undo all of the effort that the moderator has put into making the participant feel at ease, comfortable, and willing to share their honest feelings. No matter how much reassurance is given that the mirror is nothing to worry about, participants often fall victim to something called the White Bear Principle. That is, deliberate efforts to quell certain thoughts make them more likely to surface. For example, if one is told not to think about a white bear, often a white bear is exactly what appears in their mind. In sum, the only way to truly alleviate the biasing effect that comes with a giant observation mirror is to get rid of the mirror all together, so that is exactly what we did.

No matter how much reassurance is given that the mirror is nothing to worry about, participants often fall victim to something called the White Bear Principle.

The team

For those who have never experienced it, working in a 1-way mirror-style research lab is an interesting endeavor. While user-testing sessions are in progress the observation side of the lab is dark, stuffy, and silent – overall not an ideal setting for creative collaboration. Something as simple as leaving the room to use the bathroom becomes a test in stealthy-ness as any light that enters the observation room from a hallway transforms the once opaque mirrored wall into a transparent surface. Think about how the already somewhat nervous participant feels when one of the walls turns into a window complete with spectators staring them down from the other side.

Moreover, because 1-way mirrors are not well suited for protecting against sound leakage, observers are required to sit in silence while UX testing sessions are going on. Aside from the discomfort that comes along with holding in a sneeze, any and all ideas that arise while sessions are happening must be held off until the session is complete. The problem with latency between collaborative discussions is that great ideas can slip away even when copious notes are taken.

Although a sneeze or cough may create a momentary disruption, the conversations that occur during product ideation discussions are the most problematic distractions. Research from the 1950s, 60s, 70s, and 80s has found that intelligent speech is the most distracting noise. As a result of evolution, the frequency of human speech falls into where sensory receptors in the ear are at their most sensitive. Although noise in general degrades performance, speech seems to affect it the most. The brain involuntarily picks up social or interesting speech cues and the subject is forced to devote some of their cognitive ability to understanding what is being said. Cognitive distractions ultimately detract from the quality of insights that the researcher is able to accrue throughout the testing process. Actionable insights are precisely what drive design and development here at Punchkick, the ability to openly discuss those insights as they arise is an invaluable process that, in the previous lab, internal teams could not take advantage of.

Actionable insights are precisely what drive design and development here at Punchkick, the ability to openly discuss those insights as they arise is an invaluable process that, in the previous lab, internal teams could not take advantage of.

_

The biggest perceived advantage of the traditional mirrored glass research lab is that, in the case of observing a focus group, individuals behind the glass are able to see the facial expressions of each member of the group being observed. Firstly, focus groups are a terrible way to do UX research. Even the creator of focus groups, American Sociologist Robert Merton, deplored the way his methodology ultimately came to be misused and ineffective in the pursuit of understanding behavior. There’s a reason why Apple almost never uses focus groups as a research method. The second advantage of a 1-way mirror wall is to observe participants performing tasks that require a great deal of movement in a large space. For example, General Electric utilizes a 1-way mirror set up to observe participant behaviors in a life-sized kitchen, not exactly something at the forefront of Punchkick’s consciousness. What Punchkick is interested in, however, is the way participants touch, interact with, and respond to the products we create. The new system of adjustable cameras will allow the Punchkick research team to focus on the observable elements that really matter in mobile product development: the participants’ fingers, facial expressions, body language, and screen.

In V2 of the Punchkick UX lab, observers (clients included) are able to come and go as needed, sit in a well-lit room and, most importantly, openly discuss ideas as they arise. Participants are no longer be forced into a fishbowl like setting, which in turn, increases the overall quality of the invaluable insights that they provide. Moreover, the new lab facilitates high quality options for remote observers and stakeholders. Although UX research is a relatively new field of inquiry, it is never too soon to evaluate how to improve the way data are gathered and the elements that impact the quality of that data. Rather than trying to keep up, Punchkick is looking ahead and planning accordingly.