Keeping It WEIRD


There was a great article over on The Conversation the other day about how poor scientific research is at considering people with different viewpoints and backgrounds.

Specifically, the vast majority of what we know about human psychology and behavior comes from studies conducted with a narrow slice of humanity – college students, middle-class respondents living near universities and highly educated residents of wealthy, industrialized and democratic nations.

To illustrate the extent of this bias, consider that more than 90 percent of studies recently published in psychological science’s flagship journal come from countries representing less than 15 percent of the world’s population….Given that these typical participants are often outliers, many scholars now describe them and the findings associated with them using the acronym WEIRD, for Western, educated, industrialized, rich and democratic.

First, I love the WEIRD acronym, and I’m a little surprised I haven’t heard it before, or, if I have, that it didn’t stick.

More seriously though, focusing only on the WEIRD can have a damaging impact as we use research to guide how we parent, how we teach and how we interact with others. The article gives an excellent example of how even a pretty widely accepted, and simple, pattern test can lead us astray.

Consider an apparently simple pattern recognition test commonly used to assess the cognitive abilities of children. A standard item consists of a sequence of two-dimensional shapes – squares, circles and triangles – with a missing space. A child is asked to complete the sequence by choosing the appropriate shape for the missing space.

When 2,711 Zambian schoolchildren completed this task in one recent study, only 12.5 percent correctly filled in more than half of shape sequences they were shown. But when the same task was given with familiar three-dimensional objects – things like toothpicks, stones, beans and beads – nearly three times as many children achieved this goal (34.9 percent). The task was aimed at recognizing patterns, not the ability to manipulate unfamiliar two-dimensional shapes. The use of a culturally foreign tool dramatically underestimated the abilities of these children.

Naturally, this made me think about the research that we have done thus far to push the web forward. Most of it is significantly less formal than those being conducted by, for example, the psychology community the author was focusing on. So there’s already likely a few more gaps and oversights built in. Now throw in the inherent bias in the results, and it’s a little frightening, isn’t it?

Moving beyond the WEIRD is critical not just in scientific research, but in our own more web-centric research. We’ve known for a while that the worldwide web was becoming increasingly that: worldwide. As we try to reach people in different parts of the globe with very different daily realities, we have to be willing to rethink our assumptions. We have to be willing to revisit our research and findings with fresh eyes so that we can see what holds true, what doesn’t, and where.

Just how much are we overlooking?

We have anecdotal evidence that the way we view forms and shipping is overly simplistic. What other assumptions do we make in the usability of form controls that may be leaving folks out? What does a truly globally accessible form look like?

We know that data costs can be prohibitive in many parts of the globe, leading folks to have to get creative with things like local caching servers to afford to get online. We’ve started to focus less on page weight in WEIRD environments, but is that true of folks in other areas? Do the performance metrics we’re zeroing in on still represent the user experience in different situations?

There’s been some work done on better understanding what people expect from the web; I certainly don’t want to imply that there hasn’t. But the body of research is significantly smaller than the analysis based on the WEIRD. Much of what we do have is survey-based (versus more accurate forms of research) and speculation based on anecdotal evidence. I don’t think anyone could argue that we don’t still have a long way to go.

There always seems to be something new for the web to figure out, something that keeps us on our toes. Robyn Larsen has been talking a lot lately about how internationalization is our next big challenge, and I couldn’t agree more. One thing is certain: we have a lot to relearn.