The Bias in What We Build

I’ve been thinking a lot lately about our biases and their influence on what we build and how. We’re all biased in some way—it’s an inevitable side-effect of living. We experience certain things, we live in a certain environment, we have certain interactions and over time all of these experiences and factors add up to impact the way we view ourselves and the way we view others.

These biases come into play over and over again in our work, and can have devastating consequences.

There was an interesting post on The Coral Project about anonymity and its impact—or rather, non-impact—on online behavior. A frequent refrain heard when we try to understand why online behavior is so frequently so poor is that the ability to be anonymous is one of the primary reasons for the problem. J. Nathan Matias argues differently, though:

Not only would removing anonymity fail to consistently improve online community behavior – forcing real names in online communities could also increase discrimination and worsen harassment.

We need to change our entire approach to the question. Our concerns about anonymity are overly-simplistic; system design can’t solve social problems without actual social change.

While the article cites a little bit of research questioning our assumptions about anonymity online, the bulk of the article is focused on reframing our perspective of the discussion. We often consider the question of bad behavior online from the perspective of the people misbehaving. What is it that makes them feel free to be so much more vindictive in an online setting? Matias instead builds his case by focusing on the victims of this behavior.

Revealing personal information exposes people to greater levels of harassment and discrimination. While there is no conclusive evidence that displaying names and identities will reliably reduce social problems, many studies have documented the problems it creates. When people’s names and photos are shown on a platform, people who provide a service to them – drivers, hosts, buyers – reject transactions from people of color and charge them more. Revealing marital status on DonorsChoose caused donors give less to students with women teachers, in fields where women were a minority. Gender- and race-based harassment are only possible if people know a person’s gender and/or race, and real names often give strong indications around both of these categories. Requiring people to disclose that information forces those risks upon them.

He also points out that pseudonymity can be an important protective measure for harassment victims:

According to a US nationally-representative report by the Data & Society Institute, 43% of online harassment victims have changed their contact information and 26% disconnected from online networks or devices to protect themselves. When people do withdraw, they are often disconnected from the networks of support they need to survive harassment. Pseudonymity is a common protective measure. One study on the reddit platform found that women, who are more likely to receive harassment, also use multiple pseudonymous identities at greater rates than men.

In other words, one thing we can state about removing anonymity is that it increases the risk for people on the receiving end of online harassment.

Removing anonymity online, then, is yet another example of how we reflect our own biases in the decisions we make and the things we build.

It is our biases that lead us to overlook accessibility or how an application performs on a low-powered device or spotty network.

It is our biases that lead us to develop algorithms that struggle to recognize women’s voices or show more high-paying executive jobs to men than women.

And it is our biases that lead us to frame the problem of online behavior from that of the attacker. If we’ve never experienced persistent harassment, and if we don’t stop to talk to those who have, we end up with solutions that are dangerous for the people on the receiving end of that harassment—the ones who need protection the most.

In each of these situations, our biases don’t just lead us to build something that is hard to use; they cause us to actively, if unintentionally, exclude entire groups of people.

The new year is just behind us which means this is a time where a lot of people are reflecting on the past year, and on the things they want to improve on in the year to come. There are many worthy goals to pursue. There may be, however, no more important investment we can make in ourselves than to actively seek to broaden our perspective.