As designers and analysts, we often overlook qualitative data in favor of easily measured metrics. We focus on things things like money spent, products sold, click-through rates and logins because numbers are easy, and dashboards are sexy. But if our goal is to produce products or programs that impact lives, we might be focusing on the wrong things.

Making Design Matter

This was the topic of the Design Week Portland panel discussion, “Making Design Matter.” The panelists—members of the academic, private and non-profit sectors—came prepared to discuss the benefits and difficulties involved with maintaining a human-centered design approach. “Human-centered design” is just a fancy way of saying that when solving a problem, the human perspective is considered every step of the way. No matter what sector you work in, at the end of the day, most of us are designing solutions to solve human problems.

The panel sought to answer the following questions:

• How do we know that what we design is working?
• When do we measure our impact?
• How do we even begin to do so?
• And who decides what matters?
  

Panelists talked about the ways their own projects sought to address each of these complex questions, including what approaches have been problematic and why. The main takeaway was this: The methods we use to collect data and the kinds of data we collect have a huge impact on the demographics we serve. Panelist Mary Li, Director of Multnomah Idea Lab, cited the census as a prime example. Many turn to census data because it is, on its surface, pretty straightforward. The Census Bureau issues surveys to all verified US addresses. Responses are collected. As a result, we’re left with a representative snapshot of US demographics. Pretty simple, right?

Not quite.

Quantitative Data Isn’t Perfect

The census is a perfect example of how quantitative data falls short. While all residents—regardless of citizenship status—are legally obligated to participate in the census, many remain uncounted or miscounted.

The reasons for this vary. The enumerator –the person in the field collecting data –might not speak the language of the respondent. The respondent might not have all the information about the family’s history. Or the respondent might simply distrust the enumerator and purposely give inaccurate information. There are those whose addresses cannot be verified because they are houseless or their home is mobile. Then, of course, there’s the matter of simple human error. Enumerators make mistakes.

The Significance of Qualitative Data

This is where qualitative data comes in. Qualitative data allows us to understand the “why” behind seemingly simple quantitative data points. It allows us, for example, to determine why respondents might distrust census workers, and as a result, measures can be taken to mitigate respondents’ concerns. Qualitative data can not only provide us with valuable insights on the collection and analysis of quantitative data, but it can also remind us why we’re working on a given project and help us maintain a human-centered approach.

In the case of the census, understanding why the data could be faulty is critical. According to census.gov, data collected during a census are used to allocate more than $675 billion per year in federal funds. That money is spent on things like schools, hospitals, roads, social-welfare programs and public works. This means that entire communities could be unserved or underserved for a decade because we got the numbers wrong.

But let’s reframe this from a human-centered approach. A decade is enough time for a generation to be born and grow up without equal access to food or early education. A decade is enough time for thousands to die as a result of unsafe roads. It’s also enough time for thousands of factories, offices and stores to open or close and for hundreds of thousands of jobs to either be created or lost.

Does this really apply to my for-profit business?

Qualitative data can not only provide us with valuable insights on the collection and analysis of quantitative data, but it can also remind us why we’re working on a given project and help us maintain a human-centered approach.

Applying a Human-Centered Approach

Most of us aren’t working on projects with implications as far reaching as the US Census. But as we’ve already discussed, most of us are working on human problems with human impacts, so ensuring those humans are considered is just good for business. By collecting a deeper well of data and by focusing on the “why,” we can identify the social ROI, build better products and identify opportunities to scale. This translates to happy, loyal customers and expanding business opportunities.

The demand for human-centered design isn’t limited to social enterprises. In 2018, as a response to increasing concern from stakeholders, workers and customers, transportation giants Diamler, Ford, Toyota, Lyft, Uber, Waymo and Fed-Ex all joined forces to form the Partnership for Transportation Innovation and Opportunity (PTIO), a non-profit aimed at studying the impact that driverless vehicles will have on humans. Tech giants Apple, Facebook and Amazon also have dedicated social or environmental impact teams.

According to Forbes, more investors are demanding proof of impact. Governments, foundations and non-profits are increasingly being asked to measure and report social influence. But to return to our panel questions: How can we be sure we’re having our desired impact? And how do we measure that impact?

Panelist Jacen Green, Director of Impact Entrepreneurs, says that in order to evaluate the degree to which lives have been impacted by our products, we need need to know what our goal is from the outset. That’s where Impact Measurement comes in.

The Role of Impact Measurement

According to sopact.com, “Impact measurement is a process of understanding how much social change occurred and can be attributed to an organization’s activities.” It calls for organizations to define their goals in a publicly accessible mission statement against which all activity can be judged. It asserts that by starting off with the understanding of what change we’d like to create and why that change is important, we can ensure that every decision made, every study that’s designed and all data collected reinforce the overarching mission.

Green acknowledges that while this seems straightforward, it often leaves us in uncharted territory. That’s because in order to measure behavior changes or social returns, we need to delve into the world of qualitative data. And that’s a place many of us fear.

Never fear!

As a designer, I think of myself as a mediator. When designing an application, I aim to know what the user wants so I can give it to them. But I also need to know what the techies want and, most importantly, what the goal of the application is; I need to understand our why. This means I do a lot of interviews and hold many more informal water-cooler focus groups. My job thrives in the messy and revealing world of qualitative data. It allows me to help both the users and techies get their needs met, all while keeping us on track to create that next stellar product that will revolutionize the way we live.

Consider Real People When You Design

At the end of the day, I’m trying to change behavior, and qualitative data shines when trying to affect behavior or social change. It allows you to push past the confines of numbers or the binaries of profits and losses, “yeses” and “nos.”  At its root, qualitative research is as simple as asking our users/participants/clients to give us their perspective. It understands that, like humans, the problems we face often contain a great deal of nuance. It requires that we temporarily relinquish control and let the people we serve lead the way.

Green asserts that you can measure anything. “You may not know how to measure it,” he says, “but there’s a way.” And I agree. Figuring it out is half the fun. So where do we go from here? How do we ensure that our methods are human-centered? Take the Impact Measurement approach:

1. 1. Start with your why. Ask yourself, “What change do we want to create? And why does it matter?” Use your answer to define your mission statement.
   2. Talk with your users/customers/participants. Find out what change is already happening and whether you’re responsible. Ask open-ended questions. Let them rant. Give up the reins for a second and ask them how they’d solve the problem.
   3. Determine how you can do better. Look at the common themes from your users’ responses. Considering what they said, what’s the next step to achieving your organization’s goal?

Remember: no matter what problem you’re trying to solve, it’s most likely a human problem with human impacts. So keep the humans in mind.