The Myth that “More Data is Always Better”
~ 10 minute read
Photo credit: Christy Joseph Jacob
This is the third in a series of myth-busting posts on data and assessment. See also the first on the myth of bias-free data and the second on the myth of data objectivity.
I’m addicted to my phone. You might be, too. When I’m away from it, there’s this sense that I’m missing something. If I haven’t checked the news, I feel I’ve not kept up with issues I care about. And when I feel distraught or unsure of what to do, the phone offers either the facsimile of meaningful activity or endless distraction.
We are addicted to data. This is true as much at the institutional level as at the individual level.
But while individuals understand that such addiction is a problem, many organizations are under the impression that the more data, the better.
In Nexus, historian and writer Yuval Noah Harari terms the idea that more information will invariably lead to better outcomes the naïve view of information. History, he says, tells a different story. Information is often biased, misleading, and wrong. And even when the information—or data—is good, that doesn’t mean leaders wield it for the benefit of all.
If you’re skeptical about whether “naivety” will impact the nonprofit sector, consider that last year, over 90% of 1,440 nonprofits surveyed said that they believed AI would enhance their engagement with end users. But will access to technology trained on a massive corpus of data of likely varying quality actually serve mission-driven work?
What happens when organizations fall into the trap of thinking more data is always better? How can they avoid it?
The Danger of Data-First Disruption
Sherman is the new Chief People Officer of his nonprofit. His initiatives are pivotal to the new Executive Director's vision of building a Big Tech-like, data-first culture in his first year.
Under pressure to build that culture rapidly, in the past year, Sherman has made some impressive shifts in how the organization does its people work. Those shifts include weekly and required pulse surveys for all staff, AI-enhanced monitoring software on all computational devices, and predictive analytics to track performance drop-offs and support employment decisions.
Though it was a challenging decision, the measures helped inform downscaling in People Partner capacity. With all the automated HR services the new initiatives can provide, Sherman could not justify having so many staff whose individual check-ins and support were now redundant with their current systems.
But in the last year, something has changed in the culture of the organization. Everyone, Sherman included, can feel it. The tone and cadence of meetings feel stilted. Conversations between management and others are more guarded.
There’s nothing in the pulse surveys hinting at problems. Staff are more productive than ever, and according to the most recent analytics dashboard refresh, the highest-performing staff continue to stay with the organization.
Another year passes, and there’s a steady drop-off in funding and staffing, accompanied by whispers of executive turmoil. What happened?
Good Data and Cultures Take Time
Sherman’s people troubles are partly of his own making and partly based on his ED’s misconception of how data works.
In some ways, it’s admirable to believe in the power of data.
But without a clear idea of what good data is, we risk basing our decisions off “garbage data.”
The risk is so well known that it has its own aphorism: “Garbage in, garbage out” or GIGO. In fact, places like The Data Nutrition Project are actively working to bring data quality issues to the fore as institutions increasingly use AI to inform what they do.
In Sherman and his ED’s case, without taking the time to understand the organization or what makes its culture unique, this likely led to:
Pulse surveys that didn’t fully capture engagement and increased feedback fatigue
Monitoring software that reinforced a culture of surveillance
Analytics built on incorrect measures of performance
There’s nothing inherently wrong with surveys and analytics, but doing each in line with the values of the organization takes time. Monitoring software, on the other hand, has little positive effect on values-driven organizational cultures, regardless of the circumstance. A 2024 study by the US Government Accountability Office found that digital surveillance at work worsens mental health, discourages unionization, and may lead to discrimination.
Perhaps worse, valuing quantitative data over qualitative means devaluing relational data. In our example, the People Partner capacity that Sherman dismissed as redundant would have provided non-redundant, high-quality data about what was wrong with the new initiatives. Sometimes, the only way to capture good data is through honest conversation.
Purpose-Driven Data Cultures
Data cultures in organizations take many forms. They vary in how much they center the experiences of people, and how much they prioritize simply having data. They also differ in how much time, effort, and resources they dedicate to building their data cultures.
Four Quadrants of Organizational Data Cultures
As the graphic above shows:
Purpose-Driven cultures prioritize the experiences of people, understanding that doing so requires a balance of qualitative and quantitative data, and acknowledge the investment needed for good data. Knowing that change is constant, these organizations continually review both their data infrastructures for effectiveness and organizational sentiment around the data work, adjusting as needed based on both internal and external factors.
Relationally Reliant data cultures similarly prioritize relationships, but either may not recognize the amount of work needed or may not have the capacity to fully invest in the work. There may be an impression that quantitative people data will fundamentally depersonalize work. While relying on relationships can be advantageous for smaller organizations, as organizations grow, a lack of structured data may hide biased decision-making or lead to inconsistencies in processes.
Productivity-Minded cultures are less focused on people and relationships, and more on non-relational work outcomes. They tend to understand the power of data, how much is needed to invest in good data, and even what data bias constitutes. These organizations can be both high-performing and diverse. However, a lack of attention to the well-being of their people can lead to data-based initiatives (like employee monitoring and surveillance) that contribute to burnout, attrition, and even discrimination.
Convenience-Oriented data cultures are less interested in supporting their people, and while they may be excited by the possibilities of a data-first culture, a lack of understanding of what “good data” means prevents them from properly investing in initiatives. They may be looking to build engaging visualizations and compelling data-driven programs with limited funds and excessive time constraints. Without good data, leaders may be engaging in flawed decision-making practices that ultimately undermine the trust of their staff and partners. In more cynical cases, decision-makers may care little for the quality of data, as long as it can be interpreted to justify their actions (i.e. convenient for their purposes).
These data cultures exist along a spectrum, and few organizations will perfectly match a single type. In Sherman’s case, his organization falls somewhere between the convenience-oriented and productivity-minded types. While he appears to have invested resources in improving productivity monitoring (Productivity-Minded), the lack of time spent in revisiting performance evaluation criteria and how they may yield inaccurate data suggests a basic lack of understanding of what “good data” is and an implicit belief that “what we have is good enough” (Convenience-Oriented).
The Danger of Inaccurate Performance Ratings
The allure of more data to give us actionable information, quickly and at scale, can have very real negative impacts.
In her 2017 book, Weapons of Math Destruction, mathematician Cathy O’Neil describes how relying on algorithms may have led to the firing of good teachers. In 2009, the chancellor of the Washington D.C. school system, Michelle Rhee, implemented a new teacher assessment tool called IMPACT that heavily swayed decisions about teacher dismissal. Sarah Wysocki, a fifth-grade teacher, had received excellent reviews from parents and her principal, but when the new IMPACT tool gave her a low score, she was one of 206 teachers to be dismissed.
Since then, IMPACT has undergone numerous revisions, but like many teacher evaluations, it continues to be controversial. For example, in 2021, D.C. public schools found that IMPACT was racially biased against Black and Hispanic teachers, who more often work at schools with more students below the poverty line.
Like Sherman's case, D.C.’s IMPACT tool may be a performance rating divorced from actual performance.
Being purpose-driven means reminding yourself what you’re here to do as an institution. If you’re a school system, it’s imperative that you remember that your goal is to provide a good education, not to obey a new data tool. If your tool is dismissing good educators, then it may be undermining what you’re hoping to accomplish. You’re prioritizing the directives of the data over the needs of people.
5 Guiding Questions
To better understand where you are on the path to a purpose-driven data culture, consider asking yourself the following:
Support: Does my team/organization understand the value of data for informing action?
Capacity: Do we have the time and funding needed to build good data infrastructure and processes over the long term?
Values: Do we have a clear understanding of what matters and what good data looks like?
Awareness: Does leadership understand how challenging it is to acquire good data and the costs of acting on bad data?
Feedback: Are we prepared to learn and change our processes over time?
Every organization is different, and like Sherman, it’s possible you’ll have areas where your data culture aligns with multiple quadrants. There isn’t a one-to-one relationship between sets of questions and any one quadrant. But these questions can help guide you towards a clearer view of what you need to address to move further towards a purpose-driven data culture.
Of course, knowing what we need to work on is only half the battle. Most of us know exactly how harmful social media is to our mental health, and yet pulling ourselves away from it can be a challenge. You may well know that you’re not investing the time needed for truly purposeful data work, but finding the time to address that is another matter.
Still, knowing where we are can give us a better idea of where we might be if we lose focus, and where we can be if we take the time.
If you’re looking for support in your values-driven, data-informed journey or to build capacity in data fluency, let us know. You can also review our services page for more info.
