How to Build Intentional Surveys That Meet the Moment
~ 10 minute read
Assorted Lego® bricks. Photo credit: Mourizal Zativa
There’s another crisis. You’re leading a team of people who you care deeply about, want to understand how to support, but have little time and waning capacity to check-in with each of them. What you need is a structured approach that can help you quickly understand what they are facing, and help you meaningfully act. You’re thinking about a survey.
The scenario above is probably very familiar to you. Free online survey tools are now accessible to most people, and promise to give you quantitative data (so trustworthy!) with little time and effort (so efficient!). For those leading social change organizations and teams through challenging times, the allure of a quick survey to give you meaningful insights is powerful.
Take it from someone who has built some pretty good, and yes, pretty bad, surveys: Building a good survey takes work. But when done right, a good survey can serve as the bridge between where you are and where you need to be.
I’m not one for naming “best practices.” I’m an educator at heart, and years in the classroom have shown me that students differ, the world changes, and what is called “best” doesn’t work for everyone. If you’re looking to set out on a journey to be a survey professional, and see what some experts mean when they say survey best practices, there are excellent and authoritative guides for this.
But if you simply want to build a better survey to know what your people need now, here are four notes to help you on your way.
Simplify: Ask one question per question
Specify: List specific and evenly spaced response options
Collaborate: Involve others in the process
Connect: Align survey content with key organizational elements
Together, these notes should give you what you need to build intention into a survey that meets the moment.
Note 1: Simplify
A single question mark. Ask one question at a time. Photo credit: Towfiqu Barbhuiya
“How fair, supportive, and transparent is leadership’s response to this crisis?”
Whether it’s funding challenges, a toxic political environment, legal turmoil or all of the above, it’s understandable to want to know whether your response was appreciated. But some who find your actions “fair” won’t necessarily think them “supportive” or “transparent.” This is what’s called a compound question (also double-barreled question), which clumps together multiple concepts into a single item, making your question harder to answer and your results less interpretable.
One reason the error shows up is because we mistakenly assume favorable qualities go together: The survey version of the halo effect. Another reason for the error is simply wanting to save some space. After all, why ask three questions where one will do?
Sadly, that approach creates more problems than it solves. One good question about, in this case, “fairness,” is worth a lot more than one confusing question about three qualities that don’t necessarily go together. Don’t try to do everything at once.
Note 2: Specify
A person responding to the survey question: “How was your day?” Photo credit: Celpax
You’ve probably seen 5-point response options from “Strongly Disagree” to “Strongly Agree,” before. But the world of response options is vast.
Here are just a few of the ways to survey how supported your people feel during a crisis:
| Version 1 | I have all the support I need during this crisis. | ||||
| Strongly Disagree | Disagree | Neither Disagree Nor Agree | Agree | Strongly Agree | |
| Version 2 | How supported do you feel during this crisis? | ||||
| Not At All | Hardly / A Little | Moderately | Mostly | Completely | |
| Version 3 | Which of the following best describes how supported you feel during this crisis? | ||
| I have little to no support | I have some support, but not enough | I have the support that I need | |
Despite how common version 1 tends to be, research suggests that version 2 is clearer and easier to answer. After all, if I select “Disagree” in version 1, does that mean I have very little support, or that I just have less than “all the support I need?” Note how the response options in version 2 (i.e. “Not at all,” etc.) speak directly to the question, allowing people to provide a reasonable continuum of responses to it.
Another way of surveying support is version 3, which may be particularly appealing for those who want a more intuitive report-out of percentages. Saying “About 30% of staff felt moderately supported during this crisis,” simply doesn’t sound as intuitive as “About 30% of staff reported that they had some support, but not enough.”
Whatever version you go with, ensure that the response options are specific to the question and that they’re evenly spaced, conceptually. On the first point, don’t put the options from version 1 under the question for version 2 (Yes, I have seen this).
On the second point of even spacing, consider whether the difference between all the options is similar. Imagine the conceptual jump from “Hardly/Very Little” to “Completely” if the survey developer removed “Moderately” and “Mostly” from the set. It would be like asking, “How much do you like strawberries?” With the options: “I strongly dislike them,” “I dislike them, somewhat,” and “They are literally my favorite thing.” Personally, my response would be, “...is there another option?”
It’s important to know that while version 1 is extremely common, that doesn’t mean it’s always (or even often) what you want. Paradoxically, it may simply be popular because people see it so often. Psychologists refer to this bias to prefer the familiar as the mere exposure effect. It might be encouraging to know that interrupting the tendency towards the familiar may have some social benefit. So don’t be afraid to try something different.
Note 3: Collaborate
Assorted Lego® figures. The best surveys are built collaboratively. Photo credit: Kenny Eliason
Allow me to help dispel the myth of the lone expert. Every good survey I’ve ever made required feedback, consultation, and support from others. Your work will be more personally fulfilling and collectively meaningful if you involve others. That includes survey work.
Approaches for who to involve and how vary dramatically. They include professional research, focus groups and piloting, and more community-led participatory research stemming from the work of activist and educator Paulo Freire, and his work on popular education.
My note on who to involve is less a comprehensive “approach” and simply acknowledges that power dynamics and aspirational values in organizations occasionally compete. The most successful surveys I’ve been a part of have included:
People who have power: They can get your assessment moving, and may have structural power to initiate a meaningful response to the findings.
People who will listen: They will be willing to hear feedback that challenges their assumptions and change.
People who are impacted: They will be most affected by the issues in the survey, and will offer the most insightful feedback during survey development.
It is rare that an individual involved in the survey process has all three of these characteristics. This is actually the point. Collaboration is typically best when it involves a diverse group of people with varied responsibilities, skills, and identities. Don’t look for “the one” that can give you the perfect survey and save you from a crisis. They don’t exist.
Trust me when I say that building a survey in a vacuum will not lead to a better survey. Even a small, short-lived working group can have a lasting, positive impact on the survey and those involved.
Note 4: Connect
A row of four houses in sunlight. Organizations are like houses in some ways. Photo credit: Paul Kapischka
So, you have some guidance on the structure of questions, the structure of the team for building those questions, but what about the actual content? In my experience, the strongest surveys tend to connect two or more of the following organizational elements:
Foundation: Mission, vision and values
Structure: Roles and responsibilities
Community: Relationships and identities
Climate: Emerging needs and trends
You’ve developed and distributed a 10-question rapid response survey during a funding crisis. Your scrappy survey team wisely included questions about whether people feel supported during the crisis (Climate). But in your haste to get feedback, you and the team have missed critical elements. You didn’t include questions to identify which groups were most impacted (Community). There are no questions that can help clarify the kind of support needed and what resources and teams can meet that need (Structure). And the speed with which you rolled out the survey undermines some of your organization’s stated values (Foundation).
Think of these elements like aspects of a house. It’s certainly useful to know the effect of a changing climate on how livable the house is. But without connecting that information with an understanding of the foundation, the structure, and the people there, you’re limited how to address the key issue.
If it feels overwhelming to cover all four elements: Don’t worry. Not every survey is going to connect to all elements. In the example above, a rapid response survey for people in crisis shouldn’t be too long, as that would be needlessly overwhelming. Ten questions is plenty.
While you don’t need all four elements in every survey, try to include at least two. For a crisis response survey, you might focus on Climate and Community. In strategic planning, Foundation and Structure may take precedence. The key is to make intentional choices about which elements to prioritize based on your immediate needs and long-term goals.
Act Intentionally
Sometimes, you need a survey. Sometimes, you do not.
Despite my appreciation of surveys and assessments, they are merely tools that serve specific needs. Sometimes, the data you need is already in your systems. Sometimes, you don’t need any additional information. Sometimes yes, you do need to check-in with your people, 1-on-1, to know where they’re at.
Whatever you decide to do, I invite you to do so with intention. I have worked with people at all levels of organizations to build surveys. Everyone has the ability to think intentionally about the kinds of questions that need to be asked, and to decide accordingly. That last part is important, as we shouldn’t mistake inertia for intention. Intention is when the decision is rooted in clear reasons and values. Inertia is rooted in fear.
It can be difficult to think deeply about how you reach out to people when some are set on flooding you with crises. But give yourself the time to be intentional, especially when some are doing their best to take time from you. Especially when others are willing to share their time with you.
