Survey: Satisfaction Data#
Introduction#
Open Source (OS) projects do not often have a standard mechanism for collecting feedback from their communities. Gathering data is the best method to test assumptions and objectively measure how satisfied the community is. Satisfaction can be measured from two perspectives; the people who contribute and those who use the tool.
In September 2023 a group of OS contributors gathered in person at the DISC Unconference event. This guide represents a continuation of that work done by a subset of those gathered who were keen on metrics and data. We invite you to gather a new perspective and collect data with the goal of measuring satisfaction trends as a measure of the health of your project.
What is Net Promoter Score (NPS)? What is the goal of measuring NPS? [needs additional context]
What can be done with results?
Identify the reasons why your contributors and users are detractors of your open source project.
Set goals that will increase the number of promoters of your project.
Compare year over the year NPS, create dashboard (pending - can we create a dashboard)
Data privacy The privacy of respondents is important and should be protected. Do not ask questions that may lead to respondents being identified. This is especially important where the community or sample size is small.
Community: Targeted Respondents#
Contributors#
People who contribute to an OS project. CONTRIBUTIONS ARE NOT LIMITED TO CODE. Contributors can also be users but for the purposes of this survey we are asking them to evaluate their experience as Contributors.
Code contributor - a person who opens a pull request
Organizers/speakers for events, meetings, conferences
Writing, editing and updating documentation
UI or UX Design
Testing
Operations activities that move forward the goals of a project, such as but not limited to: project management, social media administration, communications administration, fundraising, governance
Users#
People who use software but do not contribute to it. Has used the software to perform a task in the past 12 months.
Documentation: How to conduct an NPS Survey#
Survey Checklist#
Familiarize yourself with NPS, read documentation
Socialize the idea of using NPS among your project’s leadership to secure support or consensus for your idea including a project plan for open and close dates for the survey and a process for analyzing data
Create a survey in any software you prefer - see the questions below
Define collection methods and where you will promote the survey
Invite your community to answer
Once you have data in hand you will need to spend some time analyzing and summarizing results
Share results with your community
What is Net Promoter Score?#
The NPS assumes a subdivision of respondents into “promoters” who provide ratings of 9 or 10, “passives” who provide ratings of 7 or 8, and “detractors” who provide ratings of 6 or lower. The net promoter score results from a calculation that involves subtracting the percentage of detractors from the percentage of promoters collected by the survey item. The result of the calculation is typically expressed as an integer rather than a percentage.
Contributor Survey Questions#
Are your contributions to Project X code, non code, both?
Hypothesis - NPS for those who contribute both is higher and we can use that as a way to demonstrate the value of encouraging engineers to make non code contributions.
On a scale of 0 to 10, how likely is it that you would recommend contributing to project x to a friend or colleague?
Please tell us more about why you provided this rating.
User Survey Questions#
On a scale of 0 to 10, On a scale of 0 to 10, how likely is it that you would recommend using project x to a friend or colleague?
Please tell us more about why you provided this rating.
How to analyze results of your survey#
Calculate NPS (pending)
Manually evaluate open ended comments
Compare segments (for Contributors survey only)
Challenges#
Collecting a representative sample. The size of an OS community can be hard to quantify therefore the percentage of response will be hard to quantify. How can we define what the target number of respondents is?
Survey should be conducted once a year. There is no benchmark for NPS in Open Source (that we’re aware of) Therefore the value that we propose is in comparing year over year satisfaction data. This means the survey will need to be conducted more than once in order to be valuable.
Project Leaders + Invitation to participate#
Anavelyz Perez (Open Science Lab) and Ana Ruvalcaba (Project Jupyter) lead on this work and plan to continue collaborating and sharing their experiences with any pilot programs which may be hosted. Perez and Ruvalcaba will work together to continue refining the documentation.
If you have conducted an NPS survey in any context or you would like to join us please open an issue.