WAVE Tri-Annual Survey
In order to measure if we are delivering on our mission at BUILT BY GIRLS, over the last 3 years I have developed and evolved a tri-annual survey. It launches 3 times a year to follow the 3 launches of matches for the mentorship program WAVE. As a mission-based organization, this data allows us to show our impact as it relates to our promises. Additionally, this constant feedback allows for us to identify user pain points and gaps in our offerings.
My Role
Coming from my position at the Girl Scouts of the USA, I knew how important showing the impact of mission based work could be. Using my previous experiences, I led the development of these outcomes and the items we then used to measure them. After my initial evaluation, the team then saw how powerful these metrics could be when trying to prove out our value and efficacy. I led this work from all aspects from ideation to creation to evaluation.
The Challenge
Since the WAVE program was not initially designed with learning outcomes in mind, working backwards provided a bit of a challenge. Creating these measures based on both marketing messages around mission and observed growth in our students necessitated nuance and trial and error to pinpoint. Another challenge was leading this work alone. Creating, launching, and analyzing this survey while completing other important research and design work called for lots of balance, but the results were well worth it. Finally, socializing this process and the results has taken diligence and time. Two years later and these survey results have now become a part of team goals where new programming is working to address our gaps.
The Process
1. Identify Outcomes
Based on my work with Girl Scouts of the USA, I not only knew how important providing metrics of success for mission based work was, but also how to develop tools that would measure these. To start this process at BUILT BY GIRLS, I identified the key outcomes based on not only the mission but goals of the WAVE program.
Mission:
BUILT BY GIRLS prepares the next generation of female and non-binary leaders to step boldly into careers powered by technology. Our community receives professional guidance and the practical tools needed to make their first career move.
WAVE Goals:
WAVE is a one-of-a-kind mentorship program that is designed to support female and non-binary students as they take the first step into their careers. Each Advisee receives personal support from a dedicated professional who will show you the career possibilities and help you develop the valuable skills needed to land your first job or internship.
Based on the mission and goals of the program, I originally identified the following key outcomes: exposure, skills, and network. Through participation in WAVE, our students will
Gain exposure to different careers and know more about different roles in tech
Learn and practice professional skills needed to gain and succeed in a job
Grow network of people who can support them in getting a job
These were initially measured through a series of likert scale and confidence measures:
5-point likert - Agree to Disagree
I know more about different careers.
I've learned about technical skills needed to do a job in tech.
I know how to grow my network of professionals
4-point confidence
Professional skills (i.e. emailing, presenting, interview skills)
Speaking with professionals
and one more…
Because of BUILT BY GIRLS, approximately how many more people do you now know who can support your educational or professional growth?
2. Evaluate
These measures proved to be a valuable to tool to help us measure if our program was effective. Using basic charts, crosstabs, and leaning on a statistical significance of p < .05, the findings from the first round of the survey demonstrated some key insights that were encouraging. They overall showed the efficacy of our program through high satisfaction, high engagement, and achievement of outcomes. They additionally highlighted key areas of growth. Based on the initial set of outcomes, below are some of the key findings from the report.
To see the full report click here.
3. Iterate
Since this initial run, this survey has been run 5 more times. Over the years I have iterated on the survey to better serve our changing program and needs. As our membership grows, as do the results of this survey as well. In the first run of the survey, my goal was to hit 300 respondents, but I fell short at 280. In my most recent launch of the survey I was at nearly 500 respondents with 498.
As for the structure of the survey, I have added more specific questions for our understanding of the outcomes achievement, while eliminating questions that overtime have proved less useful to our understanding of the holistic experience. Below are the most recent outcomes questions, with some old favorites holding their place in the list:
5-point likert - Agree to Disagree
I know more about different careers.
I've learned about skills needed to do a job in tech.
I know how to grow my network of professionals.
I know how to create a professional resume.
I know how to prepare for interviews.
4-point confidence
Professional skills (i.e. emailing, scheduling)
Applying to internships and jobs
Speaking with professionals
Selling or pitching yourself to a company or professional
2 new questions on community belonging
As a member of BUILT BY GIRLS, I feel like I am part of a community of like-minded students. (agree/disagree)
In the last year, approximately how often have you talked to other student members of the BUILT BY GIRLS community?
These new measures in addition to the old items allow us to track our progress over time while providing relevant insights into how our program is impacting our students.
Note: Along with these surveys to students, I have also launched surveys to their mentors - our professional volunteers to better understand their satisfaction with the experience.
4. Impact
The impact of this initial survey was far reaching. Some key findings that we have since taken action on are:
Diversify recruitment
While our student base is majority non-white, we found that we were overserving certain populations while underserving others. Based on this insight, we have created an ambassador program that helps us target different networks of students who could benefit from a program like ours. Additionally, we have worked to find networks at a variety of schools and partner organizations to build our base from the ground up.
Encourage session completion
While our goal was always 3 completed mentoring sessions, these findings validated this model that was created based on a hypothesis. Thanks to this insight, we continue to encourage 3 complete sessions, with the majority of our users hitting this benchmark.
Need to improve at professional skills and networking
We found that the WAVE program was hitting on our 3 outcome pillars, but that it wasn’t hitting as strongly with professional skills and networking. This insight led us to leverage our events and content as stronger sources of information on these topics. We were able to support what our students were learning in their sessions with supplemental experiences that we found improved on these outcomes.
Community belonging
The final key finding that drove change was around community belonging. This is initially a metric I was not measuring, but added into our surveys in 2020. We found that while 42.38% strongly agreed that “As a member of BUILT BY GIRLS, I feel like I am part of a community of like-minded students,” we want to boost feelings of community and connectedness. This led to a pilot in 2020 of a new program called the Peer Networking Pilot (PNP), where we leverage our same matching algorithm for WAVE to match students with each other. The pilot was highly successful and we have since moved on to productizing this new opportunity.
While this survey provides a constant touchpoint for our brand to check in with our users, a next step goal is to utilize all of the data collected over the years in a longitudinal study of the students in our program.