In this post, we will provide 5 examples of how Creating Chances measures impact on the social and emotional wellbeing of young people, and why measuring what we do allows us to build a sound business model whilst also achieving positive outcomes for the thousands of young people we work with.
Why is Impact Important?
Rachel Baffsky, Creating Chances’ Head of Research, works to unpack and understand the complexities of young people and the impact that our positive youth development pathway programs have on them.
“In understanding real impact, it is very important for social enterprises to have a strong evidence base to guide all program activities because this is the only way to ensure that they’re actually meaningfully achieving what they are intending to achieve.”
Measuring the impact of what you are trying to achieve can clarify the focus of the organisational decisions, both in positive youth development and business strategy. For example, during the COVID-19 shutdown in New South Wales, youth participants of the Creating Chances online programs demonstrated measurable improvements in their resilience, hope, and optimism among youth participants. This provided evidence to confirm that our online engagement strategy was having its intended effect, and we used this to report back to funders and schools the value of their investment.
Rachel, who is completing a PhD at the University of NSW, identifies 5 things to consider when measuring impact:
1. Collect Data
This is a good start! Generally, as we collect data about what we’re doing, we can better identify areas where we can improve. The more information we have in these areas, the clearer the gaps become. It is then up to us to work on the solutions to be better. It’s part of a continuous quality improvement process.
2. Measure to Key Outcomes
Creating Chances’ monitoring and evaluation team measure five key outcomes:
- Social Connectedness
We’re finding a lot of impact in those domains (with the exception of social connectedness). Several other positive impacts that have been identified are self-confidence (which overlaps with resilience) and peer interactions. Participants have less difficulty making friends and consequently make new friends, improve their communications and social skills, and learn to work as a team.
Our outcome measurements derive from evidence-based research gathered from around the world that we have adapted to our local context. Try to take a step back and understand what is success for you, and how can that be measured?
3. Adapt your measures for different ages
Across different Creating Chances programs, the quantitative data is measuring similar outcomes, but it is appropriated for the individual age. Accordingly, what positive peer interactions look like for someone who is 10 years old differs from what it looks like to someone who is 19. It is still the same construct, but what it looks like in effect is variant.
4. Train Facilitators
Our facilitation team are the ones on the ground, interacting with the young people every day. Over time they build positive relationships with young people, underpinned by a level of trust that you cannot replicate from an external researcher. Therefore, students are more open and honest in conversations with facilitators.
Facilitators are trained to conduct a semi-structured interview with participants that includes asking young people about the impact the program has had on their wellbeing. The interviews provide qualitative data.
Additionally, the facilitators pick a story of most significant change – one person who really stands out in the program as experiencing a personal transformation – and then they interview that young person about their journey and document it. That interview transcript is stored using NVIVO software and undergoes inductive thematic analysis.
5. Understand your Stakeholders
It is important to understand how multiple stakeholders benefit from us having a robust Monitoring and Evaluation system:
- Creating Chances can demonstrate outcomes and provide accurate feedback to them on their value of the investment.
- The teachers that run programs want to know how effective these programs are.
- Program Facilitators need to take ownership over the program, know the process and know exactly what they’re doing in terms of their impact. It’s also rewarding for them to know that they have influenced young people and had a positive effect.
- It is also important for that to be fed back to the young people so that they have ownership over what participating in this program has done for them and to build that self-awareness.
“There are a lot of program implementers out there who design activities based on intuition. It is inefficient and it’s using government resources, or the funders can waste a lot of money investing in these programs. It’s not efficient for them to be guiding practice based on their own personal knowledge or anecdotes, so using a strong evidence base is a systematic way to ensure what we’re doing represents best practice.”