Impact measurement is here to stay – but is it for the better?
The future of impact measurement in Russia
It is amazing how quickly the notion of social impact completely took over the non-profit agenda – both donors and NGOs know very well that they are “touching people’s lives”, which has certain effects and these effects can be measured.
Though opinions may vary as to what is measurable or how, there is a general agreement that social impact exists and it can and therefore should be measured. This is clear for the leading institutional funders – corporates and foundations alike, and it is on top of agenda of the Presidential Grants Fund – the governmental fund which is by far the largest donor in the country, so NGOs have to comply.
Over the last five years CAF acted as a trendsetter in the field of impact measurement in Russia, and we have travelled from the point where everyone was really skeptical about our first SROI report and the wellbeing questions we used for it to the point where literally everyone recognizes that their projects affect people’s wellbeing and tries to measure it, and even valuation of social outcomes is considered subjective but still possible. Shouldn’t we rejoice and move on to other topics requiring our immediate attention?
We don’t think our social impact measurement mission is accomplished for Russia and it looks like it hasn’t been accomplished by anyone elsewhere. This is for a number of reasons we are about to share, and the way these reasons are dealt with may have a huge impact on the future of social impact in the non-profit world.
- Though the need for impact measurement is clear, it looks like there is a lack of professional skills and expertise in the field: donors often prefer to engage independent practitioners and fully trust their judgements to measure impact, while NGOs complain about absence/lack of resources/skills and often do nothing at all.
The lack of expertise, knowledge and skills is traditionally addressed through training and being an infrastructure organization CAF Russia has provided a range of trainings for NGOs on social impact measurement. We have to admit they did not work exactly the way we expected them to, but we learned some interesting facts. NGOs understand the need to measure their impact and they already have some data at hand thanks to their direct access to stakeholders. They need help structuring the data and figuring out what else they need to collect, and this is usually very tailored work and should be done individually – not within some general impact measurement training. Also, impact measurement is still not one of the top priorities for NGOs , which is we believe mostly due to the second reason.
- The purpose of impact measurement is still unclear – it is now mostly done to prove that the impact exists, but while this was cool and fun like five years ago I dare say it no longer is – we can tell you without any complicated indicator systems or extensive data collection that you have an impact and even suggest the areas where you are making it. Is it so important for you to know if it is 7.56 or 7.65 on a 10-point scale if the only thing you are going to do with the number is publish it? Or maybe not even publish if you were aiming for at least 8-point impact on a 10-point scale. Huge resources are being spent on impact measurement, but few decisions are actually made based on impact information.
While everybody understands that impact data is good for reporting to donors and annual report publications, there is still little understanding of the benefits associated with using impact data to manage the impact of social projects. Both donors and NGOs should be more open to discussing negative impact or absence of impact and addressing it through strategic management decisions.
We understand that it should first start internally within the organizations and maybe at some point we will see it as part of the public third-sector agenda, but this is not going to happen soon – the pressure for positive social impact only is too high at the moment.
- The current trend that is taking over the sector in Russia is the attempt to find a one-size-fits-all solution, preferably online, where you could go and put in the outputs of your project so that they would be immediately and magically transformed into outcomes and impact. It is hard to put up with the idea that this is not possible and you still need to engage with stakeholders, so we are to witness more efforts of the kind.
There is no universal solution – at least at the moment. Or if there is one it will be so complex that very few organizations will be able to benefit from it.
On the other hand, the ten impact questions by SVI provide a perfect framework anyone starting out on impact measurement could use as a starting point to make sure they take into account all important aspects of social impact. It is great that with these questions it is easy to explain impact and impact measurement to anyone so that it is no longer some secret knowledge of a bunch of experts – even your beneficiaries can understand it.
- The issue of the level of rigor – it has never left us ever since the day we presented our first SROI ratio for a project in Russia with two decimals and got asked a question by one of the experts: “How come there are two decimals in your ratio when your whole research is so loose, inaccurate and biased?” The best answer here is “So that you could ask us” but there is still a need to address this concern.
There is no general consensus that the level of rigor may vary depending on how you are going to use impact data and you do not necessarily need the academia to be impressed by your data collection.
This creates a huge burden for NGOs that either decide not to do anything about impact data collection, because they are not scientists so they will never get it right, or force themselves to produce some scientific-research-like data analysis still criticized and questioned by the experts.
This situation is far from encouraging especially given the fact that there is no agreed “golden standard” of rigor. Reference to medical RCTs is as unhelpful in this case as the standard is unattainable.
Meanwhile, if we could ease the pressure on ourselves and our grantees and instead of trying to find out what our true and ultimate impact is would ask ourselves “Is there anything we can change the next time we are doing this with the same resources to improve the outcomes for our stakeholders? ” I dare say our research would be more useful and the data and analysis would be less biased.
At CAF Russia over the last five years we have embedded ongoing impact measurement into all our programmes and though bits of it are still mostly about proving our impact with multi-year datasets, even of mixed quality you can’t help turning more and more towards improving it – thinking about strategic changes to our programmes to help us minimize the negative and maximize the positive outcomes.
Internally, most of our programme managers who were quite skeptical about it in the beginning can now confirm that it is fun and useful, and makes their work more meaningful and less transactional. We hope that gradually by supporting donors and NGOs interested in measuring their impact for practical purposes we will persuade the third sector that social impact measurement is not a burden, but a useful exercise that can be done with very limited resources and should be repeated regularly and feed into strategic decision-making within any organization.
Now though this understanding might hugely spoil the market for CAF Russia as a social impact measurement/management consultant, it will hugely improve the outcomes of social projects for their beneficiaries and make decision-making easier for the funders. Therefore, though our vision may be blurred and biased as social impact measurement practitioners, we believe that the future of impact measurement is bright and the best of impact management is yet to come.