Where do Impact Analysts Focus their Attention? And Why?
This blog is taken from a recent publication Influencing Impact: Understanding impact measurement in social investment by the Social Investment Working Group.
“One day Nasreddin Hodja lost his ring. He searched and searched along the road, but couldn’t find it. Some people walking by stopped and asked him: “Hodja, what are you doing?” “I lost my ring at home, so I’m searching for it.” “Why don’t you look for it at home?” “The house is too dark. I can’t see anything there, so I’m searching for it here.”
The research conducted for SIAA working group by Professor Christie and her team at UCLA gives us a range of findings and insights.
One of their case studies especially caught my attention. Christie and her team explained how the methodologies and techniques used by the analysts in this case study, while impressive, were not necessary for the relatively simple decisions to be made by funders and others. The researchers conjecture that in such cases, impact measurement might be more a product of the analyst’s abilities and training than the decision-making requirements of stakeholders.
As illustrated by the wonderful story about the 13th century Sufi Nasreddin Hodja, sometimes the impact measurement methods some people adopt do not help them (or others) to achieve their goals.
At a more theoretical level, this can be explained within the conceptual framework Christie and her team developed for the research project: there can be both conflicts and synergies between each of the ‘purposes,’ ‘methodologies,’ and ‘primary users’ of impact measurement in social investment (see p.10 for more details).
But all this raises a more fundamental question: how do we ensure that the scope of impact measurement and the techniques employed really do match the sometimes-varying interests of multiple stakeholders?
Of course, good impact measurement always needs to manage these potential tensions. But it is especially important when it comes to social investment, because the traditional list of three stakeholder groups (beneficiaries, organisational staff, and payors/customers) is extended by a fourth group: the investors who are refunded out of the organisation’s ongoing surpluses.
Interestingly, the job of balancing interests is at first glance made more simple when budgets are tight, because the salience and scarcity of money mean that investors and staff alike are more likely to align their interests with those of payors than they would in more prosperous times. Basically, in times of austerity, the primary balancing job for impact measurement may be about allocating resources to measuring outcomes of interest to payors vs. to other outcomes of interest to beneficiaries.
I believe this issue is alive and kicking. For example, impact measurement frameworks used for social impact bonds and other payment-by-results contracts are often squarely focused on measuring outcomes against indicators of use and value to commissioners, and often treat the perspectives, insights and desires of beneficiaries themselves as a secondary consideration.
I think this is mistaken. Not only can this exclude beneficiaries’ voices from impact-based decision-making, it also puts the cart before the horse. The majority of the effort of impact measurement should instead be directed towards understanding and measuring the complex links between beneficiaries’ perceptions and interim outcomes as they receive support (or not) from organisations, their families and others. After all, it is arguably these factors that fundamentally drive the longer-term outcomes of interest to commissioners.
So what can we do? As anyone involved in impact measurement for social investment can attest, how we manage any of these issues is often highly context dependent. That’s why our researchers rightly put the transaction deal and social context at the heart of their conceptual framework (see p.10 again).
However, there are some strategic-level points that might be common across most situations.
One good place to start could be to think through an ‘impact analysis’ of impact measurement itself in the social investment context. A sample list of questions for such an analysis could be based on the Theory of Change approach:
- What is (are) the ultimate vision(s) of what we are trying to achieve with our impact measurement?
- What are the key interim outcomes and indicators, for the different stakeholders involved?
- How are these outcomes causally related, and delivered by different activities?
- Where should different stakeholders’ interests in different outcomes be either prioritised or marginalised in our impact measurement?
There is already some progress in dealing with such strategic considerations at the sector level. For example, the draft EU GECES guidelines on impact measurement recommend that impact measurement is made proportional to its use in decision making by different stakeholders, and interestingly take a narrow CBA approach to assessing thresholds. But this is only a start, and we have some way to go.
As we seek to improve in this area, we could do worse than heed the poor example of Nasreddin Hodja. We should not use impact measurement to confirm biases and produce crowd-pleasing results that allow stakeholders to make easy decisions. Rather, it should seek to shed light on issues that are topical and relevant, and do so in ways that help stakeholders make the right decisions.
Rohan Martyres is Head of Impact and Investment Strategy at CAN, UK. He leads on CAN Invest services for VCSEs, including social investment funds, and advisory in social impact and social investment readiness. He is also an accredited SROI practitioner.