Three Market Forces that Drive the Quality of Impact Measurement
This blog is taken from a recent publication Influencing Impact: Understanding impact measurement in social investment by the Social Investment Working Group.
Impact measurement is market driven. There is a market for impact measurement services in which analysts compete to sell projects to clients in the private, public, and governmental sectors. Consequently, analysts must deliver what clients want and need if they are to be successful.
At the same time, impact measurement is market driving. Through their work, analysts help their clients act more effectively in social investment markets. They also provide information that may help the customers of social enterprises act more rationally in markets for a wide range of products and services.
Putting these ideas together, we can build a simple theory of change. The quality of impact measurement depends (substantially but not exclusively) on how the market for impact measurement services functions. In turn, the quality of impact measurement affects how social investment markets and social product/service markets function.
From this perspective, the market for impact measurement services matters a great deal. So much that I believe the social benefit sector cannot succeed unless this market functions well. Unfortunately, it is not an efficient market, and the recent SIAA survey provides some clues about why. More importantly, it also provides clues about how it may be improved.
Training: The Supply Side
The survey suggests that the type of academic training analysts received influences how they measure impacts. In other words, in the market for impact measurement, analysts sell the services they know how to provide. Dr. John Gargani, Gargani + Company Inc., has 20 years of experience helping nonpro!t organisations,foundations, corporations, and government agencies measure social impact. His current work ranges from directing large-scale randomized trials of scalable social solutions to supporting innovative social start-ups. 31 Based on the pattern of coursework reported in the survey, the services analysts sell are likely characterized by basic statistical, qualitative, and economic methods, many of which were learned in introductory courses. Advanced methods, including those related to causal inference, are far less common in the marketplace and the classroom.
The prominence of basic methods is understandable. Overall, 59% of respondents (69 of 117) earned an MA, MS, or MBA as their highest degree. They completed their coursework in one to two years (if full time) in programs that likely emphasized using research evidence over methods for producing it. In comparison, only 13% of respondents (15 of 117) earned a Ph.D. in longer programs that likely required a greater mastery of methods.
The focus on economic methods is also understandable; 31% of respondents (34 of 109) were trained in business, finance, or economics, and 19% (21 of 109) were trained in public policy where economic methods play an important role. The remaining 50% were trained in a variety of other fields, including diverse social sciences and the humanities, where the emphasis on research methods may be low and no single method dominates.
Are analysts selling the most appropriate methods in the market? That depends. If we only consider the market for impact measurement services, current methods appear to be “good enough” to meet demand. If we consider the theory of change above, I believe current methods— which are often rough and approximate—may be insufficient to promote efficiency in downstream markets.
If I am right, then training has the potential to increase the social impact of our work. It seems unlikely, at least in the short term, that we can change the way universities train their students. So it is up to us, as a community, to provide training in stronger methods. Professional organizations, like Social Value International, may be able to lead these efforts and help establish which methods practicing impact analysts should master.
Use: The Demand Side
In market-driven settings, the needs of buyers and sellers are paramount, and the needs of other stakeholders may not be fully considered by them as they transact business. This may be the case in the market for impact measurement services. The most commonly endorsed purpose for measuring impact is influencing decision makers (93% strongly agree) who are either clients or the funders of clients. The least commonly endorsed purpose is meeting the needs of underserved project beneficiaries (60% strongly agree) who play no role in the market transaction that commissioned the impact measurement project.
This gap points to a social inefficiency in the market for impact measurement services. How can we, as a community, help the market better meet the needs of clients and stakeholders? One solution might be found in our work contracts. Imagine we all included common language in our contracts that required clients to endorse our professional standards. Executing a contract would then become an opportunity for clients to affirm the duty of analysts to include stakeholder perspectives, maintain independence of thought and action, and make findings public. Professional organizations like SIAA and The SROI Network (soon to be Social Value International) may be able to lead the way on this, facilitating the development of standards, contract language, incentives, and enforcement mechanisms.
Feasibility: The Market Value of Impact Information
The budgets for most impact measurement projects are small. About one third are €10,000 or less; 70% are €50,000 or less. Given this, most projects are conducted over short periods of time by individuals or small teams. Impacts, however, are typically realized over long periods of time in complex settings. Resources and reality are not in balance, limiting the quality of impact measurements.
It is tempting to suggest that budgets are small because the charity sector is (and always has been) under-resourced. I don’t find this is a satisfactory explanation. I believe the principal reason is that clients have little economic incentive to increase the quality of impact measurement. From their perspective, the value of better information does not outweigh its cost.
How can we, as a community, make our work more valuable? One solution may be to have third parties rate the quality of our impact measurements. If the ratings were made public along with guidelines for how impact measurements of varying quality should be used, it might increase the demand for better information and the budgets to produce it. Again, this may be a role for Social Value International.
Making a Difference with Impact Measurement
The SIAA survey suggests three market forces that drive the quality of impact measurement— what analysts know how to do (the supply side of the market), how clients want to use impact measurements (the demand side of the market), and the level of resources allocated for impact measurement projects (the market value of impact information).
I believe we can leverage these forces to improve the quality of impact measurements beyond that currently produced by the market. By doing so, we may be able to make other markets— those for social investments and social products and services—more efficient. The potential benefits to society are substantial, and the field is well positioned to provide them. We simply need to work together.
Dr. John Gargani, Gargani + Company Inc., has 20 years of experience helping nonpro!t organisations,foundations, corporations, and government agencies measure social impact. His current work ranges from directing large-scale randomized trials of scalable social solutions to supporting innovative social start-ups.