Charlie Denton - Innovest's Impact Measurement Lead
As this blog series progresses, we’ve built the case for the importance of impact measurement to organisations, stakeholders, investors and the market. But the actual process of impact measurement may still seem a bit of a ‘black box’. With a variety of approaches offered by organisations on how impact should be measured and managed, it can be confusing to unpick exactly how impact should be measured.
This week, we hope to wade through this confusion to highlight the strengths and weaknesses of current approaches and the key factors to consider in impact measurement. The good news is that the market is beginning to develop an agreed set of well thought out standards, the bad news is that a missing factor (or the missing counterfactual) means that many impact measurement results may still be highly misleading.
The story so far….
We’ve come a long way!
As the last blog explained, a growing awareness of the value that impact measurement brings has led to stakeholders developing different approaches to the discipline. The aim is to find an alternative to traditional development programme approach to impact evaluation without the same time burden, rigid evaluation framework or cost, which can be 10-15% of total budgets.
Evolving beyond simple qualitative reports, market leaders have developed an impressive array of frameworks and tools to help track impact. The market is gradually coalescing around a best in class set that include the Impact Management Project’s 5 Dimensions of Impact to assess impact, IRIS+ to find common benchmarked metrics that align with the United Nations Sustainable Development Goals, and the IFC led Operating Principles for Impact Management, which assess impact manager compliance with best in class frameworks and practices.
Nevertheless, current approaches are still lacking in three complex but significant ways:
1. Beneficiary outcomes - impact measurement often concentrates on the outputs of investees and enterprises, rather than the impact this has on beneficiaries. Organisations may track mosquito nets handed out, without realising they are used for fishing rather than disease control.
3. Proving additionality - at present, investors make qualitative judgements about whether their investment ‘would have happened anyway’, meaning impact results often confuse correlation with causation. We go into the consequences of this below.
Confusing Correlation with Causation
Stay with me here...
Correlation between two events simply indicates that a relationship exists, whereas causation more specifically says that one event causes the other. So when impact measurement observes correlations between impact investment and impact performance, this doesn’t imply causation. In reality, omitted variables bias, selection bias and reverse causality may drive a ‘counterfactual effect’, which is the impact that would have happened in the absence of an intervention.
To explain this, the table below may help. It looks at the self reported health status for individuals and whether they have visited a local health clinic that aims to bring essential medical facilities to an underserved rural community.
On the face of it, this table might suggest that health clinics cause people to get sicker. This is, of course, completely wrong, and misses the fact that individuals who attended this health clinic were already less healthy. Statisticians call this ‘selection bias’ and it is the same reason that only studying living people is not a good way to get information on the causes of mortality.
Similarly, a study that tracks individuals over time might find a correlation between poor health and visiting a health clinic, but this would omit things like their immune system, exposure to illness, lifestyle that both a) make people go to hospital and b) increase their chances of getting ill.
In the context of impact funds, the fact that fund managers pick leading examples of impactful businesses may mean they measure the impact of being a competent impact enterprise rather than the impact of the investor. Alternatively, the fact that an investee is on the trajectory toward becoming a profitable, impactful company may cause a fund manager to invest, or invest higher amounts, rather than the reverse.
Our Challenge to the Market
Plan, Measure, Analyse, Report, Evaluate, Repeat...
Innovest has developed an impact measurement approach that directly focuses on beneficiary outcomes within the context of their community, using statistical techniques to produce valid estimates of the causes of impact. We believe that, if our impact measurement processes are adopted across the market, it is feasible for impact investors to gain a true quantitative understanding of their impact.
It is only when this practice is adopted widely across the market that we can gain a true understanding of the value and additionality of impact investments. Without it, we risk misallocation of responsible capital into initiatives that have negligible or negative impacts on the populations they are intending to reach.
To encourage alignment across approaches, Innovest has developed a set of key principles that we hope can be adopted irrespective of context:
We just sent you an email. Please click the link in the email to confirm your subscription!
OKSubscriptions powered by Strikingly