How We Measure Impact
Stefan - October 14, 2020
As this blog series progresses, we’ve built the case for the importance of impact measurement to organisations, stakeholders, investors and the market. But the actual process of impact measurement may still seem a bit of a ‘black box’. With a variety of approaches offered by organisations on how impact should be measured and managed, it can be confusing to unpick exactly how impact should be measured.
This week, we hope to wade through this confusion to highlight the strengths and weaknesses of current approaches and the key factors to consider in impact measurement. The good news is that the market is beginning to develop an agreed set of well thought out standards, the bad news is that a missing factor (or the missing counterfactual) means that many impact measurement results may still be highly misleading.
The story so far…
We’ve come a long way!
As the last blog explained, a growing awareness of the value that impact measurement brings has led to stakeholders developing different approaches to the discipline. The aim is to find an alternative to traditional development programme approach to impact evaluation without the same time burden, rigid evaluation framework or cost, which can be 10-15% of total budgets.
Evolving beyond simple qualitative reports, market leaders have developed an impressive array of frameworks and tools to help track impact. The market is gradually coalescing around a best in class set that include the Impact Management Project’s 5 Dimensions of Impact to assess impact, IRIS+ to find common benchmarked metrics that align with the United Nations Sustainable Development Goals, and the IFC led Operating Principles for Impact Management, which assess impact manager compliance with best in class frameworks and practices.
Nevertheless, current approaches are still lacking in three complex but significant ways:
1. Beneficiary outcomes – impact measurement often concentrates on the outputs of investees and enterprises, rather than the impact this has on beneficiaries. Organisations may track mosquito nets handed out, without realising they are used for fishing rather than disease control.
2. Negative impacts – investors often track positive outcomes but fail to properly assess and report unintended negative outcomes from investment.
3. Proving additionality – at present, investors make qualitative judgements about whether their investment ‘would have happened anyway’, meaning impact results often confuse correlation with causation. We go into the consequences of this below.
Confusing correlation with causation
Stay with me here..
Correlation between two events simply indicates that a relationship exists, whereas causation more specifically says that one event causes the other. So when impact measurement observes correlations between impact investment and impact performance, this doesn’t imply causation. In reality, omitted variables bias, selection bias and reverse causality may drive a ‘counterfactual effect’, which is the impact that would have happened in the absence of an intervention.
To explain this, the table below may help. It looks at the self reported health status for individuals and whether they have visited a local health clinic that aims to bring essential medical facilities to an underserved rural community.
On the face of it, this table might suggest that health clinics cause people to get sicker. This is, of course, completely wrong, and misses the fact that individuals who attended this health clinic were already less healthy. Statisticians call this ‘selection bias’ and it is the same reason that only studying living people is not a good way to get information on the causes of mortality.
Similarly, a study that tracks individuals over time might find a correlation between poor health and visiting a health clinic, but this would omit things like their immune system, exposure to illness, lifestyle that both a) make people go to hospital and b) increase their chances of getting ill.
In the context of impact funds, the fact that fund managers pick leading examples of impactful businesses may mean they measure the impact of being a competent impact enterprise rather than the impact of the investor. Alternatively, the fact that an investee is on the trajectory toward becoming a profitable, impactful company may cause a fund manager to invest, or invest higher amounts, rather than the reverse.
Our challenge to the market
Plan, measure, analyse, report, evaluate, repeat…
Innovest has developed an impact measurement approach that directly focuses on beneficiary outcomes within the context of their community, using statistical techniques to produce valid estimates of the causes of impact. We believe that, if our impact measurement processes are adopted across the market, it is feasible for impact investors to gain a true quantitative understanding of their impact.
It is only when this practice is adopted widely across the market that we can gain a true understanding of the value and additionality of impact investments. Without it, we risk misallocation of responsible capital into initiatives that have negligible or negative impacts on the populations they are intending to reach.
To encourage alignment across approaches, Innovest has developed a set of key principles that we hope can be adopted irrespective of context:
- Finding additionality and causality – Innovest strives to create a new standard of measuring causality and additionality, which is the causal impact of an intervention on relevant outcomes
- Data collection is key – The success of any IMM process is dependent on the quality of the data collected, which depends on reliability, bias, respondents, what we can collect and when.
- Client and stakeholder focussed – Our IMM strategies are tailored to the context and needs of a client, with particular attention required on the impact of IMM on stakeholders.
- Cost efficient – IMM doesn’t require expensive randomised trials or invasive household surveys. Innovest instils lean data principles of simple, low cost measurement that can generate accurate real time results.
- Flexibility – Innovest aims to adapt its strategies to the standards, frameworks and tools required for diverse client engagements involving funds, enterprises, corporates and non-profits.
- Theory of Change driven – Measure outcomes forecasted by logical theory(s) of change
- Measure what matters – Innovest will measure outcomes, not outputs. An output only tells you that an activity has taken place, whereas an outcome is an indicator of change.
- Link Impact to Finance – Innovest aims to integrate IMM into financial processes. For an enterprise, this could mean including social value on balance sheets. For a fund, this could mean that investment pipeline research, due diligence and decisions are made with impact considerations in mind.