FIP 80: The case for not collecting impact data – and what to do instead with Loïc Watine of IPA

Podcast: Play in new window | Download | Embed

Subscribe: RSS

I talk to Loïc Watine about evaluating impact in your social enterprise. Loïc is with the right-fit evidence unit at Innovations for Poverty Action. He helps organizations consider what else they should measure if they’re not in a position to make the most of a rigorous impact evaluation.

On this episode you’ll learn:

For this interview, we’re saying impact is the difference that is being made. So for farmers, the change in their yield or their income.
When feasible and when done well, RCTs are the best way to measure impact of an intervention. You can actually tell the impact attributable to the intervention and rules any other effect out, thanks to the use of a control group for comparison.
An RCT is a significant undertaking so the business model needs to be mature enough to warrant an RCT, such that any operational kinks need to be ironed out and the business is giving it’s best shot at the desired impact.
When an RCT isn’t the right choice, and it is more accessible than people imagine, you could instead focus more on the early outcomes and look at the existing evidence from similar programs elsewhere.
The first thing social enterprises can do when wanting to measure their impact is to draw out a theory of change, which is essentially describing what happens between the specific activities the business is doing and the outcomes you’re aiming for (e.g. more jobs, better exam results, etc.)
To decide what sort of impact studies businesses should be adopting, you can use your learning-cost ratio, which is about comparing the value of the learning you’re getting and the money you’re spending on the monitoring and evaluation element.
There are simple steps social enterprises can take to verify their impact without an RCT. Such as continuously verifying that the customers are actually using the product or service, which is a vital step for the impact to be realised.
Use the CART framework to prioritise what data to collect. Is your data Credible? Is the data you’re collecting Actionable and you know how you’ll use the data? Is the cost of collecting the data Responsible, compared to what other business benefits that money can be used for? And can the data be Transportable and inform your programming outside the specific context and time in which you’ve been collecting it?
The right-fit evidence unit is an advisory unit of IPA that works with other organisations to help figure out what data they should collect with their limited resources, if it’s the right time to do an impact evaluation, how to maximise their learning-cost ratio.
Organisations get tripped up by dedicating too many resources on the final stages of the theory of change, when they could be measuring more meaningful earlier outcomes far cheaper. Another mistake is to collect too much data without a clear reason or ability to analyse it, which is a waste of resources.