International development is fixated with impact. But how do we know we’re all talking about the same thing?
Whether we like it or not, international development is fixated with impact. It’s part of almost every conversation about a development programme and none of us can ignore it. But how do we know we’re all talking about the same thing? ‘Impact’ is a term we use so often in so many different contexts, has it lost its meaning – and is it still useful?
This is what we found in The Methods Lab research project. When we first set out to develop and test innovative approaches to impact evaluation, we expected there would be most need for innovation around data analysis and causal inference. But, working with several aid programmes, we found there was confusion around what ‘impact’ meant, and that the greatest need was for support on the up-front parts of evaluation planning – and the definition of this term in particular.
Defining impact
The Oxford English dictionary gives two definitions of the word impact: ‘the action of one object coming forcibly into contact with another’ and ‘a marked effect or influence’.
These alternate meanings describe fundamentally different views of causality and evoke very different images. This mirrors how the term is used in international development; the two most widely used definitions of impact show a similar divergence of perspectives:
The counterfactual definition from statistics and econometrics defines impact by the measured difference of a predefined indicator (e.g. school test score) with the intervention and without the intervention. The OECD-DAC definition, on the other hand, defines impact much more broadly in terms of any long-term effect, whether intended, unintended, positive, negative, direct or indirect. In the words of our colleague Rick Davies, the former understanding looks for ‘causes of an effect’ and the latter looks for ‘effects of a cause’.
To add to the confusion, many agencies have their own definitions, which usually derive from one or other of these and, at the project or programme level, will usually be defined using a results chain model that distinguishes impact level results from outcome and output level results. The upshot of this is that we have a wide variety of things being labelled as impact.
Why is this a problem?
In programme design, discussions about impact set the aspirations for a programme and help to build cooperation, inform investment decisions, and are used to identify risks. During or after a programme, the way impact is defined will affect how its success or failure is perceived, and who takes credit or blame. And this then affects what we learn from the programme to help adapt it or contribute to other programmes.
Our approach is to use a common language, created by asking a series of questions. We have identified six varying characteristics of impact to increase understanding when using this term:
- Application. When you use the term impact: are you describing the goals; are you forecasting what kind of effects might be expected; or are you describing actual observed or measured change?
- Scope. Are you narrowing your description of impact to one or a few pre-defined variables (for instance, economic growth, household income, school test scores) or are you taking an open perspective of impact and looking at as many variables as makes sense so as to make a useful judgement?
- Subject and level of change. Where are you looking for impact: at the level of individual citizens, households, sub-groups or communities, at organisations or institutions, at sectors, at the environment?
- Degrees of separation. How direct is the link between the intervention and impact as you define it; can impact occur close to the intervention or is it always several steps removed? How certain is the causal link: does there have to be existing theory or empirical evidence for causal links or can impact occur in completely unexpected places?
- Immediacy, rate and durability of change. Is your understanding of impact static (that is, ‘this project had this much impact’) or can impact vary over time? And is it always a long time before impact can be observed?
- hom*ogeneity of benefits. Is impact a uniform or ‘average’ property, or does it vary across population groups or institutions with different experiences of benefits?
By thinking across these six dimensions of impact we can begin to find a middle ground between the econometric definition of impact and the very broad OECD-DAC definition – and be able to defend a contextually appropriate understanding of impact.
The Methods Lab paper, What is impact?, provides more detail on these six dimensions of impact and suggests two other frameworks: the ‘impact possibility continuum’ and five uses of impact.
