Published on: 08/03/2013
Data collection is easier and cheaper than ever. But gathering sound evidence – and using it for positive and lasting change–takes time and collaborative efforts.
With the rapid rise of mobile phones and mobile phone-based monitoring, collecting data is easier and faster than ever. Still, collecting sound evidence isn't cheap. And how will the ocean of data help improve water and sanitation services?
There is a global movement around 'evidence-based approaches' and pressures from taxpayers and financiers to ensure 'value for money' in development work. The focus on generating 'evidence' presents both dangers and opportunities. This blog captures insights from an informal lunch discussion at IRC about the use of evidence for decision-making. We concluded that we need rigorous methods for generating and collecting data. In addition you need to learn together and engage with planners, implementers and decision-makers, so that they can use evidence for better practice and policy.
IRC uses both qualitative and quantitative monitoring methods. We have recently done a lot of work around monitoring life cycle costs and developing service delivery indicators. An area for further improvement is documenting impact, examining critically how our work helps effect change. We should continue to champion evidence-based decision-making for better WASH services, but also encourage a sensible debate about what we need to measure and how. The sector needs rigorous methods for generating and collecting data. But, a narrow focus on Randomised Controlled Trials (RCTs) as the golden standard can lead to a focus on evidence instead of understanding the underlying causes for success and failure of certain interventions. So, we also need other methods for compiling evidence.
Catarina Fonseca was enthusiastic about the methods IRC is using to gather data on qualitative change processes in Bangladesh. "Our sampling strategy in BRAC WASH shows rigour. But, we need to document and share our methods and how we are using them," she said. Because of the unprecedented scale of BRAC WASH, IRC is putting a strong emphasis on documenting the monitoring process and methodology (see this article).
If you make people part of the process it's easier to convince them about the strength of the method
IRC and BRAC use monitoring data to support project teams to learn and improve their interventions. But, Joep Verhagen and Kristof Bostoen, who have been closely involved in BRAC WASH monitoring, argued that it's equally important to show people how you gather evidence. "If you make people part of the process it's easier to convince them than if you simply write about the method. We show them that we are rigorous and our data can be trusted too," they said.
Simply providing sound evidence evidence will not automatically translate into better practice or policy. One of the conclusions from last year's conference on evidence-informed policy making was many policy makers’ do not have the capacity to use research and that there seem to be few effective strategies to strengthen this capacity. It is therefore crucial to build the demand for research information from policy makers. To achieve this, it is necessary to understand the incentives which drive policy makers to look for research information and their capacity to find and evaluate it.
Effective communication of evidence also has an important role to play. Patrick Moriarty presented a few of the challenges in communicating for impact. How to present evidence so that:
Communicating evidence in a sensible way could mean doing a whole range of things besides writing academic journal articles. It could be a well-chosen case study, involving district planners in action research or taking government staff on a learning journey. The main issue is that communication happens from the start of an intervention.
'Too often data is only shared at the end of a programme. We need to move to shorter learning cycles supported by data.' Joep Verhagen
Since ‘evidence’ never speaks for itself, knowledge brokers play an important role in helping others make sense of it. Catherine Fisher, blogging for IDS, points out that research does not automatically generate knowledge. 'This [knowledge] happens through a process of sense-making in which the “knower” is an active participant, not a passive recipient, who may deliberately or inadvertently choose to reject the intended message of the product.'
So, it's of critical importance that we engage in a collaborative process of knowledge creation and change. The figure below illustrates IRC's innovation approach in Triple S, described in one of Patrick Moriarty's blogs.
Ton Schouten described the application of evidence for system change as a collective journey. "If we want to effect change and generate evidence that contributes to this, we need to understand our target audiences, what their views are, what their interests are, and how we can support them to change." To make change happen, you need more than evidence. Jaap Pels added that besides understanding the people whose behaviour you're trying to change, you need the support of committed individuals as 'change champions' and some easy change 'low-hanging fruits' that you can use to generate impetus for further change.
Further reading: An article by the Institute for Poverty, Land and Agrarian Studies describes a range of conceptual and practical limitations to Evidence-Based Policymaking: Du Toit A (2012) 'Making sense of "evidence": Notes on the discursive politics of research and pro-poor policy making', Working Paper 21. PLAAS, UWC: Bellville
At IRC we have strong opinions and we value honest and frank discussion, so you won't be surprised to hear that not all the opinions on this site represent our official policy.