Most public agencies are ill-equipped at the moment to make rational investment decisions. They need better metrics to evaluate performance.
Published on: 15/05/2015
Value for Money (VFM) analysis stems from a very simple concern shared by most people, especially when money is tight: "how can I make sure that I get the best value for my money when making a purchasing decision?"
Answering such a question is critical for public agencies, which constantly need to make difficult funding choices between an abundance of pressing issues. Yet, information at their disposal to make such funding choices is often scarce and unreliable. Deciding which development activities to fund and how is hugely more complex and sensitive than buying a car or a piece of electronic equipment. Yet, whereas one can now find dozens of price comparison websites for any goods under the sun, there is no equivalent mechanism to compare the costs and relative impacts of running a CLTS campaign versus subsidizing latrine adoption for example, or to compare the cost-effectiveness of providing access to improved sanitation vs. access to a sustainably managed water point.
Such a comparison might be unfair, as there are billions of people looking to buy electronic equipment at any one time, and what you buy is what you get for electronics. The main point, however, is that most public agencies are ill-equipped at the moment to make rational investment decisions: this is particularly true in the water, sanitation and hygiene (WASH) sector in developing countries, the area of focus for the VFM-WASH project.
This is because information is not gathered, analysed and publicized in a way that allows drawing comparisons between programme approaches, choice of activities and implementation arrangements. Some would argue that, given that WASH programmes are much more complex than buying electronics, programmes are by nature not comparable and cannot be taken out of their context. For this kind of reasons, estimating the Value for Money of WASH investments is seldom attempted beyond very crude estimates.
Other social sectors, such as the health sector, have gone much beyond such crude estimates and are using cost-effectiveness indicators on a routine basis for making funding allocation decisions.
The VFM-WASH project, initiated by DFID two years ago, posits that WASH stakeholders would have much to gain if programme funders and implementers were to adopt a stronger focus on VFM, so as to generate VFM indicators that can serve as a benchmark throughout the sector.
The WASHCost project, implemented by IRC with funding from the Bill and Melinda Gates Foundation, went some way in this direction but focused on evaluating the costs and service levels of specific investments rather than adopting an overall programmatic view.
To improve the quality of VFM analysis in the WASH sector, what is called for is a gradual increase in the level of detail and accuracy of VFM analysis being performed. From a funder’s perspective, the value of such analysis will increase when a large pool of comparative examples is built, from different countries, across years, in different sectors, etc. To make this kind of VFM analysis viable in practice, it should ideally be conducted as part of a broader evaluation activity (preferably an evaluation during the life of a programme, either yearly or mid-term). Official evaluators would usually get better access to data and have the opportunity to gather information (including qualitative) on other aspects of the programme so as to facilitate the interpretation of results. In this context, VFM analysis should be incorporated in standard ToRs for programme evaluations: the guidance note available on the VFM website can provide a basis for such standard TORs from an external evaluation perspective. Nonetheless, it is important to note that VFM analysis is not an evaluation in itself – the framework is analytical rather than evaluative.
Second, when donors start demanding “VFM analysis” on a more regular and consistent basis, programme implementers will start adopting this as part of their standard “modus operandi” and will start seeing how to use the data to improve programme management. VFM analysis can support performance-based management by giving managers crucial quantitative metrics, backed up by qualitative analysis. By comparing a poorly performing programme against a higher performing one with similar objectives and activities in the same country, a manager can identify key VFM drivers and areas in which the worse-performing programme could be improved. Programme implementers could then gradually develop the systems to allow them to address the questions they need to answer so as to improve programme management, as set out in Lucrezia Tincani’s blog on the VFW-WASH website. This will take time and effort, however, given that the WASH sector is behind in this area when compared to, for example, the health sector.
Programme implementers may of course decide to do this totally independently from funders, particularly in the case of those that are operated on the basis of more “private sector” principles. However, they would also need guidance on how to do it so as to compute indicators that are of interest to society at large (such as on sustainability or equity).
So what is called for in the WASH sector is a change in culture: programme funders and implementers should start demanding more robust metrics for evaluating performance and supporting funding decisions, without losing sight of course of other essential elements which cannot always be measured, such as the equity and cultural acceptability of their programmes.
This blog was originally published as "VFM analysis in the WASH sector: why is it needed?" on the VFM-WASH website on 11 May 2015.
At IRC we have strong opinions and we value honest and frank discussion, so you won't be surprised to hear that not all the opinions on this site represent our official policy.
