What can funders expect to learn from their poverty reduction projects – both the successes and the all-too-frequent failures? Not much, it seems, without major progress in the way projects are assessed. This was the conclusion of a recent international workshop hosted by the World Bank in Delhi, India.

 ‘I have learned that the real test of development’, World Bank president James Wolfensohn has said, ‘can be measured not by the bureaucratic approval process, but by the smile on a child’s face when a project is successful.’

South Asia is seriously adrift on this test. Far too many children are dying. While the proportion of the population living in poverty throughout South Asia has declined modestly through the 1990s, the absolute number of poor people has not fallen at all, despite rapid economic growth. Given that South Asia hosts many of the World Bank’s portfolio of 1,500 current projects, this is worrying. What is not working?

Project failure – no laughing matter

Seattle may have ushered in a new era of growing accountability and civil society participation in the international development policy debate, but we must not overlook the need for radical change at the grassroots level. Far too many well-intentioned projects end up as failures in the dusty village or muddy slum where the world’s poor people actually live and die.

Allan Meltzer, political economy professor at Carnegie Mellon University, chaired the International Financial Institution Advisory Commission, which in March this year published a hard-hitting report claiming that 55–60 per cent of World Bank-supported operations are failures.

The 1992 Wapenhams report, which claimed that two out of every five World Bank projects were failures, had already sent a shock wave through ‘The Bank’. Out went the old culture of just getting the maximum amount of dollars out the door (‘the biggest bucks for the bang’). In came James Wolfensohn and a raft of reforms including a quality assurance group, endless sustainability criteria and, most recently, comprehensive development frameworks. The Bank is arguably far more committed to learning from its mistakes than it was.

Not surprisingly, the Bank hotly disputes Meltzer’s figures, claiming that active projects in danger of not achieving their development objectives have declined from nearly 40 per cent (at the beginning of the 1990s) to an average of 22 per cent in the past two years.[1]

Learning by trial and error?

Development agencies in all sectors must embrace failure, not sweep it under the carpet. Indeed, the Bank’s Operations Evaluation Department (OED) argues that a 100 per cent success rate will always ‘be unlikely, and even undesirable’. As challenges become tougher and projects more complex and participative, too high a success rate could indicate lazy ‘cherry-picking’ of nice, safe but irrelevant projects.

Fine in theory, but the real issue addressed at the Delhi workshop is whether big funders like the Bank are actually geared up to learn from their mistakes.  Two recent independent studies say that the World Bank’s evaluation methods are closest to best practice among the multilateral development banks, but this may not be saying all that much.

Shahnaz Kazi, who has just completed a major review of Bank poverty projects in South Asia, found that just four out of every ten recent projects had adequate provision for monitoring, while 39 per cent had no provision at all for poverty monitoring. While she found a big improvement since 1993, there is still a long way to go, particularly since experience suggests that only half the projects with good monitoring designs are done well in practice. But the workshop did identify some grounds for optimism.

Two schools of thought

There have long been two caricatures among monitoring and evaluation (M&E) practitioners. One is the cold, economistic number-cruncher, striving for huge sample sizes, robust statistics and compelling evidence by subjecting locals to ‘batteries’ of questionnaires and excluding needy villagers from the project to act as ‘control groups’.

The other is the touchy-feely participation-wallah, fascinated by wacky processes that chew up the precious time of busy locals drawing pie charts in the sand, and often unable to say whether the project actually worked.

Case studies presented at the Delhi workshop demonstrated clearly that both approaches have made huge progress in recent years, with the quantitative approach becoming more participative and the participatory approach becoming more robust. These caricatures will hopefully soon be unrecognizable as the two approaches converge, assisted by a healthy cross-fertilization of ideas between North and South.

A message from India

In 1996, Sheela Patel of the Indian community group SPARC (see her article on p00) was asked by Oxfam to visit community regeneration initiatives in a number of UK cities. Patel found plenty of individual projects succeeding within their own limited terms of reference, but concluded that the overall approach to development in the UK was fundamentally flawed.

Among her reasons was the fact that community regeneration organizations were accountable not to community groups or local people but to government, and that at least 25 per cent of their work is consumed in audit enquiries required by funders.

Stung by these criticisms, a number of UK-based organizations like my think-tank, the New Economics Foundation (NEF), were spurred on to develop innovative forms of performance measurement such as community-based indicators, social auditing, social capital assessment and participatory economic analysis. While such approaches can be just as time-consuming as conventional M&E, they are as much about capacity-building as about evaluation, making the time spent more constuctive.

Working from below

‘True democracy cannot be worked by 20 men sitting at the centre,’ said Gandhi. ‘It has to be worked from below by the people of every village.’

To build on Patel’s challenging visits to the UK, NEF was commissioned last year by the Ford Foundation to explore the potential applicability in India of the new M&E approaches we have developed in the UK. Could they help to make the new tiers of local government, the so-called panchayati raj institutions, more effective?

Sanjiv Lingayah, co-author of NEF’s recent report Working from Below, concludes that techniques like social auditing and community indicators do have something to offer, and indeed already have close cousins all over South Asia. From educational radio plays in rural Mysore, to participatory indicators for joint forest management in the Western Ghats and urban report cards by the Public Affairs Centre in Bangalore, novel participatory activities are burgeoning. While some of these don’t look much like conventional M&E, they do speak volumes about project effectiveness – at least at the fairly small scale at which they have been ‘piloted’ to date.

Even cautious funders are now prepared to apply these innovations, and at scale too. The World Bank-funded District Poverty Initiative Project in Madhya Pradesh is an ambitious community-led anti-poverty programme, which will operate in 1,600 villages. Integral to the project’s M&E is a new approach to social and financial accountability called a development audit. Frances Sinha and colleagues at EDA Rural Systems have adapted NEF’s ultra-participative organizational social audit, adding a flexible framework, locally relevant principles and a stronger financial component, making the new approach well suited for rural development projects.

It is just these sorts of mix-and-match, hybrid approaches to M&E that seem to offer most hope for the next round of learning from development projects. It was Luca Pacioli, the father of double-entry book-keeping in the fourteenth century, who warned: ‘If you cannot be a good accountant, you will grope your way forward like a blind man and may meet great losses.’ Development funders like the Bank now admit they have met – inflicted is probably the right word – such losses in the past due to their inability to learn from their mistakes. If we’d let them, charities would probably like to get a few things off their chests too.

But there is a new opportunity here. Who knows, it could even make some children smile.
 
Alex MacGillivray is deputy director at the New Economics Foundation, the London-based think-tank specializing in innovative performance measurement in the UK and internationally. He can be contacted by email at alex.macgillivray@neweconomics.org

See NEF’s website at http://www.neweconomics.org for more about their work, including copies of Working from Below. All the papers presented at the Delhi workshop are available at: http://wbln1018.worldbank.org/sar/sa.nsf

1  OED Reach, 24, Spring 2000.

Community indicators
When people start eating ‘beer bananas’ rather than saving them to brew beer with, Ugandan villagers see this as a sure sign of impending famine – though not one found in the M&E textbooks.