Funders are generally agreed that more and better evaluations of funded projects would be a good thing. They would help in assessing the impact of projects and, consequently, in judging whether donors are spending money well or badly. But how do those on the receiving end feel about the reporting requirements set by donors? Alliance asked three organizations that operate programmes with government aid funding how their work is affected by donors’ evaluation systems and requirements for information.

The views of Pati Ruiz Corzo, Federal Director of the Sierra Gorda Biosphere Reserve, Mexico, are critical. Sierra Gorda has been implementing a Global Environmental Facility project in partnership with local grassroots groups. The ‘divine beings, mercenaries’ who are contracted by GEF to conduct evaluations too often seem to ‘understand nothing in depth’, she says. Rather, like so many consultants, they ‘throw around unfounded accusations’ and produce ‘biased analysis’. ‘They do not appreciate the achievements of the project and underestimate the local effort.’ She concludes that there is an element of self-interest in their findings: unless they ‘find errors, gaps or a lack of strategy, they cannot justify their paid contract’.

She feels that one of the central problems is that the requirements are devised without taking into consideration the local situation. ‘In our case the GEF project requirements were stipulated in the manner of a desktop theory completely unfamiliar with real contexts.’

Apoorva Oza, CEO of the Aga Khan Rural Support Programme (India), is at least in principle more positive about evaluation. ‘Let me state upfront,’ he says, ‘that I am a strong advocate of evaluation systems for NGOs.’ He concedes, however, that there are too many bad ones, ‘too many spurious kinds, and some whose groundwork does not match their reports. Much work needs to be done on the objectives and methodology of reporting and evaluation systems.’

By far the most positive view of evaluations of those we spoke to came from Atallah Kuttab, Director General of the Welfare Association, which operates mainly in Palestine and Lebanon. ‘Our experience is that it keeps us focused and helps us to achieve better transparency and accountability in our programmes. Also, having key donors that appreciate monitoring/evaluation systems helps us set up an institutional system within the Welfare Association, with part of the costs covered by the various donors.’

Do evaluation requirements constrain projects?

Considerably, according to Apoorva Oza. In a rapidly changing environment such as India, ‘even a five-year project becomes outdated at some time. An NGO that rigidly sticks to it will miss opportunities.’ He agrees that targets can help to maintain focus, but ‘after two or three years, when changes are required, these targets become a constraint.’

By contrast, he cites a European Commission grant which gave a certain amount of flexibility as to targets. ‘We had proposed only three line items for water resource development,’ he explains, ‘but thanks to the flexibility we enjoyed, we ended up with about 20 technologies, cheaper and more user-friendly, at the end of the project.’

Many projects, he adds, are subject to ‘rigid log-frames, rigid line items in the budget, and fixed percentages of leveraging prescribed [where donors want grantees to match a proportion of what they give as a grant].’ Information requirements are often largely burdensome since they are too ‘input-focussed’, with not enough attention paid to process or outcomes.

Here again, Atallah Kuttab has a more positive view, with an important reservation. He feels they are ‘very helpful in terms of monitoring but sometimes, in terms of assessing impact, I have my doubts.’ He adds: ‘A negative is that sometimes donors have different systems (especially the big ones) and that simply adds to the burden on our capacity with little value added to programme results.’

Do donors understand the costs? ‘In our case, yes,’ says Kuttab. Apoorva Oza agrees but feels they probably underestimate ‘the time costs of preparing and participating in the evaluation, especially for communities’.

The effect on risk-taking

But the real problem, Apoorva Oza feels, is what he describes as the annual target system that many donors employ. In conflict areas, for example, ‘if we want to insist on an inclusive approach, involving all castes in a dam project, the pressure to finish the dam in a year forces us to compromise instead of taking a stand with the dominant community.’ The preoccupation with annual targets, he feels, is ‘also a reason why many NGOs focus on implementation rather than policy as nothing can be achieved in a year in policy advocacy, which is always a risk, with high chances of failure.’

Atallah Kuttab also feels that evaluation systems inhibit risk-taking ‘as donors pay for the programme activities within specified objectives and disallow other costs’. He feels that Western donors are especially shy – or, in his own words, ‘totally frightened’ – of taking political risks, especially in advocacy programmes. He concedes, however, that it is ‘great to see most of our donors taking risks when it comes to doing work in Jerusalem, which was a taboo for many years due to the Oslo agreement.’

The end product

How useful are the reports that these evaluations produce? ‘They are much appreciated by management and donors,’ says Atallah Kuttab, ‘and that contributes to building confidence in our operational capacity, leading to more funds made available.’ So far, so good. ‘However,’ he continues, ‘the format of these reports is useful only to the donor. The Welfare Association has to do more in terms of formatting and restructuring its reports to make them useful to the wider audience.’

Pati Ruiz is characteristically forthright and condemnatory. The reports, she says, are ‘costly and voluminous documents that provide nothing new. They are full of errors and imprecisions without a constructive contribution.’

A better way?

The reporting requirements imposed by donors aside, the people we spoke to have their own systems for monitoring and assessing their work. ‘A good system we currently have,’ says Apoorva Oza, ‘is a brief annual evaluation, by the same person/s, over five years. This is better then a mid- or end-term review as feedback can be worked upon, and both NGO and evaluator learn together.’

He distinguishes two types of evaluation: explicit, where an external evaluation is made a part of the project, and implicit, where evaluation is done through the reporting requirements of the grant. ‘Explicit evaluations, if done well, have outcomes which are owned by all, and eventually acted upon by any good, self-critical NGO.’

Even Sierra Gorda, despite their bad experiences, have ‘gained in the process the professionalization of our monitoring and evaluation system’. Pati Ruiz explains: ‘We sought out SVT Group consultants from San Francisco to pioneer a methodology that reflects the social and environmental return of our work in four branches: monetizable, quantitative, qualitative and narrative. This is being adjusted in-house as a new discipline in reporting standards, and we hope to have objective indicators and numbers that give us the elements to report more transparently and concretely.’

Assessing the assessors

Among those we spoke to, therefore, donor-sponsored evaluations get a mixed press. Even where the experiences are largely positive, there is room for improvement – for example, for reporting formats that are useful for end-beneficiaries not just for donors and project managers; or for greater uniformity of reporting requirements where more than one funder is involved.

One final point: Pati Ruiz’s experience with the Sierra Gorda project has led her to suggest that the evaluators themselves should undergo some form of assessment. ‘With some exceptions,’ she suggests, ‘the consultants should be evaluated and classified in red lists.’ There should also be the possibility of ‘accompanied monitoring by people who have gained experience in the field and not at a desk’, who will therefore be more able to recognize achievements and weaknesses and to help construct processes, ‘not judging in ignorance that which they do not understand’. Many grantees will no doubt agree with her.

Alliance would like to thank the following for contributing to this article:

Atallah Kuttab Director General , Welfare Association, Palestine

Apoorva Oza CEO, Aga Khan Rural Support Programme (India)

Martha Isabel (Pati) Ruiz Corzo Federal Director, Sierra Gorda Biosphere Reserve, Mexico
Andrew Milner is Alliance Associate Editor. Email am@andrewmilner.free-online.co.uk