Few seriously question the need to measure results – but why do we do it? To satisfy funders that their money is being well spent? To empower intended beneficiaries to articulate their needs? To ensure that organizations learn from their mistakes? Different approaches to measurement have tended to reflect these very different rationales. Recently the different frameworks have become much less polarized and new approaches to measurement are adding to the ‘mix and match’ of available options.
 
The NGO community is bigger and louder than ever before. It is, as Peter Drucker argues, the fastest growing and most dynamic element of many societies. Its increased size, influence and visibility has brought with it new pressures. Since the mid-1990s, the accountability, representativeness and independence of NGOs have come under increasing scrutiny. As The Economist concluded following the Seattle riots in late 1999:

‘… citizens groups are increasingly powerful at the corporate, national and international levels. How they have become so, and what this means, are questions that urgently need to be addressed. Are citizens groups, as many of their supporters claim, the first steps towards an international civil society (whatever that might be)? Or do they represent a dangerous shift of power to unelected and unaccountable special-interest groups?'[1]

Michael Edwards, a long-time advocate of a powerful civil society, curiously echoes this view when he argues: ‘The challenge for NGOs is to show that they can put into practice the [accountability] principles that they campaign for in others.'[2]

The intimacy of NGOs as small-scale, voluntaristic initiatives built on personal trust and shared values has been all but lost, certainly for those in the mainstream international development community. Like multinational companies, these NGOs are often large and complex, dispersed geographically and culturally, and increasingly decentralized.

Measurement is important because stakeholders have differing levels of intimacy with, and knowledge of, increasingly complex development processes in which NGOs are engaged. Institutional funders, the giving public, and most of all intended beneficiaries have partial knowledge of what different NGOs do and how they perform due to limitations in time, expertise and knowledge. This loss of intimacy can be as true for those working inside NGOs as for their external stakeholders. Measurement is needed because commitment, values and trust do not alone provide a sufficient base for either effective development work or credible communication.

What are good indicators?

• They describe things simply, accurately and credibly.
• Preferably they measure performance in relation to aims.
• Usefully they enable more effective decision-making.

Indicators for efficiency and empowerment
Two basic measurement approaches emerged during the 1990s, one purporting to focus on efficiency and the other on empowerment. Efficiency approaches created indicators that supported clear hierarchies of aims, objectives and targets. These tended to be associated with classical approaches to planning, monitoring and ex-post evaluations. The most renowned of these is probably the project framework or logical framework (log frame), which identifies indicators during the planning process and in principle uses them subsequently to monitor actual performance.

This type of indicator has been subjected to a rich critique over the last decade. Central to this challenge has been the argument that development processes are (and indeed should be) organic, allowing for the evolution of both aims and processes. Seen through these spectacles, static assessment frameworks are at best suitable for tracking infrastructure development and other easily plannable projects.[3] At worst they are dismissed as being driven by bureaucratic donor requirements.

The participatory paradigm, on the other hand, saw a quite different rationale for indicators, mainly as a means of empowering intended beneficiaries to articulate their visions, needs and experiences in ways that would be intelligible and digestible to funders and other professional decision-makers. For this school, the process of engagement and creation of indicators were at least as important as their accuracy or the cost-effectiveness of their use.[4]

Overcoming the polarity
The polarity of the efficiency versus empowerment approaches to indicators has, thankfully, eroded over time. Those focused on measuring effectiveness and efficiency have recognized that indicators developed by, or at least agreed with, key stakeholders (including intended user groups) are likely to be more accurate and to create less conflict when they point towards problems and shortfalls. Those focused on empowerment have recognized that indicators can be more effective as an empowerment tool where they embody the classical SMART characteristics of a ‘good indicator’ (simple, measurable, accurate, relevant, timely).

In this less polarized context, the log frame has become more acceptable because it is recognized that it does help focus the mind during planning and can be used within a consultative, empowering process. In turn, those using the log frame and similar frameworks have more widely embraced a dialogue-based approach, if only as a more effective way of gathering information.

From inputs to outcomes
A veritable industry has grown up around the development of performance indicators. Despite much inevitable overlap, this has enhanced the quality of indicators used by many NGOs. One key area where progress has been made is the move from inputs to outcomes.

Historically indicators have focused on ‘inputs’, typically financial. Despite being largely discredited, funders and even NGOs continue to use financial expenditure as a measure of activity, even at times implying that it is a measure of effect. Recent years have seen far more emphasis placed on measuring process (what happened), outputs (what were the direct results) and outcomes (what were the overall, longer-term impacts). The move towards outcome measures requires an increased use of indicators based on stakeholders’ perceptions of actual or expected impact. These can be framed in quantitative terms, but the most sophisticated outcome measures often incorporate qualitative indicators.

From project to policy
A further development has been a shift away from the focus on ‘projects’ to an exploration of wider, indirect and higher-level impacts. One key element of this has been the greater attention paid to measuring policy impact, hardly surprising given the increased role played by NGOs in the process of public policy formation. Needless to say, robust performance indicators have not proved easy to identify and develop. Some of the more interesting attempts include the Cone framework developed at the Inter-American Foundation.[5]

Organizational learning and development
Evaluation has historically been seen as distinct from organizational learning and development. The former was associated mainly with often very public, funder inspections and reporting (compliance-based) and was predominantly project-based. The latter, on the other hand, was based more on personal interactions and confidential organizational initiatives (learning-based). This separation has done untold damage to the NGO community in that it has meant that work on measurement of development performance has all too rarely contributed to organizational learning, a prerequisite for sustainable, enhanced performance.

The key to this problem has been that NGOs are not joined-up organizations. They have not linked initiatives that attempt to measure effectiveness and accountability in different parts of the organization. Many NGOs undertake stakeholder surveys, have project management systems, and are evolving ethical purchasing practices. Most, however, have done far too little either in relating these diverse initiatives or in embedding them into the organization’s learning processes.[6]

This problem is not unique to NGOs, of course. However, the fact that NGO ‘quality assurance’ has in the past been rooted in personal commitment, values and trust has, as Hugo Slim argues, made it particularly difficult for NGOs to embrace more structured quality assurance frameworks, which tend to be seen as overly bureaucratic and politically desensitizing.[7]

Quality assurance frameworks
There are signs of a reversal of this separation of measurement for compliance and learning. Increasing numbers of (particularly larger) NGOs have moved to incorporate quality assurance frameworks into their organizational processes. These NGOs have generally drawn on existing quality assurance models such as the European Foundation for Quality Management, Investors in People and ISO9000. These frameworks all cover the basic building blocks of indicators, policy and measurement systems, target-setting and monitoring. Some also include public reporting, formal external verification and certification.

Although arguably a step in the right direction, most approaches to quality assurance suffer from significant drawbacks. They tend to become top-down, bureaucratic processes, confirming NGOs’ worst fears. They are overly focused on single ‘corporate’ aims rather than allowing for multiple perspectives. They are skewed in favour of bureaucratic conceptions of organizational efficiency rather than development (outcome) effectiveness. Finally, most of these frameworks are ‘values-blind’, and indeed are seen by some as having an eroding effect on often fragile organizational values.

Newer quality assurance frameworks
The recognition of these shortfalls by the main users of these frameworks – the business community – has led to a new generation of frameworks that focus on learning, allow for greater diversity, are organic, self-adjusting systems, and concentrate more on values and relationships than on procedures.[8]

Although these new frameworks systematize many of the values that the NGO community has historically promoted, NGOs have until recently remained reluctant to draw on them as a means of enhancing their learning, organizational efficiency, accountability and overall performance. There are, however, signs that some NGOs are warming towards more systematic social accounting, auditing and reporting.

Those NGOs (eg British Red Cross, Concern) signed up to the People in Aid code of best practice relating to the management and support of aid personnel are, for example, assessing their performance essentially through a social accounting and auditing process (see below). This enables them to integrate quality performance (the continuous improvement of human resource management) with accountability through the verification and disclosure of results.

Measuring for development
Useful indicators must be embedded within an organization’s management structure. They must be clearly linked to people’s experience, decision-making, and the learning processes that underlie long-term effectiveness. NGOs rightly distrust approaches that are costly and bureaucratic, and that disempower the very stakeholders that they exist to bring forward.

The new generation of quality assurance frameworks has sought to minimize these characteristics, and so to enhance the usefulness of measurement. In particular, they seek to balance measurement and management rigour with engagement, debate and relationship-building. Many explicitly build stakeholders into the design stage of the accounting process and the development of indicators. Some go further in engaging stakeholders in translating those indicators into organizational targets.

These are the elements that NGOs have long fought for in their debates with funders and others over measurement. What remains is for NGOs to use these new frameworks so that they can help in re-mixing measurement, management, accountability and performance.

Simon Zadek is Chair of the Institute of Social and Ethical AccountAbility and Visiting Professor at the Copenhagen Business School. He was previously Development Director of the New Economics Foundation. He can be contacted at zadek@csi.com and further information on his activities and publications can be found at www zadek.net

Peter Raynard is an independent research consultant. He has worked with organizations on issues of social responsibility and accountability for the past ten years. He can be contacted at peter@raynard.fsnet.co.uk

1  The Economist, 11–17 December 1999.
2  M Edwards, ‘Victims of their Own Success’ Guardian Weekly, 6–12 July 2000.
3  A Fowler (1997) Striking a Balance. A guide to enhancing the effectiveness of NGOs in international development London: Earthscan.
4  C Roche (1999) Impact Assessment for Development Agencies. Learning to value change Oxford:Oxfam.
5  J Blauert and S Zadek (editors) (1998) Mediating Sustainability: Growing policy from the grassroots Hartford: Kumarian (http://www.zadek.net).
6  See E Dawson (1998) ‘The Relevance of Social Audit for Oxfam GB’ Journal of Business Ethics Vol 17, No 13.
7  H Slim (1999) Future Imperatives? Quality, standards and human rights BOAG.
8  S Zadek, P Pruzan and R Evans (1997) Building Corporate AccountAbility: Emerging practice in social and ethical accounting, auditing and reporting London: Earthscan
9  ISEA (1999) AA1000: A Foundation Standard in Social and Ethical Accounting, Auditing and Reporting London: Institute of Social and Ethical AccountAbility (http://www.accountability.org.uk)