Paul Penley
Impact measurement has been dubbed the Holy Grail of all social enterprise. If non-profit and for-profit social enterprises could capture the lasting difference made from all the meetings, projects, programs, fundraisers, volunteers, budgets and staff, we would have arrived. We would know if all the donations, charity walks, social media campaigns, grant proposals and collaborative partnerships are worth it. Not surprisingly, no one has found a short cut to universal impact measurement. There is no simple formula that determines whether a charity is ‘high impact’.
In a massive survey of donors, foundations and philanthropic advisors, the Money For Good II study found that impact information is what sophisticated donors seek most but cannot easily find. So what do we do? We certainly don’t give up. In fact, I believe there are a few specific steps to take when assessing any non-profit’s impact. That’s why checking the design and outcomes of non-profit impact measurement is the final step in our 6-step process for evaluating non-profit performance.
As a reminder, the six steps involve measuring charities against specific standards for non-profit performance arranged in the following categories: (1) Leadership, (2) Financial Management, (3) Financial Sustainability, (4) Leverage, (5) Strategy, and (6) Impact. We explored the first five categories in the previous five articles.
How do you evaluate impact?
As a non-profit performance analyst, I am constantly refining how to evaluate the impact of charities in numerous program and geographic areas. It is no theory or talking point for me. I have to make clear judgments that inform the execution of millions of US grant dollars every year. The customization required to measure the effectiveness of indigent healthcare in the US, microfinance in South Africa, education in Zambia and legal advocacy in Honduras cannot be boiled down easily into universal standards for desired outcomes. So for the purpose of defining a broadly applicable approach to evaluating impact, we must focus on how impact measurement is performed rather than how much impact is reported. Assessing impact measurement design and the quality of its outcomes is applicable to all charities.
Here are eight standards divided into two categories for evaluating a charity’s impact measurement system:
Impact measurement design
1. The scope of the organization’s vision is realistic and measurable.*
2. Organization has developed a scorecard to track key performance indicators (‘KPIs’).
3. The process for tracking outcomes is standardized and reliable.*
4. Programs are cut or adjusted based on data from the impact measurement system.
Quality of outcomes
5. The results include outcomes that last, not just annual activities and one-time events.
6. Statistics about program beneficiaries are compared to baselines, benchmarks or averages to determine if programs are moving the needle on a problem.
7. Recent independent evaluations have verified the charity’s reported outcomes.
8. Organization surveys beneficiaries about program quality and effectiveness.
The standards marked with an asterisk (*) do require a judgment call rather than a simple calculation. After all, charity evaluations are both a science and an art. But in the end, these eight standards will tell you if an organization is both managing toward outcomes and achieving meaningful outcomes. Here’s why.
Impact measurement design
To measure impact appropriately, an organization must have a clear vision of what impact it is working toward (standard 1 above). That’s why the scope of the organization’s vision (or big organizational goal) must be realistic and measurable. I have evaluated organizations whose vision is to reach every child everywhere. I have also seen organizations with a ‘Big Hairy Audacious Goal’ (Jim Collins’ BHAG) to care for every orphan’s physical, emotional and spiritual needs. Both goals are unrealistic in scope. They fail to provide meaningful direction, and they overlook the multiplicity of organizations already serving large segments of those populations. Other inadequate vision statements I have read speak about uplifting and enhancing lives. If the language is too ethereal, the organization cannot measure its intended impact.
If an organization has a realistic and measurable goal in place, then leaders can create an organizational scorecard for tracking key performance indicators (‘KPI’). A scorecard with the right KPIs tells an organization if it is making progress toward its ultimate goal. Of course, KPIs go beyond measuring impact to include any number of operational activities that contribute to the organization’s healthy operation and ultimate success. However, KPIs related to program impact should be included and analyzed quarterly or monthly. There should be a standardized and reliable method for collecting data related to program impact and reviewing it among other KPIs. Hence, standards 2 and 3 above are essential for good impact measurement design. When a charity has program impact data in a set of KPIs, senior leaders can make informed adjustments to guide the organization toward greater impact. As standard 4 indicates, programs should be cut or adjusted based on data from the impact measurement system.
Quality of outcomes
If an organization is measuring the impact of its programs systematically, it will communicate its outcomes at some point, either in an annual report or the ‘Results’ section of Intelligent Philanthropy’s Nonprofit Analytical Overview. So how do we judge what we find? Use standard 5 above. The results should include outcomes that last not just annual activities and one-time events. Outcomes focus on long-term results from the ongoing activities. How have beneficiaries defied the odds or continued success six months after the program? The fact that 92 per cent of women served at Colorado’s Women’s Crisis and Outreach Center do not return to a violent relationship or enter another shelter in the following year is a great outcome.
Remember statistics without a context are meaningless. If a prisoner re-entry program reports a 19 per cent recidivism rate after two years, I don’t know if I should be excited or disappointed. But when they compare that number to the zip code average of 57 per cent, I realize their programs reduce the number of re-offenders by two-thirds. That’s impressive. If Breakthrough’s educational assistance program reports an 86 per cent graduation rate, I may wonder why it’s not in the 90s. However, when that number is compared to the 46 per cent graduation rate for the average student from the vulnerable population they serve, I realize they have nearly doubled the graduation rate. Having benchmarks or base lines or averages is essential for determining whether a program’s results are making a significant difference (standard 6).
The seventh standard for evaluating the outcomes from a charity’s impact measurement system is to look for independent verification. Since it is difficult to judge whether a charity’s set of outcomes or the internal method for collecting data accurately represents the results of their programs, independent evaluations of program effectiveness can provide the perfect final step to determine if you have found a high-impact charity. In a few cases, I have gone in on behalf of donors to perform an independent evaluation of grantee outcomes. In one case, we did discover that a training organization did in fact produce the leveraged and long-term impact it reported. In another situation, we found that the definitions behind the data were unclear and the actual reported impact was therefore untrustworthy. The organization may have had a clear goal and carefully tracked KPIs, but they were carelessly aggregating numbers from field staff that did not have the same meaning. Hence standard 7 above is needed.
Now, sometimes the impact that a charity seeks may not be the best service for program beneficiaries. Therefore, any organization committed more to making a positive impact on beneficiaries than to serving its own ends will periodically survey those it serves. Charities need to verify program quality and effectiveness from direct feedback. That’s why we recommend standard 8 above.
Go and evaluate
I’m not sure you will believe that the eight standards above for quickly assessing a charity’s impact form the Holy Grail of charity evaluation. I do hope you see them as the best we can do for a general set of standards. Since only 1 out of 5 serious donors do any kind of substantive research about a charity’s effectiveness, these eight standards do give us a blueprint to do a whole lot more with the limited time we have.
If you are wondering where to find the data needed to complete this assessment, http://www.IntelligentPhilanthropy.com will be providing the corresponding impact measurement data in the 2013 version of its 2-page non-profit Analytical Overviews (available for any US-based 501c3). Remember the six steps I have presented throughout the year for non-profit evaluations comprise only a cursory evaluation. It’s not the level of due diligence appropriate for major gifts. But these simple standards do empower donors of all sizes and sophistication to get informed quickly and give more wisely. So go get to work and find the best charities in the world that are turning your donations into the impact you desire.
Paul Penley is director of research at the philanthropic advisory firm Excellence in Giving and creator of IntelligentPhilanthropy.com
Further articles from Alliance magazine related to these topics:
Comments (0)