Dare to Compare: Uncomparable SROIs Are The Charity Sector's New Problem
The charity sector has a new problem: SROIs
A decade ago, the discussion around charity metrics was focused almost solely on overhead costs. Donors asked about administrative costs, salaries, and the cost to raise a dollar. Almost nobody was talking about any impact metrics. That was a problem. Today we have a new problem: SROI values.
Anyone who knows me would think that I just fell off my chair and bumped my head. Me saying that Social Return on Investment (SROI) values are a problem seems so opposed to what I’ve spent the past ten years of my career doing. But hear me out.
A decade ago, there were a handful of SROI values floating around on Canadian charities. Boston Consulting Group (BCG) analysed the impact on a few large charities, including Pathways to Education, Junior Achievement, and Big Brothers Big Sisters Canada (the national, not the local chapters). BCG did an incredible job putting these studies together, helping to further the cause of impact assessments for charities in Canada. BCG is congratulated and thanked for this ground breaking work in measuring impact. BCG’s analysis found SROI values of 18:1 up to 45:1, meaning that for an investment of $1, these charities were able to create $18 or $45 worth of value to society. These were very large numbers that the charities were happy to use to market their programs. Look how much value we have created!
There was no real appetite to compare these numbers and, indeed, the values were so large that it was like saying in each case “our charity takes your donation and creates so much social change you don’t have to think about the size of the multiplier, it’s huge.”
Then a couple of things happened. BCG did a study for Habitat for Humanity that showed that for every dollar it spent, it created $4 in social value. Again, seems pretty good, doesn’t it? That’s a whopping return worthy of donations, right? But also, other charities started throwing around SROI values. “We create $7 with every dollar we get”, “we have an SROI of 2:1”.
How should donors think about these values? What is good? Well, if you listen to the SROI Network (the group that has created the ultimate guide to SROI and are working very hard to bring SROI into the mainstream) you will hear that SROI values should not be compared. Donors should not look at one charity’s SROI of 45:1 and another charity’s SROI of 4:1 and make any conclusions. Wait, what? How can this be?
While charities throw out numbers to show how great their programs are, donors are not allowed to compare them. That seems to be the current view. And understandably so. When we dug into the four studies that created the four values in the chart above, we saw that there are wildly different assumptions going into each analysis. The values truly cannot be compared. Even when the same organization conducts the SROI analyses they cannot be compared, so when different groups do the analyses, there is absolutely no way to compare.
So, what the heck is the value of any of these numbers? The SROI Network says that the value lies in organizations looking at changes in their own SROI over time and understanding the differences. However, these four studies were done between 2011 and 2015, yet we have not seen any follow-up to see changes in these values. And almost never have we seen other charities repeat an SROI analysis.
Given this, I would argue that the value of almost every SROI study has been limited. And sadly, they are absolutely useless for donors. One cannot say with any certainty whether an 18:1 SROI is any better or worse than a 4:1 SROI. Again, what?? Is this possible?
Even when measuring the impact on the same charity, there is wide variation in SROI findings. Take for example Pathways to Education. Boston Consulting Group's assumptions showed 24:1 value which, by our standards, is very high impact. Charity Intelligence found roughly 3:1. By our standards, this is good impact. The Government of Canada found a return on the same program of roughly 1.4:1. This would be below average impact. Is Pathways creating $24 or $1.40 with each dollar it is given? The result depends entirely on the set of assumptions used. This is a huge problem rendering all of these one-off SROI values useless to donors.
A number of charities now claim in their annual reports and on their websites “we have an SROI of 4:1.” But this is a bit like me saying that my body temperature is 4 Thomson degrees. It really means nothing unless you can compare it. Should I be calling the doctor or going for a run?
The next time a charity says it has an SROI of 12:1, give them a call and ask “Is this good? What should I compare this with?” All of this public reporting of one-off SROI values is the new problem in the sector. And this is why Charity Intelligence is working to create consistent impact metrics that will allow donors to compare across charities and sectors, so that an SROI of 4:1 actually means something.
Why are we so passionate about understanding the impact of charities? I found the article Don’t Feed the Zombies by Kevin Starr fascinating. Researchers asked a number of donors how much difference they thought there was between the average charity and the most effective charity. They also asked some global poverty experts the same question. The results are striking. While the average donor thought the most effective charity was likely 1.5 times as effective as the average charity, the experts believe that the most effective charity is 100 times more effective than average.
The differences we see between charities are staggering. While some believe that Canadians need to give more, the reality is that we need to give better. Our giving can have so much more effectiveness if we give to those charities that are doing more with donations. If you want to find some great charities, check out our Top Impact lists.
To learn more:
Boston Consulting Group Making an Impact: Assessing Junior Achievement of Canada's Value Creation, January 2011
Boston Consulting Group Assessment of Pathways to Education, February 2011
Boston Consulting Group Big Brothers Big Sisters Canada Social Return on Investment Study, July 2013
Kilian Berz, Boston Consulting Group Transforming Lives: The Social Return on Investment of Habitat's Work in Canada, May 2015
Government of Canada Evaluation of Pathways to Education, Final Report, 2019.
Kevin Starr, Don’t Feed the Zombies, Stanford Social Innovation Review, March 28 2023
If you find Charity Intelligence’s research useful in your giving, please donate to support our work. Being funded by donors like you maintains our independence to help Canadians be informed in their giving. Canadians donate over $18 billion each year. This giving could achieve tremendous results. We hope Charity Intelligence’s research helps Canadians give better.
Legal disclaimer: The information in this report was prepared by Charity Intelligence Canada and its independent analysts from publicly available information. Charity Intelligence and its analysts have made endeavours to ensure that the data in this report is accurate and complete but accepts no liability.
The views and opinions expressed are to inform donors on matters of public interest. Views and opinions are not intended to malign any religion, ethnic group, organization, individual, or anyone or anything. Any dispute arising from your use of this website or viewing this material shall be governed by the laws of the Province of Ontario, without regard to any conflict or law provision.