Social Impact Ratings Methodology

 

Donors are always asking us “is this a good charity?” There is typically no simple answer to this question. However, Charity Intelligence believes that the best information to help donors with this question is an assessment of the social impact produced by charities for each dollar donated.

Charity Intelligence (Ci) produces consistent, comparable impact ratings for many charities across Canada. These ratings are conservative, evidence-based estimates of the social value that charities create for their clients and the wider community. They are based on two metrics:

1. Demonstrated Impact score based on estimates of the charities’ demonstrated social return on investment (SROI)

2. Data Quality score based on the quality and quantity of impact data

A discussion of each metric can be found below.

Using the Demonstrated Impact score and the Data Quality score, Ci generates an Impact Rating for each charity analyzed. To date we have assessed charities primarily in the social services and education sectors as well as in the international aid sector. Ratings appear only for those charities that we have assessed and we continue to add charities to this list, thus more Impact Ratings will appear on charity profiles over time. The Impact Rating appears as a red dot overlaid on a grid:

  Impact Rating: Average

Impact Ratings Grid

 

Components of the Impact Rating

1) Demonstrated Impact Score (Proven Impact)

We estimate and compare the amount of social good that charities generate per dollar donated. How much good, measured in dollars, do donations accomplish? Social return on investment (SROI) is the best metric we know of for this task because it attempts to measure these amounts directly.

 SROI Equation


We use standard program evaluation techniques to provide SROI estimates for every major activity by every charity we analyze. We track benefits to both clients and taxpayers/society by estimating the number of outcomes that each charity produces beyond what would have happened absent service. We then multiply these numbers by estimates of the dollar value of each outcome for clients and for society. Benefits to clients include improvements in income, quality of life, and health, while benefits to society consist of increases in tax revenues and public cost savings in areas such as health care, public assistance, and law enforcement.  The long-term, discounted values of these benefits are added together and divided by total expenditures to generate a social return on investment/donation (SROI) estimate.  

For each charity, we calculate a lower bound, a best estimate, and an upper bound SROI:

The Demonstrated Impact score is a combination of the lower bound, best estimate, and upper bound SROI, with the lower bound and best estimate weighted more heavily than the upper bound. We emphasize benchmark SROI estimates that can be solidly supported by evidence and aim to produce estimates which measure proven impact. This decision to focus on conservative, evidence-supported estimates of results means that better information about a specific charity’s results will typically lead to higher estimates of its demonstrated social impact. This provides an incentive for charities to collect and share better data and diminishes subjectivity in our evaluation process.

To make our SROI estimates comparable across charities, and even across sectors, we regularly examine all causal factor estimates for consistency. As well, each of the inputs used in our model (including outcome values, attribution shares, drop-off rates, and baseline success rates) is based on extensive research. This includes a combination of randomized controlled studies, meta-analyses, and economic cost studies. As we receive more and better evidence, our estimates are regularly updated.

 

2) Data Quality Score

The second component of the Ci impact ratings is the Data Quality score (DQS). The Data Quality score measures the quality of a charity’s impact-related evidence. It is calculated as a percentage, using a grading that assesses each charitable program on the data it provides regarding eight main components of SROI: number of unique clients, pre-program client characteristics, program outcomes, counterfactuals, duration of program effects, duration of client engagement, external validation, and spending breakdown. The Data Quality score for each individual charity program is then weighted by the charity’s spending breakdown to determine the overall Data Quality score for the entire charity.

Ci has been measuring the quality of social results reporting for several years through our Results Reporting grade. A full explanation of the Results Reporting grading is available here.

The Data Quality score and the Results Reporting grade both measure the quality of information provided by charities. Both ask charities to report a breakdown of their spending by program area, as well as quantified outputs and outcomes that are relevant and timely.

There are, however, two key differences between the Data Quality score and the Results Reporting grade: