Demonstrated Impact
What Does ‘Impact’ Mean?
We use a Social Return on Investment (SROI) methodology to measure the impact of charitable activities. The SROI is an estimate of the total dollar value of social benefits that are realized as a result of a charity’s programs divided by the charity’s costs. Costs include program, administration, and fundraising costs, as well as the cost of goods in kind used in charitable activities and amortization on assets. In brief, we are measuring how many dollars of ‘social good’ are created per $1 of costs – this we call impact, expressed as a ratio of benefits over costs with a bigger ratio indicating greater impact.
Why ‘Demonstrated’ Impact?
Our estimates of SROI depend on the availability and quality of charity data. If key data are not available, if we cannot get a clear picture of what a charity does, then we are prevented from even attempting to measure the actual impact of a charity. In such cases, instead of abandoning impact measurement altogether, we endeavour to measure impact based on what data are available, even if these are limited. We begin by collecting publicly available data on a charity’s programs, in its reports or on its website. Some charities also share data privately with us. Together, the data that we collect or that we have access to represents to us what information is known about a charity’s programs – that is, what a charity is able to demonstrate with respect to the characteristics or outcomes of its programs.
Measuring Impact
A ‘Best Estimate’ SROI
Our impact rating is typically based on a snapshot of a particular charity’s activities in one fiscal year. Using data primarily from audited financial statements, we identify the total costs incurred by a charity in a given year. We then find out all we can about what programs the charity ran in that same year, going through the data and determining as best as we can how many people were helped and how they were helped. Once we have done this, we run the data through models we have built – based on extensive research – to estimate the dollar value of the ‘social good’ created as a result of the charity’s programs. A total, best estimate of the social value created by a charity is arrived at by summing the results of the different models we have run, for all of the charity’s programs for which there were sufficient data. This is our ‘best’ estimate in the sense of it being reasonable based on the data the charity made available and what we have learned through external research about different programs or interventions; it is compared to the total costs incurred in the same year by the charity, to give us a best estimate SROI.
Data Quality or Uncertainty
Charities are not always able to provide strong data about their programs, such that the best estimates that we come up with are precisely that, estimates, which in many cases are subject to considerable uncertainty. We may not know the exact number of beneficiaries of a program, the characteristics of those beneficiaries, for how long they were in the program, or how it helped them. What is more, there are things which are inherently difficult to know, either because a charity may not have data on it, there is little external research on the topic, or little consensus in the research. The models that we use to measure social value creation each involve a set of variables, changes to any one of which affects the estimates we come up with. Depending on the data a charity is able to provide about its programs, more or fewer of these variables will be ‘known’, with the rest ‘unknown’ in the sense of our not knowing their values. For each program, we systematically explore how the total social value created by the program changes when unknown variables are given different values than what we had assumed in the best estimate. We assign levels of ‘risk’ that values of variables could be quite different from what we had assumed originally, and ultimately adjust our best estimates downward based on the extent to which the data underlying them are uncertain. Provisionally called ‘risk-adjusted’ estimates, these again are summed across all programs and divided by the charity’s total costs, to give us a ‘risk-adjusted’ SROI.
