Don't hurl misleading statistics about like a drunk under a lamppost

“Politicians use statistics like drunks use a lamppost, for support rather than illumination.” I have a degree in psychology and got part way through a physics degree (which involves quite a bit of higher maths). These two disciplines are very strict about data – how you gather it, analyse it and report it – because it is so easy to manipulate and misrepresent to promote a particular agenda.

Some years ago a company that was tendering for some work for the Directory of Social Change did some analysis of our customer base. According to its research, 80 per cent of our customers had given a particular, very worrying response. It turned out that the sample size was 10 out of a customer base of more than 15,000, a number that is not only statistically invalid, but also factually useless! If I hadn’t understood enough about statistics to ask about sample size we might have made a catastrophically bad decision.

Reporting data and statistics properly really matters. Which is why I worry about the way the Fundraising Regulator has presented statistics about complaints in its recent reports and commentary. There are two core issues that would have had my stats tutor chucking chalk. First, the use of technical statistical terms without context. For example, the regulator refers to “significant numbers” filing complaints without clarifying what “significant” means. It refers to “high numbers of complaints received” without giving a comparator.

Second, its use of percentages masks underlying numbers that matter. For example, it says that complaints to the regulator increased by 18 per cent in the period between September and March 2017/18, compared with the same period in 2016/17. This statistic is probably right, but because we don’t know what the actual number of complaints was for that period we can’t compare. It does tell us that it received an additional 367 complaints for 2017/18, which brings the total number for the whole year to 1,080.

Statistics on the number of asks to the public are incredibly hard to come by, but based on approximately £10bn of voluntary donations it is not unreasonable to assume there are millions of fundraising contacts each year. I don’t wish to diminish its significance, but 1,080 complaints is small compared with the millions of interactions that could lead to a complaint. So in that context it’s hard to argue that 1,080 is significant.

The regulator says it investigated 55 complaints and in 81 per cent of those cases the complaints were upheld. Again, 81 per cent sounds a huge proportion, but amounts in fact to only 44 complaints. So the way these figures are presented gives the false impression that lots of charities are irresponsible fundraisers.

We all want fundraising in our sector to be of the highest standard, but I don’t think it helps to have uncontextualised statistics and technical terms hurled about as if by a metaphorical drunk trying to find something to lean on to support himself. We need data soberly presented, without being under the influence of an agenda. That way the whole picture is illuminated and we can make better decisions.

This article first appeared on Third Sector.