How to spot spin and inappropriate use of statistics

Statistics can be misused, ‘spun’ or used inappropriately in many different ways. This is not always done consciously or intentionally, and the resulting facts or analysis are not necessarily wrong. They may, however, present a partial or overly simplistic picture. Darrell Huff said in the book How to Lie with Statistics:

The fact is that, despite its mathematical base, statistics is as much an art as it is a science. A great many manipulations and even distortions are possible within the bounds of propriety.

Here to spin means to deliberately draw conclusions from statistical evidence which are not supported by this data alone, or to present statistics in a way which is intended to lead their audience to draw such conclusions.

This briefing sets out some common ways in which statistics are used inappropriately or spun and gives some tips to help spot this. The tips are explained in more detail below, but the three essential questions to ask yourself when looking at statistics are:

Compared to what?     Since when?     Says who?

General questions to ask when looking at a statistic

1. What product or point of view is the author trying to ‘sell’?
2. Are there any statistics or background that is obviously missing?
3. Do the author’s conclusions logically follow from the statistics?
4. Are comparisons made like-for-like?
5. If there is any doubt about the original source of the statistic: Who created them and how, why and when were they created?

Some more specific points to look out for

• Statistics without any context, background or comparisons.
• Totals without rates or without any comparators.
• Percentages without any absolute values.
• A case made without consideration of contrary or inconclusive evidence.
• An overly simplistic view about cause and effect.
• Very large or very small numbers where the author assumes importance, or lack of it, solely on this basis.
• Absolute values, 100% or 0%, especially in forecasts or estimates
• Records or hyperbole without any further context.
• The term ‘significant’: assume it is the author’s interpretation of what constitutes large/important unless it says ‘statistically significant’.
• Ambiguous phrases such as ‘at least’, ‘as high as’, ‘includes’, ‘much more’ and so on.
• Unspecified averages (mean/median) where they could be different.
• Use of ‘average’ to mean ‘typical’, the definition of which is known only to the author.
• Lack of detail for surveys (sample size, source and the questions asked).
• Cut-down, uneven or missing chart axes.
• Percentage changes in percentages, rates or index numbers.
• Changes in relative risks without reference to absolute changes.
• Statistics on money that compare different time periods without using real prices.
• Statistics on money that do not spell out the time periods in question.
• Over precision (intended to lend an air of authority).
• Statistics that seem wildly unlikely or that look too good to be true.
• Data on things people may want kept secret, such as the number of illegal immigrants, drug use, sexual relationships and extreme views.
• If it is scientific data have the results been published in a reputable peer-reviewed journal? This doesn’t make it infallible, just less likely to be spun or contain inappropriate use of data.
• Unsourced statistics.