The growing epidemic of stats misuse

Hilary Salt is founder of First Actuarial plc. She is currently the Actuarial Post’s Actuary of the Year. Late in 2011 she was asked to contribute a piece to the Independent’s Battle of the Ideas. This is what she wrote

hilary-salt.

Twenty years ago, if a conversation at a party began with someone asking me what I did, I could reply with ‘I work in pensions’ and we could move swiftly on to much more interesting topics like football or music.

Now the ‘p’ word produces such a desire to discuss retirement provision that I hide behind a different word. For a while I used the code ‘I work with statistics’ but now I’ve abandoned that too.

A word which once produced a polite frown now induces a discussion about the statistical basis for evidence of red wine being good/bad for you, or about how meaningful (or not) school league tables are or about measuring the impact of QE on economic growth.

But we have a difficult relationship with statistics. On one level, we seem to have replaced the 10 Commandments with the 10 Statistics – running our lives taking into account the need to limit our alcohol units, eat our 5 a day, read to our children, pay down our debts and reduce our stress levels.

These behaviours are made mandatory by cautionary tales of the statistical proof of what will happen if we don’t force ourselves into the statistically blessed box. At another level, people show a cynicism of statistics which whilst not new (lies, damned lies….) does perhaps show a deeper level of mistrust than has existed previously.

In particular, any statistic produced by a pressure group or even worse a commercial organisation (or anybody funded by a commercial organisation) is immediately suspect.

This cynicism is often well deserved as many organisations (including some who should know much better) use statistics carelessly.

Untitled 15 300x211 The growing epidemic of stats misuse

For me, the ‘big four’ examples of careless use are:

1. Confusing correlation and causation – just because two things are associated with each other does not mean one causes the other (the recent SpongeBob SquarePants debate was a useful example)

2. Assuming past trends can be projected to show future experience. For example if the past experience is shown in this graph, what would you assume happens after point A?

The actual future experience could be any of the lines shown below:

Untitled 22 300x155 The growing epidemic of stats misuse

3. Scaring people with big numbers. In the UK, we spend £69.5bn on State pension benefits (2009-10 figure). Getting this figure wrong by 1% (that is getting it almost bob on) means over/underestimating by almost £700 million. Asking about the significance of a number is always important.

4. Exploiting the lottery effect. If the stakes are high enough (a big money win or ‘catching’ cancer), we suspend disbelief easily.

While many people have an instinctive feel for the misuse of statistics, they still seem to practise a ‘to be on the safe side’ adherence to the latest claim of what the numbers say they must do.

In part I think it is because people have lost the self confidence to challenge statistics. Although I could descend into a critique of maths education, I think it is important to recognise four wider (and inter-related) effects which act to undermine the general ability to understand and challenge statistics.

The first is the tendency not to challenge or dissect another person’s statistics, but to respond instead with a different statistic.

Modern etiquette seems to frown on any questioning of others’ figures, however implausible they may be. Instead we must be tolerant and non judgemental and instead present alternative evidence.

This just produces a blaze of competing and contradictory figures which most normal people can only deal with by sticking their fingers in their ears and shouting ‘Ner ner I can’t hear you!’

Second, we live in a world which values simple stories and easy connections. It is far more compelling to believe that MMR causes autism than to conduct long term studies of the effect of many different drivers.

Third, we too often treat social science in the same way as natural science – making people into predictable and passive objects of experiments. The idea that on receiving a stimulus, a person-machine will always produce the same response may seem silly but it is one which underpins many statistical claims.

Finally and perhaps causing and exaggerating all these issues, is the falling away of a belief that we can rationally understand the world and take action to shape and change both it and ourselves. We need to challenge all these tendencies to remaster statistics.

 

SpongeBob SquarePants (character)

About henry tapper

Founder of the Pension PlayPen,, partner of Stella, father of Olly . I am the Pension Plowman
This entry was posted in actuaries, Martin Lewis, Music, pensions, Trades Union Congress, Treasury and tagged , , , , , , , . Bookmark the permalink.

1 Response to The growing epidemic of stats misuse

  1. Pingback: The problem with actuaries | The Vision of the Pension Plowman

Leave a Reply