ARVAC Bulletin 99

Problems with evidence-based policy

Secretly recorded at a private dinner last year, the government’s Respect Tsar, Louise Casey, was overheard uttering the unministerial phrase: “If No.10 says bloody ‘evidence-based policy’ to me once more I’ll deck them.” As researchers on the track of evidence, it seems to me we should look a little bit more closely at what she said. What, after all, is wrong with evidence-based policy? Surely it is better than policy based on hunch.

And yet anyone who flies too close to the official mind will realise that there is a misunderstanding there about what constitutes evidence. Those in government who demand hard evidence, and mean figures – and have never been taught that such things just aren’t possible, because what is most important is not susceptible to such things – have created a bastard form of evidence-based policy which is no such thing. That is why people like Louise Casey, who are frustrated because nothing happens without this chimera that never arrives, see ‘evidence-based policy’ as an excuse for complete inactivity. Why does it never arrive? Well, here are some reasons that social researchers know all too well but officials often fail to grasp.

1. Counting means comparing like with like, and that is inexact.
It means defining what is being counted, so that it can be categorised, when different people, different cultures and different eras, count differently – because they define things differently.

The problem of translation is well-known. Pepsi-Cola found that their famous slogan ‘Come alive with the Pepsi generation’, translated into Chinese, meant ‘Pepsi will bring your ancestors back from the grave’. Anyone who has studied crime statistics knows that they change with people’s perception of different crimes, as well as with the definitions of them – and whether people think there is any point in reporting them. They are, as much as anything else, measurements of fear and security.

These problems are particularly important when money values get involved. The first practitioner of cost-benefit analysis, Adolphe Jullien in the 1840s, was asked to work out the precise cost per kilometre travelled of each rail passenger, to work out what the fares should be on the new railways. He came up with the figure of 0.01254 francs per kilometre. “Ah, but what about administration and interest on capital?”, he was asked. And to meet these objections, he arbitrarily doubled the figure. It was more accurate, but no longer precise. These kinds of trade-offs lie behind the hardest government statistics anywhere.

2. You cannot count what is really important
Simply because it is so hard to measure what is really important, governments and institutions pin down something else. They have to. But the consequences of pinning down the wrong thing are severe: all their resources get focussed on achieving something they did not quite intend. How do you measure the success of a military unit in the Vietnam War? Answer: body count. Result: terrible loss of life among the Vietnamese, but no American victory. How do you make sure schools are living up to parents’ expectations? Answer: test the children as much as possible. Result: exhausted kids who can see no further than exams. How do you make trains more punctual? Answer: measure how often they’re late. Result: train companies simply lengthen the official journey times.

3. Controlling people with numbers does not work
The principle that numerical measurements will always be inaccurate if they are used as a means of control is now known as Goodhart’s Law. The reason is that, however incompetent staff may be, they will always be skilful enough to make targets work for them rather than against them. Take for example, the rule that patients shouldn’t be kept on hospital trolleys for more than four hours. In practice, some hospitals have got round this by putting them in chairs. Others have bought more expensive kinds of trolleys and re-designated them as ‘mobile beds’. This is the fundamental flaw of the target culture, which social researchers find themselves drawn inadvertently into collusion with.

4. Numbers have lives of their own
The story of Soviet glass manufacturing is horribly reminiscent of 21st century Whitehall. When output was evaluated by weight, it was too thick; when it was switched to targets by area, it was found to be too thin. Most of the key figures were falsified – something you can do during a reign of terror – but they carried within them a terrible authoritarianism to try to force them to be true. Which is why strikes had to be redefined as ‘sabotage’, and why after 1939, employees had to be fired if they were once more than twenty minutes late for work.

We don’t do that kind of thing nowadays. But we do measure our progress towards so-called ‘best practice’, locking local authorities into a dull and uninspired version of what may once have been best, but is now hopelessly second rate.

5. Number-crunchers tend to find what they look for
In quantum physics, the mere presence of the observer in sub-atomic particle experiments can change the results. In anthropology, researchers have to report on their own cultural reactions as a way of offsetting the same effect. Once you start looking critically at the measurement obsession, you keep falling over a strange phenomenon, which is that the official statistics tend to get worse when society is worried about something.

UK child abuse statistics stuck at the 1,500-a-year mark until 1984, when unprecedented publicity catapulted the issue to the top of the public agenda. Between 1984 and 85, child sex abuse cases shot up by 90 per cent. And in the following year they did the same again. Whatever the reason for this – hidden abuse or moral panic – what you get is not an accurate reflection of the real situation.

Conclusions
The real problem is how to interpret numbers. The basic problem of interpretation is that, no matter how many screeds of figures are available, they will not automatically tell you what causes what. That requires common sense, good judgement and intuition – none of which are encouraged by the neo-utilitarian ‘evidence-based’ policy at the heart of government. It is accepted wisdom to say the real answer is distinguishing between measuring outputs and outcomes, but this begs the question.

What is the outcome measure for the NHS for example? The number of patients successfully treated? Or is it the health of the population? The statistics for those would be diametrically different.

Outcome measurements assume that our institutions should be permanent. They are about organisational control. They don’t let us imagine whether we might be better off with different institutions instead. Real outcome measurements – if you can find them – are usually outside the control of institutions anyway.

What do we do as social researchers in the face of this serious confusion in the official mind? We can de-standardise the counting process so that the subjects of measurement do their own measuring - the pupils, the patients or the poor. One Latin American city measures air pollution by the number of days you can see the Andes from the city centre. That is a so-called ‘hot indicator’ that can inspire and involve people, when measuring ozone parts per million has to be done silently in a laboratory by a technician. We can make more use of stories than figures because they can often communicate complex, paradoxical truths better than figures. And we can ask questions: because they can devastate political statistics.

Yes, the carbon monoxide rate’s gone down, but is the air cleaner? Yes, our university professors have produced a record number of published papers, but is their teaching any good? Evidence-based policy is as vulnerable as the Emperor’s New Clothes to the incisive, intuitive human question.

The real problem is one of centralisation. Empires require measurement systems to control their outlying departments which they dare not trust to take their own decisions. Any measure of decentralisation can roll back the contradictions at the heart of evidence-based policy. We need school league tables that much less if we know our local headmaster.

So yes, social researchers need to provide evidence, but we have to remember we are part of a wider movement that in the end depends as much on common sense and humanity than it does on technocracy. We need to tell the truth about the importance of being generally right. As James Anyon, the first accountant in the USA, urged his students:
“Use figures as little as you can. Think and act upon facts, truths and principles and regard figures only as things to express these.”

David Boyle is an associate of the New Economics Foundation and the author of The Tyranny of Numbers.

Contact: Davidboyle1958@aol.com