What is the state of the “empiricism agenda” to understand “what works” in policy? And what is it that we don’t know?

I’m here at the What Works Global Summit (WWGS) in London, where David Halpern and Peter John are discussing the role of randomized trials in society. The WWGS is a gathering of practitioners in international development, policing, education, public health, activism, and many other areas where people have applied quantitative methods to get causal estimates on the outcomes of their social interventions.

The main speaker, David Halpern, is the Chief Executive at the Behavioral Insights Team, led the team since its creation, and before that was the chief analyst of the UK Prime Minister’s Strategy Unit. David is also national advisor of the What Works Network, I’ve blogged about the the Behavioral Insights Team here before, sharing a talk by Oliver Hauser on the role of randomized trials in policymaking. Charing the conversation is Peter John, professor of political science and public policy at UCL, and lead author of a book (and article) called Nudge Nudge, Think Think: Two Strategies for Changing Civic Behavior (I summarize it here).

David starts out by referring to Archie Cochrane’s book Efficiency and Effectiveness (1972), where he set out the argument for the use of randomized trials in medicine. In a side note, Cochrane asked:

what other profession encourages publications about its error, and experimental investigations into the effect of their actions? Which magistrate, judge, or headmaster has encouraged RCTs into their ‘therapeutic’ and ‘deterrent’ actions?…. Let us remember the number of bridges that have fallen down.

At the time, Cochrane was arguing for the basic randomized controlled trial. RCTs offer simple math for calculating potential outcomes, even as more causal methods are being available. In recent years, David has explained RCTs by describing how the British started to win in cycling. The British cycling team have worked on marginal gains: picking apart their work, testing different ideas, and making incremental improvements. If the UK could do that for cycling, why couldn’t they do it for other areas of public interest? David describes this as “radical incrementalism.” At the same time, it’s also important to be able to make “leaps” — describing Graeme Obree, whose crazy ideas about bicycle design allowed him to break the velodrome world speed record.

Next, David tells us about the journey from the “nudge unit” to the UK’s more recent efforts with What Works centres. Back in 2010, the UK government created a “nudge unit,” which aside from the policies it tested, has had a larger legacy to introduce the idea of empiricism into policy circles. To illustrate their work, David tells the story of an experiment the group did to test different kinds of tax letters to citizens. The group tested added extra lines to the letters, saying things like “Nine out of ten people pay their tax on time.” By building different different kinds of interventions, they got effects of over five percentage points. Next, they asked about the effects among the people who were least likely to have an effect. By testing different messages with these groups, they were able to increase tax payment by seven percentage points, asking the question “what works for who?”

Why does this matter? These experiments tested an intervention that cost very little to try — just changing the working of tax letters. So these experiments powerfully-demonstrated what governments could achieve by putting minimal effort into randomized trials. This has opened up opportunities for different kinds of experiments, like an experiment that tried to support learners to stay in further education colleges. They tried offering affirming values, saying something about grit, or texting student’s friends to ask their friends how their learning was going. The intervention had a nearly six percentage point effect on school attendance, and they’re waiting to see what the effect may be on scores.

The Behavioral Insights imported and exposed ministers to the idea of evidence based policy. The What Works movement is setting out to support people to generate new findings, transmit their findings, and support communities to adopt those findings. Together, they have created six “What Works Centres” in departments from health to policing, to support the UK government to evaluate social policy. Even in the health world, may issues of service delivery or public health have not be evaluated as well as pharmaceuticals. Davides arguesthat education randomized trials happened once a year at most, but thanks to the founding of the Education Endowment Foundation, they have funded 127 trials across 7,200 schools, creating resources for school administrators to make decisions based on the findings. The Early Intervention Foundation supports research on young people beyond school. Other groups include the What Works Crime Reduction, the centre focusing on Local Economic Growth, the Centre for Ageing Better, and the What Works Wellbeing centre. These centres are part of a larger network and wider work. David outlines the following goals:

Stimulating ministerial interest. Rather than telling ministers they should hold off on an idea until there’s an answer, we should tell ministers: you have more than one idea; why don’t we implement the policy, try several things at once, and trim the one that is least effectively?
(Re-)training the policy profession so policymakers are familiar with building policies that can be iterated and tested.
The Trial Advisory Panel, which can offer feedback and advice on how to design trials
Publishing what we don’t know, but ought to. David wants governments to publish lists of things that we don’t know that we ought to know the answer to.
Moving from efficiency reviews to efficacy reviews in government.

In the next five years, David wants to live in a world where “radical incrementalism” is routine everywhere in government. He wants to see more work to identify what kinds of things work for what people. This will require governments to link people’s data more closely. David wants to see empiricism applied to social work and the rest of the criminal justice system beyond police. Next, David wants to foster public demand and understanding. He wants people to interrogate the evidence behind different kinds of claims, and to expect that public services are taking an evidence-based approach. Finally, he wants to see the “What Works” enterprise grow internationally. David describes knowledge from experiments as a public good that we all contribute to every time we do an experiment.

David briefly notes that as we think about a society with more widespread experiments, it’s important to take serious the public concerns about privacy and social control associated with experimentation.

David concludes by pointing to the “terra incognita” on a map in the early renaissance. He argues that by working together across countries and regions, we can support each other to fill in the gaps of experimental knowledge on the outcomes of policies.

Questions and Answers

Q: Where do experiments fit in the politics of mistrust, when many in the public may be less likely to care about evidence? David responds: Most ministers in government come in with strong beliefs, including policies. Some of them will not have a good evidence base. But there are so many choices in any given area. One approach is to ask what goals a person has where they haven’t specified– and then help them work on that. And then ask what other questions aren’t headline news but which are amenable to evidence. The White House Social & Behavioral Sciences Team has published very dull things: things that everyone would agree to, and testing variations. If the UK nudge unit had started on radical reforms, they would have gotten knocked aside. There are million of other things to test that are less controversial. We can leave politicians their headline ideas. In some cases, policymakers have written “provisional” positions into legislation, budgeting resources to test them.

Observation: A participant from the Brookings Institution told a story from the White House Social and Behavioral Sciences Team. They’ve been running for two years, and many of their things have worked. He then talked excitedly about some of their failures. David responds that experimental groups have had many early successes because many government processes can easily be improved–anyone can do better. Over time, it will be important to develop good ways to talk about null results once the averages start to balance out.

Question: someone asks how to convince government to pay attention to evidence? David responds that with the What Works Centres, they have focused on building community among practitioners, who are able to carry out experiments independently of the national government. That community work can often create an appetite within government, says David.

Q: How do you think about ethics with policy trials? Halpern responds that he believes it’s important to have an overt panel of the public involved in decisions about what the public thinks are acceptable policy experiments. If we want a more experimental government, we may need to turn to juries as a model for maintaining ethics in the public interest.