Evidence-based policy: data has its limits

 

Dr. Warren Pearce asks why, when there is such widespread support for evidence-based policy, is it so hard in practice? The answer, he argues, can be found in an array of definitions used for evidence and the shifting nature of policy that demands different kinds of evidence at different times.

 

“Aaarghhh! Politics and policy-making is so frustrating! We spend so much time conducting careful scientific analysis in all kinds of fields of enquiry. The results are published in papers subjected to rigorous peer review, then increasingly subjected to extended peer review, then further challenge in the academic literature. This is society’s most robust knowledge. Yet…yet…the politicians just won’t listen. Climate change…badgers…drugs policy…why won’t they listen to us?!”

Definitions: the devil is in the detail

So goes the wailing of many advocates of evidence-based policy (EBP). Support for the concept is apparently widespread. Why wouldn’t it be? It’s fairly hard (though not impossible) to argue against EBP as a general concept. So why is it apparently so hard in practice? One reason is that what we glibly refer to as ‘the evidence’ is not a given: criteria for defining evidence are not necessarily shared between policy participants such as stakeholders, politicians and publics. Neither is policy a fixed object in space and time, such as a regulation, briefing document or target. Rather, it is an ongoing process of understanding between participants. In short, we must be aware of the various meanings of EBP in three particular criteria:

1. If ‘evidence’ is policy-useful information, how does knowledge turn into useful information? And how does the context for knowledge-usage affect definitions of evidence?

2. Even if a piece of evidence commands widespread support, what are the different meanings it holds for different policy participants?

3. What does EBP mean to policy participants as a framework for guiding action? Are some participants empowered or disempowered by the demands of EBP?

The multiple meanings of ‘evidence’

I address some of these questions in a new paper on the implementation of local climate change policy, which uses a typology by Carol Weiss to understand the multiple meanings of ‘evidence’:

Weiss' typology of policy research

Here, we can see that ‘the evidence’ is not a given, and different assumptions and contexts give rise to the adoption of different forms of evidence. However, other influences can come to bear on evidence choices. In particular, the trend for quantitative indicators and targets ushered in by New Labour continues to hold sway in Whitehall and local government. As the old saying goes “the person with the numbers always wins”: if a policy area is to be taken seriously within government, then quantitative benchmarks and targets are a must. In my paper, I explain how local climate policy managers recognised that adopting targets for reducing carbon dioxide emissions was essential in order to gain credibility, even though they had few policy levers available with which to influence emissions.

To improve policy we need to use a broader range and style of perspectives

This desire for measurement and comparability across government runs contrary to Weiss’s presentation of evidence as context-specific. Moreover, it has led to misunderstandings of what evidence for policy should be. A turn to data without ideas and arguments leads to the kinds of frustration I caricatured at the beginning of this piece. Responses to climate change, TB in badgers and drug-related health issues can only be properly explained by understanding the functions of different kinds of evidence at different times. And only with these understandings can we hope to improve policy through the utilisation of a broader range of perspectives.

 

Dr. Warren Pearce, Research Fellow, Institute for Science and Society, University of Nottingham

@WarrenPearce

Warren is a Research Fellow on the ‘Making Science Public’ programme, researching public debates about climate change in the UK. He is the co-editor of a new special issue of ‘Evidence & Policy’ on evidence and meaning in policy making, within which he writes an editorial and a paper on implementing local climate change policy. This blog post is in part based on the ideas in those two articles.

 

Views expressed are the author’s own and do not necessarily represent those of the Alliance for Useful Evidence. To learn more about the Alliance and join us (it’s free and open to all) click here.