Articles Day 8: opening up data for better, cheaper evidence and the new army of armchair evaluators

Day 8: opening up data for better, cheaper evidence and the new army of armchair evaluators


We have talked about the need for more and better use of evidence, but this does not always mean commissioning costly academic research. Instead we can find new ways of utilising the information already available and empowering wider society to make use of it. This means that as well as innovating with new programmes and policies, we also need to innovate with the tools we use to evaluate them.

From the perspective of a small provider of public services, the main reasons why evidence is not often forthcoming could be summarised as: cost, capacity and complexity. The giants of big pharma are able to invest in labs, experts and clinical trials, but a small neighbourhood-based youth organisation may find it hard to even get their hands on relevant data, let alone make comparisons with other organisations and groups in other areas. Commissioning academics to perform a rigorous trials or evaluation could cost as much as the service itself.

However, open data is a recent innovation trend that provides opportunities to lower the cost and increase the capacity for evaluation of smaller-scale interventions.

Making the most of what is available

Many billions of data points exist (it would be interesting to know exactly how many) that when assembled together could shed light on how effectively different public services perform, at relatively low cost. But many of these currently languish on internal databases and locked in filing cabinets, or buried in non-reusable reports to public bodies and funders.

Government, both national and local, holds fine-grained and robust data that, when shared, could help smaller organisations assess their impact in a much more evidence-based manner. However, getting hold of such data is not always easy – many datasets are held across different organisations, in different and potentially inaccessible formats. Open data portals can provide a hub for organisations to easily post datasets as well as host apps that enable others to easily interrogate them. The London Datastore, for instance, provides a wealth of datasets with users encouraged to interrogate the raw data and present it in a meaningful way.

Other bodies also hold a wealth of information about social programmes. Charitable funders routinely require monitoring reports from grant recipients, but these rarely see daylight. Sharing such information with other funders could help them build up an evidence-base on what is effective. NESTA’s approach to learning for the Neighbourhood Challenge programme has short-circuited this loop, asking funded projects to blog their progress direct to the public. Public service providers can also benefit from opening up their data: TfL’s developer zone provides a wealth of live feeds that can be used to both make services more useful and evaluate how they perform.

Stimulating new sources of interpretation and evaluation

To think of open data as simply a means for organisations to access previously withheld datasets is to miss the second way in which open data can promote the use of evidence. Ministers have promised an army of “armchair auditors” access to open data to help “root out waste and inefficiency”.  Of course much of the public would find such data difficult to manipulate: we need innovative intermediaries to help us understand and manipulate data. NESTA’s own Make it Local has partnered local councils with design agencies to demonstrate how locally, public bodies can make their data really useful to citizens. The media can also play a role: data journalism is driving demand for statistical and technical skills within traditional media outlets as readers become more and more familiar with medians, geo-coding, APIs and visualisations.

Necessary but not sufficient

While open data should be welcomed and encouraged – the Government’s progress so far with  and current consultation appear to be moving the agenda in the right direction  –  its ability to empower people in and of itself should not be overstated. Open data can help us push the envelope of opportunities to bring evidence to bear on social policy, but there are plenty of programmes and interventions where open data alone is not sufficient. Some data are not collected: we often know little about those that don’t use particular interventions or public services, or vital social transactions that take place in civil society, families and informal networks. There are holes in the data landscape for other obvious reasons: we also have no official record, for example, of the amount of illegal drugs sold in the UK market. Similarly, the design of policies can limit the usefulness of open data in testing the effectiveness of policies: rolling out policies nationally, rather than in randomly selected trail areas means we have no way of guessing what might otherwise have happened in the absence of the policy. All of these gaps still require careful and often sophisticated field research.

Open data provides opportunities to lower the cost and increase the demand for evidence, but it will also take time to shift the debate from the domain of the nerds to a wider democratisation of evidence.