Evidence is a journey. Should it lead to proving or improving?

 

If you view gathering evidence as simply a means of demonstrating outcomes, you’re missing a trick. It’s most valuable when part of a journey of iterative improvement, writes Frances Flaxington (Catch22). 

Image 1 Evidence is a Journey

We often assume that generating evidence is a linear progression towards proving whether a service works. In reality the process is often two steps forward, one step back. At Realising Ambition we believe that evidence has a role in not only proving the impact of a service on outcomes, but in supporting organisations to improve their service delivery.

Realising Ambition is a UK-wide programme, replicating 25 outstanding projects that have a strong track record of helping young people to avoid pathways into offending. It is funded by the Big Lottery Fund, who commissioned the collection of data on the projects’ outcomes.

We initially wanted to see whether outcomes were moving in the directions expected. As the programme has developed, however, we have placed a greater emphasis on how the data collected can support both service refinement and adaptation, and organisational development.

It’s not black and white

The fact is that evidence rarely provides a clear-cut truth – that a service works or is cost-beneficial. Rather, evidence can support or challenge the beliefs that we, and others, have and it can point to ways in which a service might be improved.

Generating evidence can even challenge existing evidential foundations, leading to changes in the way evidence is collected and acted upon. It is not a sequential process.

At Realising Ambition we – and particularly our consortium partner Dartington Social Research Unit – have been supporting the organisations to collect, analyse and use data generated in order to inform service refinement efforts. We wanted to build organisations’ ownership over the emerging data and to support their capacity to work with evidence in the future. Our goal was organisational empowerment rather than ‘top-down’ evaluation.

The organisations delivering the projects have learned to test whether outcomes match expectations, identify where adaptations may be required in order to maximise impact and fit the local delivery context, and form a baseline that adaptations can be tested against.

An example of this is Children’s Parliament, which prior to the programme relied on observations, feedback and one to one conversations with beneficiaries and families. We helped to formalise the organisation’s approach, and Children’s Parliament developed a number of bespoke processes to measure the learning and development of each child taking part in its project, Children’s Parliament Community Initiative.

Whilst working with these organisations, we have gathered learning about effective evidence activities. We found that routine monitoring should be kept simple and limited to key outcomes that a project is trying to achieve.

No easy fix

It also became clear that good response rates in outcome monitoring can be achieved, but this requires commitment and a strong organisational culture of data-driven practice. Data generated should be valued and used at all levels of an organisation, from front-line practice to executive leadership.

You can read more about our evidence journeys with the projects, and the resulting learning, in our fourth Programme Insight. Programme Insights are a series of briefings that share reflections, learning and practical implications from the Realising Ambition programme.

The Insight also features data on six of the projects’ outcomes, which indicate that their service refinement was worthwhile: for the most intensive early intervention services, on average, over 50% of children showed improvements in behaviour and the proportion of young people with high levels of behaviour difficulties fell from an average of 54% to 35%. We hope that their evidence journeys continue to bring them to young people who will benefit from their work and to funders who realise that the proof is in the improving.

Frances Flaxington

Frances Flaxington is Strategic Director for Catch22 and leads the organisation’s Young People and Families Directorate, where she holds responsibility for its delivery and growth. Contact her on Twitter @catch22frances and share your thoughts about evidence, replication and early intervention with #RealisingAmbition.

A Realising Ambition consortium partner, Catch22 is a social business, a not for profit business with a social mission. For over 200 years it has designed and delivered services that build resilience and aspiration in people and communities.

Views expressed are the author’s own and do not necessarily represent those of the Alliance for Useful Evidence. Join us (it’s free and open to all) and find out more about the how we champion the use of evidence in social policy and practice.