Reflections on a real-world RCT

 

Realising Ambition is a UK-wide £25m Big Lottery Fund programme replicating 25 services aimed at preventing children and young people from entering the criminal justice system. This programme, managed by Catch22 in partnership with the Dartington Social Research Unit, Substance and the Young Foundation, is supporting 3 of the organisations delivering these services to undergo an experimental evaluation through a randomised controlled trial (RCT). One of these organisations is Chance UK.

In this blog Chance UK’s Chief Executive Officer Gracia McGrath explains what their involvement in an RCT has meant for their organisation and what they have learnt from their experience so far.

RCTs in the social sciences can be contentious, partly because of the time and costs involved in running them and the ethical issues that can arise if it becomes obvious some of the control subjects have poorer outcomes. Chance UK has been asked on several occasions why we agreed to put ourselves and our organisation through this process. For us the answer is straightforward: we recognise the need to robustly test the impact of our service on the young children we work with – in this case, our early intervention mentoring programme. Qualitative data is important and, amongst other things, helps with emotional engagement from funders, volunteers and supporters. However, a well conducted RCT will provide robust evidence about impact on outcomes that is attributable to a service – a type of evidence that is in short supply across the sector.

So half-way through the RCT what have we learned about the benefits and challenges of undertaking an RCT in the challenging and messy real world of service delivery?

Start with a good understanding of what you’re delivering

Chance UK has undergone a number of external evaluations and we always benefit from fresh eyes looking at our work and asking critical questions. Working  with the Dartington Social Research Unit on our Theory of Change meant that we were all clear on what Chance UK are trying to achieve and precisely what means we are using to do so. Processes to make sure we were delivering our service in keeping with the original model have also provided a good grounding for the RCT. They have enabled us to monitor the content of both our mentoring and our parent work more closely, and in fact these have proved so helpful that we will continue to use them after our RCT is completed.

Don’t underestimate the complexity of undergoing a trial

The impact on a small organisation cannot be underestimated. Although the RCT itself was funded, the infrastructure of Chance UK meant little extra capacity for the additional planning, data collection and dealing with operational enquiries from inside and outside the organisation. An RCT has also required us to process double the number of referrals to meet both the numbers needed for the control group who were not receiving our service and the young people who are.  In order to ensure that we had the expertise to lead on this work in house we needed to create a new post; an additional cost we did not anticipate at the beginning of the process. This, and having a dedicated Quality & Evaluation Manager at Chance UK for the organisation, made the process easier to manage.

Bring your staff with you

Internally, we have learned the importance of ensuring everyone in the organisation knows what we are doing within the RCT and why. We have provided staff with assistance throughout, whether in relation to technical aspects of trial implementation, or reassurance that we are doing the right thing. We recognised that if we want the best and most accurate data to be collected we need frontline staff to believe in, and understand, what they are being asked to do.

You are what you measure

Our RCT experience has significantly improved our knowledge of evaluation throughout the organisation and cemented our commitment to evaluating everything we do. We also have to acknowledge that there is a risk attached to any trial and we need to be honest about that.  Most of all we have learned that, while challenging, it is possible for a small organisation to undergo an RCT with the right external, expert support. Chance UK remains committed to developing a robust evidence base for our work to ensure we are delivering high quality services to young people we support.

Gracia McGrath, OBE, is the Chief Executive Officer of Chance UK.

Realising Ambition will release the findings from its randomised control trials upon completion of the programme in June 2017.

Views expressed are the author’s own and do not necessarily represent those of the Alliance for Useful Evidence. Join us (it’s free and open to all) and find out more about the how we champion the use of evidence in social policy and practice.