Catherine-Rose Stocks-Rankin shares her reflections on using Contribution Analysis to evaluate the impact of Iriss – an intermediary organisation with a focus on knowledge sharing, evidence-informed practice and innovation.
What’s the question?
Iriss aspires to support culture change in the design, delivery and experience of social services in Scotland. This is reflected in its three outcomes of: embedding collaboration; enhancing people’s knowledge, resources and skills to use evidence and innovate; and placing people at the centre of the design of services and support.
But how do we know if Iriss achieves these outcomes? And what are the mechanisms, the risks, and the enablers for Iriss’ work? These are the questions I was tasked with answering when I arrived at the organisation in October 2013.
Finding, and filling, the gaps in the evidence
When I began my evaluation, there was very little existing literature on the role of intermediary organisations – much less the impact that they have on the sector. Even the term ‘intermediary’ was little known. We know something of the impact of social services and there is evidence on the impact of public policy. But, intermediaries? What do they do?
There is a range of organisations in Scotland and the UK which consider themselves ‘intermediaries’. SCIE refers to itself as a ‘knowledge broker organisation’; Joint Improvement Team uses the term ‘improvement organisation’; the ALLIANCE is a ‘third sector intermediary’; and Nesta uses the term ‘i-teams’ – each one referring to intermediary work.
One output of our research is a theory of change for intermediary work – a narrative which describes the journey from inputs and activities to outputs and outcomes – which fits with Iriss’ theory of change that shows a spectrum of activity, too. I drew inspiration from the typologies created by RURU and Nesta.
Three kinds of (good) evidence
IRISS suggests that we pay attention to peer-reviewed research evidence; the stories and experiences of people accessing support and practitioner wisdom.
Following suit, my evaluation makes claims based on:
• a summary of the literature on evidence-informed practice, innovation and knowledge media
• the experience of people accessing Iriss’ resources
• the experience of Iriss staff as practitioners and promoters of evidence-use; innovation; and clear, accessible, communication.
I generated different kinds of data for each of these three categories, reviewed the literature that inspires the three Iriss programme teams and carried out 45 interviews with people who make use of Iriss’ resources. This evidence was augmented by a survey of the Scottish social services sector about Iriss’ reach and impact.
But most of my evidence gathering was rooted in Iriss itself. Working with the programme teams for 18 months, together we developed a theory of change to show the journey from inputs to outcomes. This narrative focuses on the mechanisms that enable change (as well the risks that Iriss encounters!). Producing this theory of change was a collaborative process – designed to encourage reflection from the team and honesty about what has worked well and why.
What has Iriss gained?
- We now use CA to plan projects, reflect and improve current work, develop strategic plans and evaluation impact
- CA prompts the programme teams to keep ‘why’ and ‘how’ explicitly in mind when planning projects, helping ensure outcomes are achieved.
- Talking about risks and enablers ensures that the planning and delivery is robust
- Partners in a project are prompted to talk openly about what works and doesn’t work for them
- We now have a better understanding of, and evidence to support, Iriss’ activities, values and approach as an intermediary organisation – read our report and view our infographic below of the contribution analysis story!
Views expressed are the author’s own and do not necessarily represent those of the Alliance for Useful Evidence. Remember you can join us (it’s free and open to all) and find out more about the how we champion the use of evidence in social policy and practice.