The road from evaluation to public service reform

In 2014, in a report for the UK Cabinet Office, Professor Jonathan Shepherd introduced the concept of the evidence ecosystem. For the first time this integrated the processes of evidence generation, evidence synthesis, the production of guidance from this synthesis, and implementation of this guidance in public services. In the period post-COVID when public finances and services are stretched as never before, these foundations of reliable evidence and prompt reform are crucially important. Here, seven years on, he summarises his new report on the health of this ecosystem in 2020.

“Our safety, health, education, and much else depend on the effectiveness and efficiency of public services. Without a constantly maintained evidence ecosystem opportunities to improve services and minimise waste will be missed and interventions which do more harm than good will be retained.

The effects of the COVID-19 pandemic on national economies are substantial. In the UK, Bank of England estimates are of a 14% shrinkage1 and in the United States current forecasts, taking account of resumption of economic activity in the summer or autumn of 2020, project declines in GDP of between 2.4% and 8.7% for 2020 relative to 2019.

In this context, the efficient use of public resources is more critical than ever. During the pandemic, however, scientific evidence, the synthesis of this evidence, and guidance for government ministers and practitioners based on this have established themselves steadily more securely as a basis for decisions.

This process has been subject to continuous public and professional scrutiny and has not just survived but is seen as crucial to decision making by national governments and front line practitioners – for example in school classrooms, the NHS, care homes and police forces. It is now time to ensure that this process is robust in a public services context.

The last decade has seen substantial growth in the generation and synthesis of evidence of “what works” and what’s cost effective. Infrastructure to support this, exemplified by the new, vibrant, UK What Works Network with its sector specific What Works Centres and coordinating What Works Council, has also been built.

But this rapid expansion is not without its problems. A tsunami of unregulated, non-standardised evidence and evidence reviews from hundreds, perhaps thousands of interest groups, professional bodies and other organisations has flooded policy making.

A government minister recently reflected that he is “overwhelmed with evidence” while a special advisor observed that he is “quite cynical of evidence presented to me as everyone has ‘evidence’ to back themselves up.”

Even in the What Works Network, with its important guiding principles, a leading contributor noted that, “Each Centre has different evidence standards. Expecting policymakers to get to grips with each one is unreasonable.”

There is a parallel with the early years of the industrial revolution, when new, non-standardised products, literally the nuts and bolts of a new age, brought baffling complexity and began to limit progress. To help solve this problem Sir Joseph Whitworth standardised screw threads and machine measurement so that they could be manufactured and used in the same reproducible way in Liverpool, New York, and Madrid, massively increasing efficiency, innovation, and productivity.

The recommendations in my 2020 review of the evidence ecosystem seek to adapt this lesson from history to meet the challenges it now faces. The necessary international standards already exist. They need to be applied.

• Standards for evidence generation, evidence synthesis and guidance production should be set across the What Works Network and incorporated into the Network’s IMPACT principles. MHRA, CONSORT, Cochrane/Campbell, and AGREE standards should be the starting point in a systemic approach to this.
• Organisations which carry out systematic reviews of evidence of intervention effectiveness and cost benefit should, by earning recognition through an accreditation process, be able to demonstrate that they meet Cochrane/Campbell review standards.
• The NICE accreditation programme for guidance producers should be reopened to new applicants in the health and care sectors and adapted and extended across other What Works Centres and widely advertised in the sectors in which they operate.
• Technology appraisals, as carried out by NICE, should be used by other What Works Centres to assess technologies developed in their sectors, and the findings given statutory force, as with the findings of NICE technology appraisals in the NHS.
• The Research Assessment Framework in higher education should be adapted to facilitate assessment of the impact of What Works Centres.
• Methods used to increase safety in healthcare should be applied to ensure that distilled evidence and guidance derived from it prompts rapid change in the public sector.
• To increase responsiveness to authoritative guidance What Works Centres should develop formal relationships with service regulators and professional bodies in their sectors.

Whitworth realised that standardisation would transform engineering. Public service transformation is just as achievable if the evidence ecosystem is subject to the same discipline.

Professor Jonathan Shepherd, Crime and Security Research Institute, Cardiff University ShepherdJP@cardiff.ac.uk