Table 3 Results from the deductive analysis for spread/scale text to the Framework for Going to Full Scale

Phase of scale-up: what phase of the scale-up process is the trial working at?

 Phase 1: set-up

Prepares the ground for introduction and testing of the intervention that will be taken to full scale.

Establishes an entry point for the planned intervention into the existing health system.

Includes a clear articulation of what needs to be scaled up and defines the ambition for “full scale.”

Initial test sites, early adopters, and potential “champions” of the intervention are identified.

Materials and training are designed with scalability in mind.

Acknowledgment that some tailoring may be required to meet site-specific needs.

More set-up/planning/pre-testing needed before scaling up (connects to phase 3).

We purposely designed this intervention to be relatively low in intensity but wider in reach, maximizing generalizability and dissemination possibilities. [48]

The intervention in which prescribers received patient leaflets and clinic posters as well as the interactive workshops is low cost to scale-up but it did require intensive development and pretesting with end users. [49]

We ensured it was as comparable and structured as possible while also allowing site-specific tailoring to address differing clinic needs and to allow a sense of ownership of the project by the healthcare providers implementing the activities. [50]

We would now recommend more intensive field work involving iterative cycles of testing and refining interventions prior to scaling up for definitive evaluation. [22]

 Phase 2: develop the scalable unit

An early test and demonstration phase.

Scalable unit: typically, a small administrative unit (e.g., sub-district/district or clinical ward/division) that includes key infrastructural components and relationship architecture that are likely to be encountered in the system at full scale.

If the ambition of scale is large (e.g., county, province, health system), a scalable unit could comprise multiple levels of care and the communities that are served by a large health system, or a divisional unit of care in a hospital setting or large clinic system.

Goes beyond the design phase by conducting small pilots to understand the intervention and potential for scale.

To mitigate this risk [that the trial was perceived as intrusive and disruptive to workflow], the solution was trialed in 3 sequential small-scale pilots. [51]

Before beginning the QI project, 1 author … piloted the training in 1 clinic. He refined the approach on the basis of physicians’ feedback and then prepared physician training leaders to deliver it through an in person “train the trainer” session. [19]

This eCRT showed that it was feasible to use the [clinical intervention] to evaluate interventions that may be readily scaled up to the population level. Feedback received in the eCRT process evaluation, together with evidence from other trials cited above, identifies ways to increase engagement in the intervention and increase effect sizes. [34]

By first testing different forms of nudges, we could optimize the design of the intervention before implementing and scaling it within the EHR, which involves the additional time and expense of programming new functionality. [52]

 Phase 3: test of scale-up

Testing the set of interventions to be taken to scale.

Spreads the intervention to a variety of settings that are likely to represent contexts that will be encountered at full scale.

The underlying theory of change and the change package from a successful early demonstration need to be tested in a broader range of settings before going to full scale.

Test necessary infrastructure (e.g., data systems and supply chain) required to support full-scale implementation and build the human capacity and capability (e.g., leadership, managerial, and frontline capacity needed to support the method being used to scale up).

Important opportunity to build the belief and will of leaders and frontline staff to support the changes.

Trials conducted before going to full scale. These are larger studies than just pilots (phase 2), as they had multiple sites, settings etc. and aim to be conducted in “real world” conditions.

Some differences between sites were found; some allowed for adaptation between sites.

(Discussion about infrastructure included below)

We investigated in a nationwide trial the feasibility and effectiveness of a large-scale, quarterly prescription feedback intervention on antibiotic use in primary care over 2 years using routinely collected claims data in Switzerland. [53]

We wanted to build on the experiences from the work by Verstappen et al. and undertake a large-scale implementation of the strategy in a pragmatic trial with much room for the LQICs to adapt the strategy to their own needs and without any researchers being present embedded within the existing network of LQICs under real-world conditions, increasing confidence in wider applicability to routine general practice settings. [44]

We sought to evaluate the efforts of a large pediatric health care system to improve HPV vaccination coverage among adolescent patients using existing, research-based materials that were adapted to reflect local stakeholders and settings. … Understanding how large health care systems conduct HPV vaccination QI is important given the potential for system-wide efforts to influence many clinics, providers, and patients. [19]

This highly pragmatic trial showed the effectiveness of a low intensity feedback intervention delivered by the NHS and implemented across nearly all practices in three geographical areas. With the rapid growth of patient level datasets based on electronic medical records or pharmacy claims data, the potential for feedback interventions to improve prescribing safety is considerable, and many healthcare systems could deploy similar interventions now. [54]

 Phase 4: go to full scale

Unfolds rapidly to enable a larger number of sites to adopt and/or replicate the intervention.

A well-tested set of interventions, supported by a reliable data feedback system, is adopted by frontline staff on a larger scale.

The focus is on rapid uptake of the intervention through replication.

Intervention is delivered at scale (population-level, full health system, across a province or country, etc.).

After a careful scaling of the intervention, ample communication, and stakeholder support, we were able to perform a large-scale randomized controlled trial covering all Australian states. [51]

Our study shows that quarterly provided prescription feedback over 2 years is possible at low costs on a nationwide scale. [53]

Our large-scale nationwide study results extend those of a recent single centre study showing ADR improvement with a short educational intervention. [55]

A population-wide, randomised, intervention trial of audit and feedback to more than 1400 community pharmacies. [47]

Adoption mechanisms

 Better ideas

Ideas that are designed for scalability.

Evident superiority of the intervention.

Simplicity.

Alignment with the culture of the new implementers.

Focused on the initial ideas/principals used to inform the trial design.

Building or tailoring based on the literature.

We conducted a rapid systematic review to put the results in context, specifically focusing on large, countrywide approaches not involving elements that would be difficult to be implemented on a large scale (such as on-site visits or educational elements). [53]

We tailored the intervention to conform to principles identified in the literature as associated with improvements in processes and outcomes of care in the ICU: 1) an effective intervention in the ICU must be multifaceted, incorporating education, protocols, and feedback directed at multiple levels of providers … 3) the intervention must be in a format that can be exportable and generalizable to other institutions. [56]

By first testing different forms of nudges, we could optimize the design of the intervention before implementing and scaling it within the EHR, which involves the additional time and expense of programming new functionality. [52]

 Leadership

The capacity for leading large-scale change needs to be developed as part of the scale-up process. Leaders can be coached to understand the difference between simply raising awareness of a new practice and what it takes to lead and ensure its broad adoption.

Not found

N/A

 Communication

Communicating the value of the intervention to both leadership and the implementers (frontline staff).

Not found

N/A

 Policy

The identification and/or development of regulatory or administrative policies.

Policy can have a supportive or disruptive effect.

Not found

N/A

 Culture of urgency and persistence

Consideration of why others would want to join the effort and whether there is a glaring gap in performance or an urgent need.

Checking the amount of will and energy needed to stay the course in bringing interventions to—and achieving results at—full scale. Levels of will and energy may fluctuate over time.

Only focused on urgency about the need for the intervention, rarely about the impact of this urgency.

That prescribers changed their practice so quickly, and to the extent of almost eliminating use of antimalarial drugs for non-malarial cases in the intervention arms can be interpreted in the context of an increasing national drive for parasite-based malaria diagnosis, with a country-wide scale-up of RDTs that has been ongoing since 2010, which could have raised awareness and readiness for change. [49]

Support systems (infrastructure)

 Human capability for scale-up

Scale-up will require team leaders who can use change management approaches to guide and mentor teams at the front line and improvement specialists who can lead and design QI-based programs for those who need additional training.

The project needs be able to communicate quantitative results and the underlying stories of success and challenge. Data managers need training in analytic and reporting capabilities that are best suited to QI methods (e.g., run charts and statistical process control).

Focus on implementation in “usual circumstances,” including needing minimal implementation support, and trying not to be labor intensive.

Less focus on specific skills of team leaders, data managers etc.

Comprehensive, whole-office-focused interventions are more time-intensive to implement on a large scale, and may involve contributions from non-revenue generating staff (e.g., administrative staff, ADHD care coordinators). [57]

The implementation of [study name] was challenging with the restrictions on logistics, time, and funding, especially when dealing with an intervention requiring behavioural changes and implementation in complex healthcare systems. [20]

We developed a team-based implementation and engagement model using both a physician expert and a practice facilitator because it quickly became clear that assigning sole responsibility to the physician expert for advising, communicating, and coordinating with change teams (at the clinic) was overly burdensome and not scalable. [30] forward citation from [31]

 Infrastructure for scale-up

Common structural considerations include:

- Additional tools (e.g., checklists, data capture systems)

- Communication systems (e.g., materials and messages, mentoring relationships, structured programs)

- Key personnel (e.g., data capturers, quality improvement mentors)

Focuses on embedding into existing infrastructure (EHR, existing resources, local talent etc.) to support scale-up.

Helpful to scale in systems where organizations use the same system (same EMR etc.).

Our intervention was a low-cost mechanism, built on existing infrastructure. [38]

System, structural, and organisational support for system-wide changes to enable implementation strategies to be rolled out and scaled up (e.g., legislation, resources, mechanisms for communication and collaboration between health sectors). [58]

Given that the national infrastructure needed to support program implementation already exists, widespread dissemination of a modified [name] program represents a unique opportunity to address geographic disparities in adolescent vaccination as well as the lackluster uptake of HPV vaccine nationally. [59]

Our experience suggests that adapting existing materials and harnessing local talent (in the form of physicians who are already high performers) are feasible in the context of a large pediatric health care system and should be considered by other systems as a way to extend reach. [19]

 Data collection and reporting systems

Reliable systems that regularly tracks and provides feedback on the performance of key processes and outcomes.

Large-scale implementation cannot occur or be sustained unless routine data systems are accurate, complete, and timely.

Data that tracks key processes and outcomes that are targeted by the intervention need to be shared frequently with frontline staff and system leaders to inform ongoing improvement.

Directly linked with the “infrastructure” theme since the focus was usually on using embedded data systems, including electronic health records, and open data platforms.

With the rapid growth of patient level datasets based on electronic medical records or pharmacy claims data, the potential for feedback interventions to improve prescribing safety is considerable, and many healthcare systems could deploy similar interventions now. [54]

Routinely collected, accumulating data in administrative data sets offers a cost-effective opportunity to implement and evaluate antimicrobial stewardship interventions at scale across large populations. [60]

Since the underlying data are all publicly available, feedback of this kind could be provided by many different interested parties. [61]

Open data platforms can provide a low-cost route for wide-scale audit and feedback. [38]

 Learning systems

A mechanism for collecting, vetting, and rapidly sharing change ideas or interventions.

Mainly focused on the benefits of implementation laboratories, clinical networks, or taking a “learning health systems” approach.

The [name] programme effectively represented a nascent ‘implementation laboratory’ embedded within 10 CCGs. It is possible to develop and test incremental ways of improving the delivery of health care that cumulatively both improve patient care and develop the scientific basis of health-care provision. … Embedding trials in an existing network or major improvement initiative facilitates recruitment and helps ensure ‘real-world’ generalisability. We recommend that researchers build collaborations with those responsible for large-scale regional or national improvement to establish implementation laboratories. [29]

This study demonstrates the benefits of health system–academic collaborations on delivery innovations and the ability to scale nudges when they are codeveloped between clinicians and health systems. [62]

Clinical networks are increasingly being viewed as a vehicle through which evidence-based care can be embedded into healthcare systems using a collegial approach to agree on and implement a range of strategies within hospitals. [58]

 Design for sustainability

Plan for the intervention to be sustained.

Covered in the “sustainability” coding

Quotes are about the need to consider sustainability and scalability.

Audit and feedback is a pragmatic, scalable intervention to improve antibiotic use, and when coupled with evaluation systems using administrative databases it could generate sustainable and large reductions in antibiotic use. [60]

These interventions should be designed to fit into routine primary care practice and policy settings to ensure effectiveness, sustainability, and scalability. [18]

Although a transient increase in thrombolysis rates was evident during the active phase of implementation support, the negative overall result of the [name] trial confirms the recognized challenge of delivering and sustaining health systems change and suggests the need for further implementation research into novel strategies for thrombolysis implementation at scale. [36]