Developing a checklist for guideline implementation planning: review and synthesis of guideline development and implementation advice

Abstract

Background

Developers, users and others have requested or advocated for guidance on how to plan for, and implement guidelines concurrent to their development given that existing resources are lacking such information. The purpose of this research was to develop a guideline implementation planning checklist.

Methods

Documents that described or evaluated the processes of planning or undertaking implementation were identified in several publications that had systematically identified such resources, and by searching medical literature databases (MEDLINE, EMBASE). Data that described implementation planning; how to develop guideline versions or tools that would support user implementation; and options and mechanisms for disseminating or implementing guidelines were independently extracted from eligible documents by the principal investigator and a trained research assistant. Data were integrated to create a unique list of guideline implementation planning processes and considerations.

Results

Thirty-five documents were eligible. Of these, 16 (45.7%) provided sparse information on implementation planning, 25 (71.4%) mentioned different versions or tools for implementation, and 30 (85.7%) listed options for dissemination or implementation. None provided instructions for operationalizing implementation strategies. Data were integrated into a multi-item Guideline Implementation Planning Checklist including considerations for implementation planning (12), development of implementation tools (8), types of implementation tools (12), and options for dissemination (11) and implementation (12).

Conclusions

Developers or users can apply the Guideline Implementation Planning Checklist to prepare for and/or undertake guideline implementation. Further development of the checklist is warranted to elaborate on all components. In ongoing research, we will consult with the international guideline community to do so. At the same time, guideline implementation is complex, so developers and users would benefit from training, and by including knowledge translation experts and brokers on implementation planning committees.

Background

Guidelines are documents that synthesize current evidence on how to most effectively organize and deliver health services for a given condition [1]. They inform healthcare decision-making and can serve as the basis for policy, planning, evaluation, and quality improvement. When newly developed, or when evaluation identifies suboptimal compliance, efforts are needed to promote awareness, acceptance, adoption, and adherence to guidelines. Such efforts include dissemination (posting on a web site, publishing in a journal, presenting information at a meeting) or implementation (purposeful strategies that employ educational, social, organizational, financial or technological means of promoting guideline use) [2]. However, guidelines are not always translated into policy or practice [3,4]. A systematic review of studies in which guideline use was evaluated revealed that adoption and adherence were low even when awareness of, and agreement with guidelines among target users was high [5]. This suggests that dissemination efforts are suitable for sharing guidelines but lack of, or poor implementation limits guideline use.

Guideline implementation is challenged by many issues. The effectiveness of most implementation strategies is small and inconsistent, limiting their impact on guideline adoption [6]. A variety of contextual factors at the individual, institutional and system level often co-exist and pose additional challenges to guideline implementation and use [7]. Promotional efforts by guideline developers may be constrained by lack of resources so implementation is often the responsibility of target users [8,9]. Regardless of whether implementation is undertaken by developers or users, implementation is further complicated by two key factors. One, while instruments exist to assess barriers of guideline use, or organizational capacity or readiness to adopt guidelines, these do not reliably identify the most appropriate implementation strategy for a given guideline [10,11]. Two, implementation planning most often occurs upon guideline completion. Implementation could be more successful if planning were concurrent rather than consecutive to guideline development so that the recommendations were clear and useable, target users were primed for adoption, and their needs and preferences, and insight on contextual factors could inform implementation planning [12]. This may also reduce the time required for guidelines to be adopted into policy or practice by avoiding a lengthy waiting period from guideline completion to implementation planning, and actual execution of implementation activities.

Developers, users and others have requested or advocated for guidance on how to plan and implement guidelines [13-15]. While taxonomies [16,17] and models [18] of guideline implementation strategies and approaches are available, and considerable research has synthesized primary studies to report on the effectiveness of guideline implementation strategies [19], these sources do not offer guidance on how to plan for the implementation of guidelines. Schunemann et al. recently issued a checklist for developing guidelines, however, it focused largely on planning for, and undertaking guideline development, and addressed implementation in brief following completion of guideline development [20]. Developers and users would benefit from information on the steps and considerations for guideline implementation planning. This would supplement the Schunemann et al. checklist [20] and support guideline implementation and use among developers and users, leading to improved use of guidelines in health care decision-making. The purpose of this research was to develop a guideline implementation planning checklist.

Methods

Approach

A working group representing guideline developers, implementers and researchers was assembled and met by teleconference on January 14, 2014 to discuss the type of resource that would be of greatest use to those implementing guidelines. The working group was drawn from members of the Guidelines International Network (G-I-N) Implementation Working Group. G-I-N is a global network comprised of 100 organizations and 131 individual members representing 48 countries from all continents (www.g-i-n.net). The intent was to create a checklist rather than a manual with detailed instructions and templates which would require considerable time and resources to initially develop, and for each required update. Furthermore, implementation must be tailored to context so a single manual could not provide advice that is broadly relevant. Instead, a checklist could be more quickly developed given our limited resources for doing so; draw upon useful resources that already existed; and provide quick, practical advice that could be broadly applied, and easily updated as needed. We examined the content of documents that described the process of planning for guideline implementation, and organized and summarized the information into a checklist. Ethical approval was not required as data were collected from publicly available resources. Although not a traditional systematic review, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) criteria guided reporting of the methods and findings (Additional file 1: Table S1) [21].

Sampling

Documents were identified in the reference lists of recently published studies that had systematically identified manuals and other relevant documents describing guideline development [20,22,23]. The studies described in these publications, while relevant, had focused on guideline development and may not have been comprehensive. Therefore, we searched OVID MEDLINE and EMBASE from 2004 to 2013 on January 25, 2014 (Additional file 2: Table S2) to identify additional studies, reports or manuals that focused on guideline implementation, or any guideline development manuals that may have been missed in previous studies. Search terms were informed by research that generated search strategies for implementation topics which optimized sensitivity and specificity [24,25]. The principal investigator and a trained research assistant independently reviewed the search results to assess eligibility. Eligible documents included English-language studies, reports or instructional manuals that were publicly available or published in a peer-reviewed journal, and described or evaluated the processes of planning or undertaking implementation. Those published from 2004 to 2013 were included so that recommended processes would be reasonably current. Guidelines that described clinical recommendations were excluded, as were single studies or systematic reviews evaluating the effectiveness of implementation strategies since our intent was to describe implementation planning and not to evaluate interventions. Manuals prepared by for-profit health care delivery or guideline development organizations, and meeting abstracts were not eligible. All items considered eligible by at least one reviewer were retrieved for full text screening during data extraction.

Data collection

Data were first extracted by the principal investigator. The full text of each eligible document was perused to identify, extract and tabulate any information that referred to or described implementation planning (defined as instructions for when and how to plan and prepare for implementation); implementation tools (defined as instructions for developing guideline content, versions or tools that support implementation); and dissemination and implementation (defined as strategies and instructions for distributing, sharing, promoting, and applying guideline recommendations). The location of the information within each document was also noted. The full extent of information available in each source was extracted, though not necessarily verbatim, except when it was too lengthy or detailed to duplicate. A research assistant checked extracted data against the original source to ensure that all relevant information was accurately and fully retrieved from each item.

Data analysis

The total number of eligible manuals produced by different types of organizations, and the number of manuals addressing different aspects of implementation were summarized. Extracted data were integrated to create a step-by-step list of unique processes and considerations for implementing guidelines from planning to execution.

Results

The screening process and results are outlined in Figure 1. The Schunemann [20], Vernooij [22] and Ansari [23] articles referenced 102, 35, and 19 relevant documents, respectively. The MEDLINE and EMBASE searches together retrieved 959 articles. Following the removal of duplicates, 1,076 titles were screened. Of these, 955 titles were eliminated (924 ineligible topic, 5 ineligible publication type, 9 published before 2004, and 17 not English language). Screening of the full text of 121 remaining items eliminated a further 86 (1 not publicly available, 85 implementation not addressed). As a result, 35 unique documents were eligible for review. The majority of these reflected guideline development methods used by professional societies or foundations (n = 22) or government agencies (n = 10) plus one each from a not-for-profit agency, academic organization, and a peer-reviewed research publication on guideline implementation.

Figure 1

PRISMA flow diagram of search results.

Data extracted from eligible documents are presented in Additional file 3: Table S3. Of the 35 documents, 16 (45.7%) provided information on implementation planning, 25 (71.4%) mentioned different versions or tools that were meant to support implementation, and 30 (85.7%) mentioned strategies that could be used to disseminate or implement guidelines or guideline implementation tools. Most resources offered sparse guidance for implementation planning, choosing and operationalizing dissemination and implementation strategies, or developing alternate versions or tools that supported implementation.

Data across all sources were integrated to create a list of unique steps and considerations for guideline implementation planning (Table 1). Key implementation planning actions included assembling an implementation planning team that may or may not be part of the guideline development group; assembling resources for implementation; auditing baseline practice; assessing barriers and interacting with stakeholders to gather contextual information that would inform the selection of dissemination and implementation strategies, and the development of guideline implementation tools; and the preparation of an implementation plan specifying strategies, roles, responsibilities, timelines, and measures by which to evaluate the implementation process and outcomes. A number of options for adjunct products to support user implementation were identified including alternate versions and implementation tools, though only one resource provided information about processes that could be used to develop implementation tools. Several resources offered similar options for dissemination and implementation. Frequently mentioned dissemination options included posting guidelines and implementation tools on a web site and publishing them in journals. The most frequently mentioned implementation options included educational meetings, and audit and feedback.

Table 1 Guideline implementation planning checklist

Discussion

We developed a Guideline Implementation Planning Checklist by conducting a comprehensive review of existing resources. This can be applied by guideline developers or users to prepare for and/or undertake implementation. Information was variable across documents, and, in general, quite sparse. In large part this was due to the fact that most resources focused on guideline development, and few were dedicated to guideline implementation. As a result, the Guideline Implementation Planning Checklist is brief. However, it provides developers or users with a framework for implementation planning that coincides with guideline development, can be adapted to context-specific factors and therefore may be broadly applicable, can be employed by guideline developers or users so its application is flexible and, as a synthesis of available information, it is more detailed than existing resources.

This research is limited in that our search strategy or screening may have failed to retrieve or identify all relevant resources, stringency of screening criteria may have excluded otherwise eligible documents, and we may not have accurately extracted or synthesized data. To mitigate these issues we performed a systematic search of previous comprehensive syntheses of guideline development and implementation documents, and of indexed databases for published healthcare research; and data extraction and synthesis was performed independently by two individuals which enhances reliability of the findings.

As noted, the content of the Guideline Implementation Planning Checklist is limited by the content of existing documents from which it was compiled which, in turn, is influenced by limited evidence for these processes. With respect to planning for implementation, there is no evidence on the effectiveness of planning steps or considerations, nor is it practical that such studies would be conducted. For example, the impact of forming an implementation team or developing an implementation plan on the conduct and outcomes of implementation planning is a logistical consideration, and would likely not be a priority for research funders. Evidence is mixed on the impact of assessing barriers as a means of choosing and tailoring interventions. For example, a Cochrane systematic review of 25 studies found that interventions tailored based on identified barriers were more likely to improve professional practice compared with no intervention or with dissemination of guidelines, however, methods used to identify barriers and tailor interventions were not described in eligible studies so how best to do this remains unclear [61]. Another systematic review of 63 studies across which the same instrument was used to assess barriers found that, although it identified numerous barriers, this was not reflected in the tailoring of implementation strategies, and was not associated with evidence adoption [62]. While several eligible documents recommended engaging stakeholders throughout the implementation planning process, few details were offered. Knowledge brokers can function as liaisons between those generating and applying evidence. Qualitative analysis of a randomized controlled trial revealed numerous ways in which knowledge brokers enhanced the capacity of public health officials to practice evidence-informed decision-making, however, the trial found their impact to be equivalent to that of evidence summaries so the authors recommended ongoing research to optimize their role [63]. The principles and practices of action research or integrated knowledge translation could be used to elaborate on how knowledge brokers could establish and nurture partnerships with stakeholders [64,65]. While evidence underlying the planning strategies identified in eligible documents may be limited, wide consultation with, and the consensus of guideline developers and implementers could be used to enhance the implementation planning component of the Guideline Implementation Planning Checklist until further evidence becomes available.

With respect to developing guideline implementation tools, this content could be expanded based on recent research in which Gagliardi et al. engaged the international guideline community to establish a 12-item framework of the desirable features of implementation tools, and identified 11 broad methodological steps, each with several sub-steps and considerations for developing implementation tools [66]. This knowledge can be applied to assess, and endorse or adapt existing implementation tools, or develop new implementation tools. Addition of these methods to the Guideline Implementation Planning Checklist may further enhance its relevance. However, further research is needed to develop guides or templates specific to different types of implementation tools.

With respect to dissemination, the list of options available in eligible documents and compiled in the Guideline Implementation Planning Checklist matches several items within the Professional Interventions domain of taxonomies of methods that have been used to implement guidelines including distribution and advertising of guideline material through mailing and mass media [16,17]. However, neither the taxonomies nor eligible documents comment on the comprehensiveness of the list of dissemination options, or on their effectiveness for achieving awareness and use of guidelines among different target audiences. In fact, the comparative effectiveness of such strategies is unclear. A Cochrane systematic review of printed educational materials, which included guidelines, reviewed data from 45 studies and concluded that, when used alone, such material can have a small impact on professional practice [67]. Another Cochrane systematic review of eight studies found that mass mailing of printed bulletins summarizing evidence may improve practice when there is a single, clear message, if the change is relatively simple to accomplish, and there is growing awareness by users of the evidence that a change in practice is required [68]. A systematic review of 42 articles reporting 38 dissemination studies by the Agency for Healthcare Research and Quality (AHRQ) similarly identified little comparative effectiveness research upon which to base the selection of dissemination strategies [69]. While evidence was limited, the AHRQ review found that reach strategies (dissemination using regular mail, email, or social or mass media) were as effective as ability strategies (providing users with additional resources or tools), and that multicomponent strategies involving a combination of research, ability and motivational strategies (opinion leaders, champions, etc.) appeared to be more effective than using one strategy alone, particularly for guideline adherence. Thus, some evidence exists to show that various means of dissemination can reach target audiences. This is further confirmed by a systematic review of studies that evaluated guideline implementation efforts which found that dissemination of guidelines through web sites, journal publications or other means resulted in awareness of, and agreement with guidelines among target users [5]. However, the same systematic review revealed that target users struggled with adoption, and other research involving interviews and focus groups with clinicians similarly revealed that they were frustrated with and uncertain about how to implement guidelines [14]. Therefore, guideline developers and users require implementation advice.

Eligible documents offered several implementation options but the compiled list was not as comprehensive as the options in published taxonomies of implementation strategies [16,17]. Moreover, the implementation strategies most frequently recommended in eligible documents were educational meetings and audit and feedback. Considerable evidence suggests that the impact of educational meetings is small and, while perhaps necessary as an initial means of raising awareness and engaging stakeholders, may need to be accompanied by other strategies that are informed by a contextual needs assessment [70]. Audit and feedback has also been investigated in a large number of trials and can achieve small to moderate impact when it is delivered by a supervisor or respected colleague, presented frequently, features both specific goals and action plans, aims to decrease the targeted behavior, baseline performance is low, and recipients are non-physicians [71]. In future research, the Guideline Implementation Planning Checklist could be improved by referring users to taxonomies of dissemination and implementation strategies [16,17], and systematic reviews that evaluate their effectiveness [6].

Given the conflicting evidence on the effectiveness of various implementation strategies, and the nuanced application of these strategies which requires contextual analysis and tailoring of strategies, it simply may not be possible to capture those details in a checklist or, indeed, even in a more detailed instructional manual such as those from which data were extracted. Therefore, the Guideline Implementation Planning Checklist could be applied in conjunction with additional strategies to develop implementation capacity among guideline developers and users. For example, a few eligible resources mentioned that the implementation team should include one or more knowledge translation experts. This may refer to researchers, managers or clinicians with expertise or experience in guideline implementation. It may also refer to knowledge brokers, who would liaise with both the guideline development or implementation group, and with target stakeholders to plan, develop and/or apply the most feasible and appropriate implementation strategies [72]. Alternatively, individuals charged with developing or implementing guidelines may require implementation training. Knowledge translation training programs have been established in the United States, United Kingdom and Canada [73-75]. Gagliardi et al. conducted a systematic review to generate guidance on how to develop a mentorship program for developing implementation capacity [76]. Through the G-I-N Implementation Working Group, we are pilot-testing the use of remote coaching by implementation experts as a means of providing guideline developers and implementers with guidance on an as-needed basis during the course of their initiatives.

In summary, this research identified that many of the recommended methods for planning guideline implementation may be intuitive but are not supported by evidence, and that lists of suggested options for dissemination and implementation are incomplete and lack descriptions of, or references to supporting evidence which would help developers or implementers choose which to apply. Further research is needed to generate evidence on how to assess barriers and use that information for implementation planning; describe the role of knowledge brokers or other knowledge translation experts in implementation planning; create guides or templates for developing specific types of implementation tools; and identify strategies that may be needed by developers and implementers in conjunction with the Guideline Implementation Planning Checklist to better plan guideline implementation. In ongoing research we will update the Guideline Implementation Planning Checklist by including references or links to relevant resources such as taxonomies of dissemination and implementation strategies, and systematic reviews that describe the effectiveness of those strategies. We will also consult with international guideline developers, implementers, and users to elaborate and improve the methods included in the Guideline Implementation Planning Checklist based on their needs and experiences. We may also conduct research in partnership with guideline developers, implementers and users to evaluate either concurrently or retrospectively their use of the Guideline Implementation Planning Checklist as a means of further refining its components.

Conclusions

Developers or users can apply the Guideline Implementation Planning Checklist to prepare for and/or undertake guideline implementation. Further development of the checklist is warranted to elaborate on all components. In ongoing research we will consult with the international guideline community to do so. At the same time, guideline implementation is complex, so developers and users would benefit from training, and by including knowledge translation experts and brokers on implementation planning committees.