The GUIDES checklist: development of a tool to improve the successful use of guideline-based computerised clinical decision support

Background Computerised decision support (CDS) based on trustworthy clinical guidelines is a key component of a learning healthcare system. Research shows that the effectiveness of CDS is mixed. Multifaceted context, system, recommendation and implementation factors may potentially affect the success of CDS interventions. This paper describes the development of a checklist that is intended to support professionals to implement CDS successfully. Methods We developed the checklist through an iterative process that involved a systematic review of evidence and frameworks, a synthesis of the success factors identified in the review, feedback from an international expert panel that evaluated the checklist in relation to a list of desirable framework attributes, consultations with patients and healthcare consumers and pilot testing of the checklist. Results We screened 5347 papers and selected 71 papers with relevant information on success factors for guideline-based CDS. From the selected papers, we developed a 16-factor checklist that is divided in four domains, i.e. the CDS context, content, system and implementation domains. The panel of experts evaluated the checklist positively as an instrument that could support people implementing guideline-based CDS across a wide range of settings globally. Patients and healthcare consumers identified guideline-based CDS as an important quality improvement intervention and perceived the GUIDES checklist as a suitable and useful strategy. Conclusions The GUIDES checklist can support professionals in considering the factors that affect the success of CDS interventions. It may facilitate a deeper and more accurate understanding of the factors shaping CDS effectiveness. Relying on a structured approach may prevent that important factors are missed. Electronic supplementary material The online version of this article (10.1186/s13012-018-0772-3) contains supplementary material, which is available to authorized users.

• Healthcare settings such as primary, secondary and tertiary care and such as small and large practices.
• Types of care practices including recommendations related to prevention, diagnostic tests and strategies, and treatment and follow-up for chronic and non-chronic conditions.
• Targeted uses such as healthcare provider-directed information, patientdirected information or both.
• Types of CDS functions such as data presentation, alerts, reminders, references to supporting information, computerised order entry systems, dose calculators, medication reviews, calculations of prediction rules and severity-of-illness assessments, shared decision-making tools, and populationbased functions.
Every CDS tool and setting has unique characteristics and we recognise that some elements in this checklist are not applicable to every context.

Background
In health systems, patient care must be based on the highest-quality relevant medical knowledge. Evidence-based guidelines help healthcare providers and patients to make well-informed healthcare decisions [1] . Unfortunately, there are gaps in the journey from the publication of evidence to the implementation of that evidence in clinical practice. [2,3] This may cause excessive preventable mortality and morbidity and impacts on healthcare costs.
Changes in healthcare practice are not achieved simply by publishing guidelines. [4] Specific implementation strategies are also needed, such as the use of computerised decision support (CDS). This technology uses patient-specific data (for example a diagnosis, a prescription, or a combination of data elements) to provide relevant medical knowledge at the point-of-need. Rationale for the GUIDES checklist Substantial investments have been made in healthcare information technology with decision support capabilities. However, despite popular claims, the benefits of CDS remain modest. [5,6] Many initiatives have fallen short of expectations and, [7] in some instances, unintended consequences have led to patient harm. [8,9] CDS is a complex intervention and its success (or failure) is affected by many factors. The implementation of CDS systems is growing, and is being applied to increasingly larger and more complex interventions. The first published trials of CDS date back to the 1970s. However, we are still developing an understanding of which factors, or combinations of factors, make CDS effective. [5,10] Caution is therefore needed to ensure that CDS is applied in the best possible way. [11] Given the complexity of CDS, we argue that a structured approach to implementation is important for professionals because it facilitates a deeper and more accurate understanding of which factors make CDS more (or less) effective . Relying on less structured, individual approaches can lead to key factors and influences being overlooked. The approach, we suggest, is the use of a checklist to guide CDS implementation. [12] Objectives of the GUIDES checklist The purpose of the GUIDES checklist is to help professionals to reflect, in a more structured way, on the factors affecting the success of CDS interventions. The overall aim of the checklist is to increase the success of guideline-based CDS in terms how they improve healthcare, health outcomes, cost management, and patient and provider satisfaction. [13] This checklist is designed to be flexible and applicable to many settings and uses, including different:

INTRODUCTION
While the focus of this work is on CDS, we recognise that CDS is not a magic bullet. Particular circumstances may require other implementation interventions or multifaceted strategies. [14] Any decision to use CDS or other, additional interventions should be based on an assessment of the determinants of healthcare practice that affect whether the desired changes can be achieved. [15] Scope of the GUIDES checklist The scope of the checklist includes: • Guidelines: because these are systematically developed statements to inform healthcare provider and patient decisions about appropriate healthcare for specific clinical circumstances. [1] CDS can implement both strong and weak recommendations* and these can be either in favour of or against a particular medical strategy. [16,17] Other types of evidence can also provide valuable CDS content and are also included in the scope.
• Computer-based decision support: because the use of a computer, rather than a manual process, to generate decision support has been identified as a more effective approach. [18] Manually generated decision support may provide a relatively cheap alternative to CDS, provided it is effective. [19] What the GUIDES checklist is not: • The checklist is not a tool for evaluating the quality of guidelines or evaluating the implementability of a guideline recommendation into CDS. Please refer to the AGREE II instrument and the GLIA instrument for more information on how these can be assessed. [20,21] • The checklist is not a tool for choosing CDS targets. For that purpose, please consult "Improving Outcomes with Clinical Decision Support: An Implementer's Guide". [22] • This diagram is adapted from the formula by Fixsen on successful uses of evidencebased programs in human service settings. [23] Each domain includes four factors, and there are 16 factors in total: • All the factors listed are important and all must be considered. The level of importance of the various factors may vary in specific circumstances (for example, "Governance of the CDS implementation is appropriate" may be less important in a small practice setting).
• In most instances, it is clear why each factor has been assigned to a particular domain. We recognise that in a few instances factors could be assigned to more than one domain (e.g. the factor "The system delivers the decision support directly to the right target person" has a relation to both the CDS system domain and the implementation domain). Some factors are also interrelated, such as "Stakeholders and users accept CDS", which is also affected by factors in the content, system and implementation domain.
• We provide a rationale explaining the importance of each factor, present sample questions to consider, and include an answer scale. The printed and online versions include space for notes.
• The rationale can either be an assumption or it can be supported by scientific evidence. Statements on the level of confidence that a factor affects the outcomes of a CDS initiative will be published in a separate systematic review report. [24,25] The GUIDES checklist is available in four different formats: in an outpatient, inpatient, emergency department, or intensive care unit setting).
References: [22, How to evaluate They also allow the same conclusions to be derived from the same data sets if the same How to evaluate How to evaluate Negative examples could include: • The healthcare providers perceive the CDS system to be a threat to their communication with patients or as a threat to their clinical autonomy, medical liability, or their professional privacy.
• The healthcare providers believe that CDS support is, in effect, a formulaic 'cookbook' approach to medicine and that the primary purpose of the advice is to decrease healthcare costs.
• The healthcare providers distrust the CDS system because they think it may fail to protect patient privacy and patient data. • Consider the following questions: • Is the required hardware available and what will the impact be of adding CDS to the existing information systems?
• Is it feasible to introduce CDS, given the current workload and the usual work processes?
• If necessary, can the workload or the work processes be changed or can the CDS system improve the workload or work processes? supports CDS; computers are also available in common work areas.
• The workflow of the healthcare providers is carefully studied before the introduction of the system and the system is customised to fit in with the current workflow.
•  [79,80] Providing such information can help healthcare providers and patients to make better-informed healthcare decisions, and helps them to critically appraise the decision support.
Users must be able to critically appraise the recommendations by, for example, checking the underlying original research.
To accomplish this, users should be able to move easily from the CDS advice to the findings of related systematic reviews and primary studies.

Examples
Positive examples could include: • The system has been pilot tested and the healthcare providers involved agree that the decision support advice is relevant and accurate.
• The decision support comes with an explanation about why the decision support was triggered.
• Some recommendations are well adhered to by experienced practitioners, but less adhered to by inexperienced providers.
The system makes it possible to customise who receives which decision support.
Negative examples could include: • The system provides advice for situations in which healthcare providers would also choose the recommended action without receiving any decision support.  3.  [88,89] To stimulate healthcare providers to use a CDS system, the time needed to add or correct patient data should be minimal.
Adherence is more likely if a system helps users to complete the recommended actions.
Healthcare providers prefer systems that interface easily with other computerised information systems. Better integration helps to prevent systems from becoming fragmented and having to change applications.
User needs and preferences may vary and customisable systems may therefore be more useful.
References: [22, 28-  Examples Positive examples could include: • The IT hardware can provide suitable, stable and fast CDS.
• After pilot testing a system, the healthcare providers found that its impact on clinical work was acceptable.
• The system interacts with other computerised information systems and healthcare providers can do most of their tasks within the same application.
• The CDS prepopulates a treatment order with recommended actions (drugs, tests,

or procedures)
Negative examples could include: • Patient data entered using the CDS is not automatically stored within the electronic health record of a patient and information must be entered twice.
• The system requires a lot of user effort (e.g. too many mouse clicks, scrolling, window changes, password prompts, etc.
• Limited interfaces between the decision support system and the order entry system creates a significant hurdle.
• Regularly CDS system updates create too much downtime.

DOMAIN 3: THE CDS SYSTEM
Notes: Follow-up actions: To ensure that the information provided to users is noticed and easy to process, CDS information displays should be eyecatching, intuitive, concise, and consistent.

The decision support is well delivered
Ambiguous or confusing information may lead to errors in decision making and should be avoided.
To improve user understanding, information could be presented, for example, in a layered format in which the key information is displayed first, and additional content is provided in expandable sub-layers. [92] CDS systems can include specific functions

Examples
Positive examples could include: • In a decentralised home care setting, the decision support is provided through automatic emails.
• The decision support is provided on paper when it is not possible to access CDS during a patient interaction.  References: [22,29,31,33,34,36,38,39,41,44,47,51,59,62,67,[82][83][84] Examples Positive examples could include: • The decision support is available to healthcare providers before their patient encounters and helps them to be better prepared when seeing patients. Negative examples could include: • Decision support only becomes available after a treatment choice is made and the data are entered in the system.
• The decision support is provided independently of contact with the patient and may be forgotten during the contact.
• The CDS suggests stopping an action that has already occurred and the healthcare professional has to interrupt the workflow to revert the initial action.  Negative examples could include: • The system is activated without any communication or information being provided. The users are not aware that a screen providing decision support is available.
• A training session was organised but many of the users did not attend.
• The system is down due to technical problems and the users have not been informed.
An implementation strategy that assesses the barriers and facilitating factors, and that plans actions accordingly, is likely to improve the success of quality improvement efforts.
Monitoring the outcomes of a CDS initiative is an important way to evaluating the impact of the system and to identify important deviations from evidencebased care. Some CDS systems allow clinical quality measures to be reported automatically. [98] References: [22, 28, 30-35, 37, 44, 45, 47, 51, 57, 62, 63, 73, 77, 83, 84, 99] How to evaluate Consider the following questions: • Is the implementation of the CDS stepwise?
• Is a plan in place to collect user feedback and to monitor system usage, performance and outcomes?
• Are malfunctions and other problems with use of the CDS quickly fixed?

Examples
Positive examples could include: • The implementation of the system is done in phases, initially using a smaller group of users, then a larger group, and finally a large user group.
• System logs are monitored and end-user input is collected to improve the system continuously.
•  Good governance requires that CDS is implemented sustainably and equitably with regard to resource-poor communities. [100] Without it, CDS implementation can potentially undermine health equity.