Tailoring implementation strategies for evidence-based recommendations using computerised clinical decision support systems: protocol for the development of the GUIDES tools
© Van de Velde et al. 2016
Received: 10 February 2016
Accepted: 25 February 2016
Published: 5 March 2016
A computerised clinical decision support system (CCDSS) is a technology that uses patient-specific data to provide relevant medical knowledge at the point of care. It is considered to be an important quality improvement intervention, and the implementation of CCDSS is growing substantially. However, the significant investments do not consistently result in value for money due to content, context, system and implementation issues. The Guideline Implementation with Decision Support (GUIDES) project aims to improve the impact of CCDSS through optimised implementation based on high-quality evidence-based recommendations. To achieve this, we will develop tools that address the factors that determine successful CCDSS implementation.
We will develop the GUIDES tools in four steps, using the methods and results of the Tailored Implementation for Chronic Diseases (TICD) project as a starting point: (1) a review of research evidence and frameworks on the determinants of implementing recommendations using CCDSS; (2) a synthesis of a comprehensive framework for the identified determinants; (3) the development of tools for use of the framework and (4) pilot testing the utility of the tools through the development of a tailored CCDSS intervention in Norway, Belgium and Finland. We selected the conservative management of knee osteoarthritis as a prototype condition for the pilot. During the process, the authors will collaborate with an international expert group to provide input and feedback on the tools.
This project will provide guidance and tools on methods of identifying implementation determinants and selecting strategies to implement evidence-based recommendations through CCDSS. We will make the GUIDES tools available to CCDSS developers, implementers, researchers, funders, clinicians, managers, educators, and policymakers internationally. The tools and recommendations will be generic, which makes them scalable to a large spectrum of conditions. Ultimately, the better implementation of CCDSS may lead to better-informed decisions and improved care and patient outcomes for a wide range of conditions.
Application of the best research evidence in clinical practice can improve the quality and safety of health care. Successfully translating evidence into practice requires that clinicians are aware of the evidence, agree with it, are confident about delivering the intervention and adhere to it in appropriate situations . Furthermore, patients should agree and adhere to the treatment . When there is more than one reasonable healthcare option, decision-making involves weighing the benefits and harms of the options, often with scientific uncertainty, and the preferences of the patients. Unfortunately, there is often a gap between the recommended care and the care that patients receive, and patient adherence to appropriate care can be poor . Furthermore, some healthcare interventions may not be needed or may even be harmful. Finally, expenses can decrease if care options are chosen according to their comparative cost-effectiveness .
A computerised clinical decision support system (CCDSS) is an information technology to aid clinicians and patients in making healthcare decisions, based on patient-specific data . This is a broad term, and CCDSS comes in many types and functions . It can be Internet-based, installed on a local personal computer or a networked electronic health record or function on a handheld device. The computer-generated decision support can be provided on screen or on paper. By linking healthcare providers with relevant guidelines at the point of care, CCDSS can improve the adherence to evidence-based recommendations . CCDSS can also empower and motivate patients by providing patient-directed advice and improve disease self-management [6, 7]. Further, CCDSS could facilitate shared decision-making by healthcare providers and patients together by providing decision aids [8, 9].
Hopeful governments have made substantial investments in healthcare information technology with decision support capabilities [10, 11]. It has become apparent, however, that the required investment is greater than initially planned and the results of many initiatives fall short of expectation [12, 13]. Systematic reviews of CCDSS have reported modest improvements in the healthcare process [14, 15]. Even the newest generation of CCDSS, touted as evidence-based and fully integrated in the electronic health record, deliver only modest reductions in morbidity .
Despite popular claims, the mere integration of CCDSS in electronic health records is not sufficient to obtain clinical benefit. Some studies estimate that alerts are ignored in 49 to 96 % of the cases . Even though CCDSS can potentially improve the quality of care, it also holds the risk of new types of adverse effects, sometimes leading to patient harm [18, 19]. Unintended consequences of CCDSS come in various categories, with a report mentioning 47 types. Alert fatigue is a well-known example; less well known is automation bias where users tend to over-rely on computer output [20, 21]. Over 380 examples have been described for computerised provider order-entry systems, which are often integrated with CCDSS . Furthermore, recommendations provided by CCDSS are not always up-to-date or evidence-based and failing to meet standards for trustworthiness, well agreed upon in the guideline community . The benefits for patients remain uncertain if these problems are not addressed.
The Tailored Implementation for Chronic Diseases (TICD) project is developed of methods to tailor knowledge implementation interventions in chronic illness care  consistent with evidence of the benefits of tailored implementation . In tailored implementation, the strategies for knowledge translation are based on an assessment of determinants of healthcare practice to achieve desired changes . The TICD project, however, did not address the specifics of CCDSS implementation.
Current CCDSS implementations are often only superficially tailored to local context and do not always involve the users, neither the patients nor the health professional [26, 27]. Although multiple determinants of successful development and use of CCDSS have been proposed [5, 28, 29], some have been contested again . As stated in a recent editorial by the lead author of a comprehensive systematic review, we are only beginning to understand what factors make the implementation of decision support effective and it might be necessary to chart a new course . Caution is needed since the implementation of CCDSS systems is growing substantially .
Therefore, with the Guideline Implementation with Decision Support project (GUIDES), we aim to improve the impact of CCDSS through optimised implementation, based on high-quality and trustworthy decision support content . Here, we describe our methods to (1) investigate the factors that determine successful implementation strategies for evidence-based recommendations with CCDSS, (2) develop tools such as checklists, worksheets, and practical recommendations to address these factors and (3) validate the utility of these tools through the development of a tailored implementation strategy.
A review of research evidence and frameworks on the determinants of implementing recommendations using CCDSS
A synthesis of a comprehensive framework for the identified determinants
The development of tools for use of the framework and
- 4.Pilot testing the utility of the tools through the development of a tailored CCDSS intervention. Finally, the results from the actions described here will inform the design of a large-cluster randomised controlled trial to evaluate the effectiveness of the tailored CCDSS and for identifying optimal implementation strategies in computerised decision support. Figure 1 provides a schematic overview of the GUIDES project.
The GUIDES author group includes experts from different countries (Belgium, Canada, Finland and Norway) with a strong commitment to evidence-based medicine and broad expertise related to the development, implementation and evaluation of CCDSS [12, 24, 32–36]. Two authors (IK, POV) are also involved in the development of systems for decision support, which potentially could create a conflict of interest. If a conflict would arise for specific implementation determinants, these experts will not participate in the discussion on how to develop guidance for this item . The authors will also collaborate with an international expert group to provide input and feedback on the tools.
Review of research evidence and frameworks
Two researchers (SVDV, SF) will use systematic methods to review frameworks, systematic reviews, process evaluations and qualitative evidence pertaining to factors for successful CCDSS implementation. The research questions are: (1) Which factors contribute to the successful implementation of CCDSS; (2) Which frameworks are being used to organise these factors and (3) What is the evidence for the impact of these factors?
We focus on the implementation of evidence-based recommendations using CCDSS, but we also include research evidence and frameworks for the implementation of CCDSS in general. We consider the different types of CCDSS with any objective (e.g. diagnosis, treatment, test ordering, screening), in any healthcare setting and directed at healthcare professionals and/or patients. From a holistic point of view, we also include papers on the implementation of health information systems in general, if CCDSS is included in the scope. We define success as any desired change in the effectiveness, safety, efficiency, responsiveness, or equity of health services, based upon the outcomes as proposed by the Cochrane Effective Practice and Organisation of Care (EPOC) review group [38, 39]. If such data is available, we also consider success as reduction in costs or more efficient use of resources. We exclude papers about sending reminder messages for attendance at upcoming healthcare appointments, studies on health information systems that only target patients or studies on Internet-based treatment of specific conditions.
We define a framework as any overview or classification of determinants of the successful implementation of CCDSS. Some known examples of general frameworks include the Five Rights model or the ten commandments for effective CCDSS [5, 29]. Frameworks will be included if they allow the extraction of data that can be formulated as determinants for the implementation of CCDSS. We exclude frameworks if they are too specific (e.g. only locally applicable or limited to a specific CCDSS product) or too broad to operationalise them into a checklist.
The screening for systematic reviews will focus on reviews that allow the identification of factors related to successful implementation of CCDSS based on randomised controlled trials. Given that multiple systematic reviews are available, we will select the most recent reviews, those with a broad scope that includes multiple settings, clinical domains or CCDSS functions and those of the best methodological quality. When excluding reviews, we will check that no important information on potential factors is lost.
Further, we will identify process evaluations or qualitative studies that followed trials on the effectiveness of CCDSS [40, 41]. Such studies can provide valuable insights in the intended or unintended effects of an intervention. We define qualitative studies as studies that used collected data using focus groups, individual interviews, observation and document analysis, and that used qualitative methods to analyse the data [42, 43]. If systematic reviews of qualitative studies or process evaluations are available, these are also included.
After an initial screening for factorial trials on CCDSS success features, we decided to also conduct a systematic review of randomised trials that compare the impact of different CCDSS determinants. This also includes trials on the effect of adjacent interventions (e.g. academic detailing) to increase the success of CCDSS. A protocol for this systematic review is registered in the PROSPERO database (CRD42016033738).
The search strategy is described in the Appendix. Our own files and the reference lists of relevant papers will supplement this search. To select the studies, the two reviewers will screen titles and abstracts of all the retrieved references. One author (SVDV) will then screen full texts and further exclude irrelevant studies. Any disagreements will be resolved by discussion with a third person (POV). We do not apply any language restrictions.
Synthesis of results in a comprehensive framework
The reviewers will independently assess the selected frameworks according to desirable framework attributes as defined in the TICD project .
One reviewer (SVDV) will extract the identified domains and determinants from each selected paper and will use the list of identified determinants and domains to compose a draft framework. To construct this, we will expand on the TICD checklist . When the specificity of CCDSS requires this, we will adapt existing items, add new concepts or exclude irrelevant items. For every included factor, we foresee a short explanation. SF will quality check the new aggregated list by comparing it with the TICD checklist and the list of factors identified in the review.
If systematic reviews or reviews of qualitative studies provide evidence for the importance of a specific determinant, we will indicate the confidence we have in the evidence as high, moderate or low. Depending on the type of evidence, we will assess this by means of the GRADE or CerQual approach [42, 45]. To evaluate the quality, two reviewers will independently apply the ROBIS tool for systematic reviews and the CerQual tool for reviews of qualitative evidence [42, 46].
Next, we will invite experts within the field and CCDSS developers or implementers to evaluate the preliminary framework. To compile the list of people, we will include the corresponding authors of every relevant paper found in the review phase and combine this with the suggestions by the members of the GUIDES project. It is our goal to include persons across different country settings. Experts will be requested to declare potential conflicts of interests. Every participating expert will receive the preliminary framework and the first part of a structured feedback form with desirable framework attributes (see Additional file 1). The feedback form has been tested previously during the TICD project. We will not use formal consensus methods during this consultation of the expert group. SVDV will summarise the scores per question and will make a thematic summary of open-ended comments or suggestions. SF will verify the accuracy of this procedure. The group of authors will then discuss the feedback and agree in consensus on the revisions that should be made.
Next, we will send the revised version to the expert group with the request to score independently each item in the framework on the question ‘How important is this determinant for the success of a CCDSS intervention’ by means of a Likert scale from 1 (not important) to 5 (very important). Then, for each item, we will calculate the median score, range and percentage of ‘4’ and ‘5’ scores.
Development of the GUIDES tools
Based on the new framework, the existing TICD worksheets, discussion among the authors and feedback from key experts, we plan to develop tools to support the tailoring of implementation strategies for recommendations with CCDSS. We will also integrate the findings from the DECIDE project, which develops and evaluates communication strategies to support evidence informed decisions .
The tools will contain an overview of CCDSS implementation success factors and best practices, checklists and worksheets regarding the identification of implementation determinants and the matching to implementation interventions. They will also contain practical recommendations on planning and running tailored CCDSS implementation interventions.
Each tool will include definitions and explanations on how to use it combined with examples. To ensure usability and usefulness of the tools, we will consider its comprehensiveness, consistency of framing, and usability. The group of experts will receive another invitation for feedback, this time on the tools that have been developed using part 2 of the feedback form (Additional file 1).
Pilot testing of the GUIDES tools
The tools will be tested through the development of a tailored implementation strategy with CCDSS for patients with knee osteoarthritis. From many potentially relevant conditions, we have chosen knee osteoarthritis as a prototype because it is associated with considerable disability in the general population, it is a high economic burden, causes a major public health problem and quality indicator pass rates are remarkably low [48–50]. The clinical information used in the CCDSS will come from high-quality clinical practice guidelines, systematic reviews and patient decision aids on knee osteoarthritis [51, 52].
We submitted the protocol for this pilot study as a separate publication in Implementation Science. In short, we will organise focus groups in Norway, Belgium and Finland to inform the development of a tailored CCDSS implementation strategy for knee osteoarthritis recommendations. We will test decision support both for situations related to strong and weak (or conditional) recommendations and for comorbidity-related issues. The experience from these applications will be used to fine-tune the tools.
To get multiple perspectives on the use of the GUIDES tools, we also plan to invite CCDSS developers and implementers to a workshop . We will collect their feedback and discuss within the author group if any adjustments to the tools are needed.
Further, we will use the GUIDES framework in a systematic review of trials on CCDSS success features (PROSPERO database registration number CRD42016033738). During the data extraction process, pairs of reviewers will independently code the identified trials using the determinants that are listed in the GUIDES framework. We will calculate inter-rater agreement and record the frequency of coding .
It is clear that substantial work is necessary to realise the full potential of CCDSS, and guidance for the optimal design and implementation of CCDSS is urgently required . This project will provide guidance and tools on methods of identifying implementation determinants and selecting strategies to implement evidence-based recommendations through CCDSS. We will make the GUIDES tools available to CCDSS developers, implementers, researchers, funders, clinicians, managers, educators and policymakers internationally. The tools and recommendations will be generic which makes them scalable to a large spectrum of conditions. Due to time and budgetary constraints, we can only pilot the GUIDES tools for one clinical condition. This is a limitation of the project since implementation determinants might differ between clinical topics.
Having patient information available in the electronic health record in a structured/coded format is considered to be important for the implementation of CCDSS . This allows healthcare professionals to record the patient characteristics and makes it possible for computers to process the data. However, the design of clinical documentation tools to bridge the gap between daily medical data and international nomenclatures and classifications remains a challenge. Furthermore, these tools need to be adopted by the healthcare providers and to date, significant amounts of patient data are not available in a structured/coded format . Overcoming this barrier is obviously a challenge that is wider than the objectives and the time span of this project.
This project receives funding from the EU’s Horizon 2020 Research and Innovation Programme under the Marie Sklodowska-Curie grant agreement no. 654981.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Scott IA, Glasziou PP. Improving effectiveness of clinical medicine: the need for better translation of science into practice. Med J Aust. 2012;197(7):374–8.View ArticlePubMedGoogle Scholar
- Scott I. Ten clinician-driven strategies for maximising value of Australian health care. Aust Health Rev. 2014;38(2):125–33. doi:10.1071/AH13248.View ArticlePubMedGoogle Scholar
- Haynes RB, Wilczynski NL, Computerized Clinical Decision Support System Systematic Review Team. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: methods of a decision-maker-researcher partnership systematic review. Implement Sci. 2010;5:12. doi:10.1186/1748-5908-5-12.View ArticlePubMedPubMed CentralGoogle Scholar
- Wright A, Sittig DF, Ash JS, Feblowitz J, Meltzer S, McMullen C, et al. Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems. J Am Med Inform Assoc. 2011;18(3):232–42.View ArticlePubMedPubMed CentralGoogle Scholar
- Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10(6):523–30. doi:10.1197/jamia.M1370.View ArticlePubMedPubMed CentralGoogle Scholar
- Quinn CC, Clough SS, Minor JM, Lender D, Okafor MC, Gruber-Baldini A. WellDoc mobile diabetes management randomized controlled trial: change in clinical and behavioral outcomes and patient and physician satisfaction. Diabetes Technol Ther. 2008;10(3):160–8. doi:10.1089/dia.2008.0283.View ArticlePubMedGoogle Scholar
- Coiera E, Lau AY, Tsafnat G, Sintchenko V, Magrabi F. The changing nature of clinical decision support systems: a focus on consumers, genomics, public health and decision safety. Yearb Med Inform. 2009:84–95.Google Scholar
- Agoritsas T, Heen AF, Brandt L, Alonso-Coello P, Kristiansen A, Akl EA, et al. Decision aids that really promote shared decision making: the pace quickens. BMJ. 2015;350:g7624. doi:10.1136/bmj.g7624.View ArticlePubMedPubMed CentralGoogle Scholar
- Elwyn G, Scholl I, Tietbohl C, Mann M, Edwards AG, Clay C, et al. “Many miles to go …”: a systematic review of the implementation of patient decision support interventions into routine clinical practice. BMC Med Inform Decis Mak. 2013;13 Suppl 2:S14. doi:10.1186/1472-6947-13-S2-S14.View ArticlePubMedPubMed CentralGoogle Scholar
- Murphy EV. Clinical decision support: effectiveness in improving quality processes and clinical outcomes and factors that may influence success. Yale J Biol Med. 2014;87(2):187–97.PubMedPubMed CentralGoogle Scholar
- E-Health for a Healthier Europe! Swedish Presidency of the European Union. 2009. https://joinup.ec.europa.eu/community/epractice/document/eu-ehealth-healthier-europe. Accessed 18 Jan 2016.
- Roshanov PS, Fernandes N, Wilczynski JM, Hemens BJ, You JJ, Handler SM, et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ. 2013;346:f657. doi:10.1136/bmj.f657.View ArticlePubMedGoogle Scholar
- Watson R. European Union leads way on e-health, but obstacles remain. BMJ. 2010;341:c5195. doi:10.1136/bmj.c5195.View ArticlePubMedGoogle Scholar
- Lobach D, Sanders GD, Bright TJ, Wong A, Dhurjati R, Bristow E. et al. Enabling health care decisionmaking through clinical decision support and knowledge management. Evid Rep Technol Assess (Full Rep). 2012(203):1–784.Google Scholar
- Shojania KG, Jennings A, Mayhew A, Ramsay CR, Eccles MP, Grimshaw J. The effects of on-screen, point of care computer reminders on processes and outcomes of care. Cochrane Database Syst Rev. 2009;3:CD001096. doi:10.1002/14651858.CD001096.pub2.Google Scholar
- Moja L, Kwag KH, Lytras T, Bertizzolo L, Brandt L, Pecoraro V, et al. Effectiveness of computerized decision support systems linked to electronic health records: a systematic review and meta-analysis. Am J Public Health. 2014;104(12):e12–22. doi:10.2105/AJPH.2014.302164.View ArticlePubMedPubMed CentralGoogle Scholar
- van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13(2):138–47. doi:10.1197/jamia.M1809.View ArticlePubMedPubMed CentralGoogle Scholar
- Myers RB, Jones SL, Sittig DF. Review of reported clinical information system adverse events in US Food and Drug Administration databases. Appl Clin Inform. 2011;2(1):63–74. doi:10.4338/ACI-2010-11-RA-0064.View ArticlePubMedPubMed CentralGoogle Scholar
- Coiera E, Westbrook J, Wyatt J. The safety and quality of decision support systems. Yearb Med Inform. 2006:20–5.Google Scholar
- Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Some unintended consequences of clinical decision support systems. AMIA Annu Symp Proc. 2007:26–30.Google Scholar
- Goddard K, Roudsari A, Wyatt JC. Automation bias—a hidden issue for clinical decision support system use. Stud Health Technol Inform. 2011;164:17–22.PubMedGoogle Scholar
- Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: findings from a mixed methods exploration. Int J Med Inform. 2009;78 Suppl 1:S69–76. doi:10.1016/j.ijmedinf.2008.07.015.View ArticlePubMedGoogle Scholar
- Qaseem A, Forland F, Macbeth F, Ollenschlager G, Phillips S, van der Wees P, et al. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156(7):525–31. doi:10.7326/0003-4819-156-7-201204030-00009.View ArticlePubMedGoogle Scholar
- Wensing M, Oxman A, Baker R, Godycki-Cwirko M, Flottorp S, Szecsenyi J, et al. Tailored Implementation For Chronic Diseases (TICD): a project protocol. Implement Sci. 2011;6:103. doi:10.1186/1748-5908-6-103.View ArticlePubMedPubMed CentralGoogle Scholar
- Baker R, Camosso-Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, et al. Tailored interventions to address determinants of practice. Cochrane Database Syst Rev. 2015;4:CD005470. doi:10.1002/14651858.CD005470.pub3.Google Scholar
- Lobach DF. The road to effective clinical decision support: are we there yet? BMJ. 2013;346:f1616. doi:10.1136/bmj.f1616.View ArticlePubMedGoogle Scholar
- Miller A, Moon B, Anders S, Walden R, Brown S, Montella D. Integrating computerized clinical decision support systems into clinical work: a meta-synthesis of qualitative research. Int J Med Inform. 2015. doi:10.1016/j.ijmedinf.2015.09.005.PubMedGoogle Scholar
- Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765. doi:10.1136/bmj.38398.500764.8F.View ArticlePubMedPubMed CentralGoogle Scholar
- Sirajuddin AM, Osheroff JA, Sittig DF, Chuo J, Velasco F, Collins DA. Implementation pearls from a new guidebook on improving medication use and outcomes with clinical decision support. Effective CDS is essential for addressing healthcare performance improvement imperatives. J Healthc Inf Manag. 2009;23(4):38–45.PubMedPubMed CentralGoogle Scholar
- Coiera E, Aarts J, Kulikowski C. The dangerous decade. J Am Med Inform Assoc. 2012;19(1):2–5. doi:10.1136/amiajnl-2011-000674.View ArticlePubMedGoogle Scholar
- GUIDES project website. 2015. http://www.guidesproject.org/. Accessed Jan 19 2016.
- Vandvik PO, Brandt L, Alonso-Coello P, Treweek S, Akl EA, Kristiansen A, et al. Creating clinical practice guidelines we can trust, use, and share: a new era is imminent. Chest. 2013;144(2):381–9. doi:10.1378/chest.13-0746.View ArticlePubMedGoogle Scholar
- Van de Velde S, Vander Stichele R, Fauquert B, Geens S, Heselmans A, Ramaekers D, et al. EBMPracticeNet: a bilingual national electronic point-of-care project for retrieval of evidence-based clinical guideline information and decision support. JMIR Res Protoc. 2013;2(2):e23. doi:10.2196/resprot.2644.View ArticlePubMedPubMed CentralGoogle Scholar
- Kortteisto T, Komulainen J, Makela M, Kunnamo I, Kaila M. Clinical decision support must be useful, functional is not enough: a qualitative study of computer-based clinical decision support in primary care. BMC Health Serv Res. 2012;12:349. doi:10.1186/1472-6963-12-349.View ArticlePubMedPubMed CentralGoogle Scholar
- Makela M, Kunnamo L. Implementing evidence in Finnish primary care. Use of electronic guidelines in daily practice. Scand J Prim Health Care. 2001;19(4):214–7.View ArticlePubMedGoogle Scholar
- Varonen H, Kortteisto T, Kaila M, Group EBS. What may help or hinder the implementation of computerized decision support systems (CDSSs): a focus group study with physicians. Fam Pract. 2008;25(3):162–7. doi:10.1093/fampra/cmn020.View ArticlePubMedGoogle Scholar
- Hirsh J, Guyatt G. Clinical experts or methodologists to write clinical guidelines? Lancet. 2009;374(9686):273–5. doi:10.1016/S0140-6736(09)60787-X.View ArticlePubMedGoogle Scholar
- Crossing the quality chasm: a new health system for the 21st century. Washington, D.C.: National Academy Press; 2001.Google Scholar
- Acir N, Oztura I, Kuntalp M, Baklan B, Guzelis C. Automatic detection of epileptiform events in EEG by a three-stage procedure based on artificial neural networks. IEEE Trans Biomed Eng. 2005;52(1):30–40.View ArticlePubMedGoogle Scholar
- Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials. 2013;14:15. doi:10.1186/1745-6215-14-15.View ArticlePubMedPubMed CentralGoogle Scholar
- Scott SD, Rotter T, Hartling L, Chambers T, Bannar-Martin KH. A protocol for a systematic review of the use of process evaluations in knowledge translation research. Syst Rev. 2014;3:149. doi:10.1186/2046-4053-3-149.View ArticlePubMedPubMed CentralGoogle Scholar
- Glenton C, Colvin CJ, Carlsen B, Swartz A, Lewin S, Noyes J, et al. Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and child health: qualitative evidence synthesis. Cochrane Database Syst Rev. 2013;10:CD010414. doi:10.1002/14651858.CD010414.pub2.Google Scholar
- Glenton C, Lewin S, Scheel IB. Still too little qualitative research to shed light on results from reviews of effectiveness trials: a case study of a Cochrane review on the use of lay health workers. Implement Sci. 2011;6:53. doi:10.1186/1748-5908-6-53.View ArticlePubMedPubMed CentralGoogle Scholar
- Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, et al. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:35. doi:10.1186/1748-5908-8-35.View ArticlePubMedPubMed CentralGoogle Scholar
- Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64(4):401–6. doi:10.1016/j.jclinepi.2010.07.015.View ArticlePubMedGoogle Scholar
- Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2015. doi:10.1016/j.jclinepi.2015.06.005.PubMedGoogle Scholar
- Treweek S, Oxman AD, Alderson P, Bossuyt PM, Brandt L, Brozek J, et al. Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6. doi:10.1186/1748-5908-8-6.View ArticlePubMedPubMed CentralGoogle Scholar
- Askari M, Wierenga PC, Eslami S, Medlock S, de Rooij SE, Abu-Hanna A. Assessing quality of care of elderly patients using the ACOVE quality indicator set: a systematic review. PLoS One. 2011;6(12):e28631. doi:10.1371/journal.pone.0028631.View ArticlePubMedPubMed CentralGoogle Scholar
- Jamtvedt G, Dahm KT, Holm I, Odegaard-Jensen J, Flottorp S. Choice of treatment modalities was not influenced by pain, severity or co-morbidity in patients with knee osteoarthritis. Physiother Res Int. 2010;15(1):16–23. doi:10.1002/pri.452.PubMedGoogle Scholar
- Gronhaug G, Osteras N, Hagen KB. Quality of hip and knee osteoarthritis management in primary health care in a Norwegian county: a cross-sectional survey. BMC Health Serv Res. 2014;14:598. doi:10.1186/s12913-014-0598-x.View ArticlePubMedPubMed CentralGoogle Scholar
- Brosseau L, Rahman P, Toupin-April K, Poitras S, King J, De Angelis G, et al. A systematic critical appraisal for non-pharmacological management of osteoarthritis using the appraisal of guidelines research and evaluation II instrument. PLoS One. 2014;9(1):e82986. doi:10.1371/journal.pone.0082986.View ArticlePubMedPubMed CentralGoogle Scholar
- Nelson AE, Allen KD, Golightly YM, Goode AP, Jordan JM. A systematic review of recommendations and guidelines for the management of osteoarthritis: the chronic osteoarthritis management initiative of the U.S. bone and joint initiative. Semin Arthritis Rheum. 2014;43(6):701–12. doi:10.1016/j.semarthrit.2013.11.012.View ArticlePubMedGoogle Scholar
- Ash JS, Sittig DF, McMullen CK, Wright A, Bunce A, Mohan V, et al. Multiple perspectives on clinical decision support: a qualitative study of fifteen clinical and vendor organizations. BMC Med Inform Decis Mak. 2015;15:35. doi:10.1186/s12911-015-0156-4.View ArticlePubMedPubMed CentralGoogle Scholar
- Berlin A, Sorani M, Sim I. A taxonomic description of computer-based clinical decision support systems. J Biomed Inform. 2006;39(6):656–67. http://dx.doi.org/10.1016/j.jbi.2005.12.003.View ArticlePubMedGoogle Scholar
- Kalra D, Fernando B, Morrison Z, Sheikh A. A review of the empirical evidence of the value of structuring and coding of clinical information within electronic health records for direct patient care. Inform Prim Care. 2012;20(3):171–80.View ArticlePubMedGoogle Scholar
- Ho YX, Gadd CS, Kohorst KL, Rosenbloom ST. A qualitative analysis evaluating the purposes and practices of clinical documentation. Appl Clin Inform. 2014;5(1):153–68. doi:10.4338/ACI-2013-10-RA-0081.View ArticlePubMedPubMed CentralGoogle Scholar