Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Computerized clinical decision support systems for drug prescribing and management: A decision-maker-researcher partnership systematic review

  • Brian J Hemens1,
  • Anne Holbrook2, 3, 4,
  • Marita Tonkin4,
  • Jean A Mackay1,
  • Lorraine Weise-Kelly1,
  • Tamara Navarro1,
  • Nancy L Wilczynski1,
  • R Brian Haynes1, 2, 3Email author and
  • the CCDSS Systematic Review Team
Implementation Science20116:89

DOI: 10.1186/1748-5908-6-89

Received: 6 April 2011

Accepted: 3 August 2011

Published: 3 August 2011

Abstract

Background

Computerized clinical decision support systems (CCDSSs) for drug therapy management are designed to promote safe and effective medication use. Evidence documenting the effectiveness of CCDSSs for improving drug therapy is necessary for informed adoption decisions. The objective of this review was to systematically review randomized controlled trials assessing the effects of CCDSSs for drug therapy management on process of care and patient outcomes. We also sought to identify system and study characteristics that predicted benefit.

Methods

We conducted a decision-maker-researcher partnership systematic review. We updated our earlier reviews (1998, 2005) by searching MEDLINE, EMBASE, EBM Reviews, Inspec, and other databases, and consulting reference lists through January 2010. Authors of 82% of included studies confirmed or supplemented extracted data. We included only randomized controlled trials that evaluated the effect on process of care or patient outcomes of a CCDSS for drug therapy management compared to care provided without a CCDSS. A study was considered to have a positive effect (i.e., CCDSS showed improvement) if at least 50% of the relevant study outcomes were statistically significantly positive.

Results

Sixty-five studies met our inclusion criteria, including 41 new studies since our previous review. Methodological quality was generally high and unchanged with time. CCDSSs improved process of care performance in 37 of the 59 studies assessing this type of outcome (64%, 57% of all studies). Twenty-nine trials assessed patient outcomes, of which six trials (21%, 9% of all trials) reported improvements.

Conclusions

CCDSSs inconsistently improved process of care measures and seldomly improved patient outcomes. Lack of clear patient benefit and lack of data on harms and costs preclude a recommendation to adopt CCDSSs for drug therapy management.

Background

Computerized clinical decision support systems (CCDSSs) algorithmically apply an electronic knowledge base to individual patient data to generate and present suggested actions intended to enhance health and healthcare [13]. CCDSSs for drug therapy management are used to facilitate evidence-informed medication use [4], reduce the incidence of harmful medication errors [5], and improve healthcare system efficiency [2, 4, 6]. In this review, we considered any CCDSS that provides recommendations to healthcare providers regarding the initiation, modification, monitoring, or discontinuation of drug therapy, based on the patient's characteristics. Systems designed solely to provide advice on the management of narrow therapeutic index drugs based on in vivo monitoring and pharmacokinetic principles (therapeutic drug monitoring [7]) were excluded because they are in a complementary in-depth review on therapeutic drug monitoring and dosing (submitted to Implementation Science). Variety in the structure and function of CCDSSs complicates methodologically sound investigations and comparisons of these interventions. A CCDSS may be integrated with one or more of electronic medical records (EMR), computerized provider order entry systems (CPOE) or electronic transmission of prescriptions to the point of dispensing. CCDSSs require input of patient data to deliver advice, and this may be accomplished via integration with patient information repositories or by manual entry. Optimally, the knowledge base of a CCDSS used to generate recommendations is evidence-informed, though this may not always be the case. Advice may be delivered to many kinds of providers through a variety of media across diverse settings of care. Systems may be developed 'in house' to meet the requirements of a specific organization or acquired from a commercial vendor.

Decision makers, clinicians, and patients should require sound evidence of CCDSS benefits, risks and costs prior to general adoption, as for any health intervention. Randomized controlled trials (RCTs) represent the gold standard for unbiased comparisons of alternative interventions [8].

Our previous review [2] included 24 RCTs of a CCDSS for drug therapy. Of the 13 trials measuring patient-important outcomes, only one detected benefit with a CCDSS. Small study size limited detection of change in patient-important clinical endpoints.

Lack of data on which to base overall conclusions on the effects of CCDSSs together with the increased pace of research in the field prompted this update of our previous review. To aid decision makers and providers, we evaluated the effects of CCDSSs on process of care and patient outcomes via cumulative synthesis of relevant RCTs. This review is one of a series of reviews considering the effects of CCDSSs across multiple application areas (therapeutic drug monitoring and dosing, primary preventive care, diagnostic test ordering, acute care management, and chronic disease management).

Methods

This review was conducted in accord with a published protocol http://www.implementationscience.com/content/5/1/12[9]. Some trials have been included in more than one review because they were relevant to more than one CCDSS intervention area. Specific details for the drug therapy review follow.

Research questions

For this review, we were primarily interested in determining: 1) Do CCDSSs improve performance on drug-related process of care measures or patient outcomes compared to usual care? 2) What features or characteristics of studies or systems are associated with improved process or patient measures? Based partly on our previous review [2], we expected studies demonstrating benefit from CCDSSs would: a) be integrated with an existing EMR or CPOE system (versus a standalone system); b) deliver decision support before or during a patient care encounter where the decision that is being supported was taken (versus supply of decision support at any other time); c) actively suggest treatments or other actions (versus supply general information or access to general information); d) be used in a patient care setting affiliated with an academic institution (versus any other setting); e) have developers of the CCDSS who were also the study investigators (versus study investigators not associated with developers); f) measure intermediate/surrogate patient outcomes (versus patient-important outcomes); g) describe higher rates of user satisfaction (versus no or low rates of user satisfaction).

Partnering with decision makers

This review was conducted in partnership with senior hospital managers and clinical leaders with an academic research team in the field of knowledge translation, from healthcare research to clinical practice. Decision makers provided key input as to the kind of data needed about CCDSS to drive effective choices and these needs were incorporated into the research plan where feasible.

Search strategy

The search methods employed have been described in detail elsewhere [9]. Briefly, a comprehensive search (2004 to 2010) of major biomedical databases (MEDLINE, EMBASE, Ovid's EBM Reviews, and Inspec) yielded citations for screening. Pairs of reviewers independently evaluated each citation and abstract. A third reader resolved disagreements where necessary. Inter-reviewer agreement on study eligibility was measured via unweighted Cohen's kappa (κ). Studies from our previous reviews were carried forward to this review if they met the inclusion criteria, effectively extending our search from database inception to 2010.

Study selection

Studies were included for review if they described an RCT comparing outcomes for a group of providers or patients using a CCDSS compared with care without the CCDSS. Non-experimental or quasi-experimental investigations were excluded. For inclusion, we required that independent providers or post-graduate trainee (e.g. medical residents) providers be identified as primary users of the CCDSS. The intervention CCDSS was required to provide patient-specific output in the form of assessments, management options, or recommendations to the clinical user. Studies were excluded if the system was used solely by students, only provided summaries of information for patients, only provided feedback on groups of patients without feedback about individual patients, only provided computer-aided instruction, or were used for image analysis. The six CCDSS intervention areas in this series of reviews used a common eligibility screening process [9] to identify reports of trials of CCDSSs for any purpose. Studies were then further screened to determine if the system provided advice regarding drug therapy.

Data extraction

Independent reviewers extracted key data concerning study methods, CCDSS and population characteristics, possible sources of bias, and outcomes in duplicate. Primary authors of each study were asked to review the extracted data for their study and offer comments on the extracted data.

Assessment of study quality

Included studies were evaluated on five dimensions of quality--including concealment of allocation, appropriate unit of allocation, appropriate adjustment for baseline differences, adequate follow-up, and appropriate outcome assessment--to yield a 10-point methods score [9].

Assessment of CCDSS intervention effects

Outcome selection and improvement determinations

Each included trial describing a CCDSS that provided advice exclusively or predominantly about drug therapy was classified as drug therapy management only (Rx-only). Systems that gave advice on drug therapy as part of a more complex intervention were categorised as 'multi-faceted' CCDSSs. Improvement was considered to have occurred where 50% or more of the selected outcomes showed a benefit with a CCDSS compared to control. To determine whether improvement occurred, all outcomes were selected from the first of: primary, then pre-specified, then any outcome(s), as defined by study authors (i.e., if a primary outcome was reported for a trial this was used to determine improvement to the exclusion of any other reported outcomes). Where no outcomes were defined as primary, but the study reported a sample size calculation for an outcome, we defined that outcome as primary. These criteria are more specific than those used in our previous review [2]; therefore, the assignment of effect was adjusted for some studies included in the 2005 review. Process of care outcomes for multi-faceted CCDSS studies were selected only if they were clearly drug-related. Multi-faceted systems that reported a patient outcome but did not report a drug-related process of care outcome intermediary were excluded as non-responsive to our research questions. Where there were multiple intervention arms, the arm testing the most sophisticated CCDSS was used to determine improvement. Two reviewers, working independently and blinded to study results, classified trials as drug treatment-only or multi-faceted, and initially identified the outcomes used to determine improvement, with disagreements resolved by consensus.

Data synthesis and analysis

Data were summarized using descriptive summary measures, including proportions for categorical variables and means (± SD) for continuous variables. For interpretation, a 2-sided p < 0.05 indicated statistical significance. For individual studies we report the measures of association and p -values reported in the studies.

We did not attempt a meta-analysis because of differences across studies of participants, settings, disease conditions, interventions, and outcomes. Tests of association between study and CCDSS factors and improved outcomes were tested using the univariate Fisher's exact test. Multivariate analyses were conducted using multinomial logistic regression. All analyses were conducted using SPSS v. 17.

Sensitivity analyses were conducted to determine if the class of outcome selected to judge improvement affected our results. We also identified cluster randomized trials where units of allocation and units of analysis were appropriately matched or mismatched. The proportions of successful trials with matched versus mismatched units were compared.

Results

A total of 14,952 possibly relevant records were identified [9]. After excluding duplicate records, 14,188 records were screened to yield 329 articles eligible for full-text screening. Of those, 166 trials met our criteria for a CCDSS; Cohen's κ for reviewer agreement on trial eligibility was 0.93 (95% confidence interval [CI], 0.91 to 0.94). Initially, 71 trials were judged relevant to drug therapy management. Six of these trials [1015] were excluded because they studied a multi-faceted CCDSS that included drug therapy, but did not report any drug-related process outcomes. A total of 65 trials reported in 74 papers were included [1689] (see Figure 1). Twenty-four RCTs [6062, 6567, 69, 7176, 78, 8089] had been included in the previous version of our review [2]. Study authors confirmed or supplemented our data extraction for 53 of 65 included studies (82%) [1620, 23, 2533, 3539, 4245, 4749, 5355, 57, 58, 6062, 6572, 7476, 78, 81, 8386, 88]. Forty-seven included studies contribute outcomes to this review as well as other CCDSS interventions in the series; four studies [49, 56, 76, 80] to four reviews, 16 studies [16, 19, 21, 28, 40, 44, 45, 53, 55, 59, 62, 64, 68, 69, 74, 7779, 82, 85, 89] to three reviews, and 27 studies [20, 22, 23, 26, 27, 29, 31, 32, 34, 35, 39, 4143, 4648, 50, 52, 54, 60, 63, 66, 70, 72, 75, 81, 8688] to two reviews; but we focused here on drug prescribing-relevant outcomes.
https://static-content.springer.com/image/art%3A10.1186%2F1748-5908-6-89/MediaObjects/13012_2011_Article_411_Fig1_HTML.jpg
Figure 1

Flow diagram of included and excluded studies for the update 1 January 2004 to 6 January 2010 with specifics for drug prescribing and management*. *Details provided in: Haynes RB et al. [9]. Two updating searches were performed, for 2004 to 2009 and to 6 January 2010 and the results of the search process are consolidated here.

Summary of trial quality is reported in Additional file 1, Table S1; system characteristics in Additional file 2, Table S2; study characteristics in Additional file 3, Table S3; outcome data in Additional file 4, Table S4 and Table 1, and other CCDSS-related outcomes in Additional file 5, Table S5.
Table 1

Summary of results for CCDSS trials of drug prescribing

Study

Methods score

Indication

No. of centres/providers/patients

Process of care outcomes

CCDSS Effecta

Patient outcomes

CCDSS Effecta

Studies of drug-only interventions

Field, 2009[17, 24]

7

Alerts to promote appropriate drug prescribing and monitoring for patients with renal insufficiency in long-term care.

1*/10/833

Appropriate final drug orders.

0

...

...

Fortuna, 2009[18]

10

Alerts to consider cost when prescribing hypnotics for adults in primary and urgent care.

14*/257/...

Change in hypnotic drug prescriptions.

+

...

...

Lo, 2009[20]

10

Alerts to order laboratory tests when prescribing new medications in primary care.

22*/366/2765

Ordering appropriate baseline laboratory tests.

0

...

...

Terrell, 2009[23]

9

Alerts to avoid inappropriate prescriptions in geriatric outpatients during discharge from emergency care.

1/63*/5,162

Emergency department visits resulting in prescriptions for ≥1 of the 9 targeted inappropriate medications.

+

...

...

Gurwitz, 2008[25]

7

Alerts to prevent adverse drug events in long-term care.

2*/37/1,118

...

...

Adverse drug events.

0

Hicks, 2008[26]

7

Reminders for management of hypertension in adults in primary care.

14*/.../2,027

Visits with adherence to guideline medication prescribing within one week.

+

BP controlled.

0

Matheny, 2008[28]

8

Reminders for routine medication laboratory monitoring in primary care.

20*/303/1,922

Ordering appropriate laboratory tests.

0

...

...

Reeve, 2008[30]

8

Reminders for use of aspirin in diabetic adults in primary care.

52*/150/258,979

Number of aspirin interventions in diabetic patients.

+

...

...

Davis, 2007[32]

9

Alerts for appropriate prescribing for upper respiratory tract infections in paediatric outpatients.

2/44*/12,195

Prescriptions consistent with recommendations.

+

...

...

Heidenreich, 2007[33]

7

Reminders to prescribe β-blockers for inpatients and outpatients with reduced LVEF.

3/50/1,546*

Patients with prescriptions for any β-blocker over nine months.

+

Survival free of heart failure hospitalization at one year.

0

Martens, 2007[34, 46]

9

Reminders for prescribing of antibiotics and drugs for asthma, COPD, and dyslipidaemia.

23*/53/3,496

Sum scores for appropriate prescribing of antibiotics, statins, cholesterol-lowering drugs or drugs for asthma or COPD.

0

...

...

Peterson, 2007[35]

4

Dosing advice for high-risk drugs in geriatric patients in a tertiary care academic health centre.

1/778/2,981*

Ratio of overall prescribed to recommended doses.

+

...

...

Raebel, 2007a[37]

8

Alerts to review potentially inappropriate prescriptions in ambulatory geriatric patients.

21/.../59,680*

Dispensings of targeted potentially inappropriate medications.

+

...

...

Raebel, 2007b[36]

7

Alerts to avoid teratogenic drugs in pregnant ambulatory patients.

.../.../11,100*

Dispensed category D or X drugs.

+

...

...

Thomson, 2007[38]

8

Presented information for treatment decisions about warfarin or aspirin therapy for patients with atrial fibrillation in primary care.

2/2/109*

Difference in decision conflict scale score immediately post-clinic.

+

Admission to hospital; adverse events including transient ischemic attack, bleeding or stroke followed by general practitioner consultation or admission; patient anxiety.

0

Verstappen, 2007[39]

6

Management of methotrexate for early rheumatoid arthritis in adult outpatients.

6/.../299*

...

...

Patients in remission for ≥3 months in first two years.

+

Feldstein, 2006a[22, 41]

10

Alerts to order laboratory tests when prescribing new medications in primary care.

15*/200/961

Baseline laboratory monitoring completed by day 25.

+

...

...

Judge, 2006[42]

8

Alerts to avoid inappropriate prescriptions in long-term care.

1*/27/445

Appropriate prescriber response to alerts.

0

...

...

Kattan, 2006[43]

8

Feedback provided for management of drug therapy for severe asthma in paediatric outpatients.

.../435/937*

Time to compliance with recommended therapy step; visits resulting in medication step-up after step-up recommendation.

+

Maximum symptom days per two weeks.

0

Palen, 2006[47]

9

Reminders for laboratory monitoring based on medication orders in primary care.

16/207*/26,586

Overall compliance with ordering the recommended laboratory monitoring.

0

...

...

Paul, 2006[48]

10

Recommendations for empiric antibiotic treatment in hospital inpatients.

15*/.../2,326

Overall rate of appropriate antibiotic treatment.

+

Duration of hospital stay or fever; 30-day mortality.

0

Derose, 2005[50]

7

Reminders to prescribe ACE-Is, angiotensin receptor blockers and/or statins in outpatients with diabetes or atherosclerosis.

.../1089/8,557*

Patients with prescriptions for at least one of ACE-I, angiotensin receptor blocker, or statin.

+

...

...

Heidenreich, 2005[51]

6

Reminders to prescribe ACE-I or alternative for inpatients and outpatients with reduced LVEF.

1/.../600*

Patients with prescriptions for ≥ moderate daily dose of ACE-I or appropriate alternative.

0

Mortality; renal function; creatinine; systolic BP; diastolic BP.

0

Raebel, 2005[54]

8

Alerts to order laboratory tests when prescribing new medications in primary care.

.../.../400,000*

Laboratory test completed at time prescription is dispensed.

+

...

...

Krall, 2004[58]

8

Alerts to prescribe of low dose aspirin therapy in primary care.

.../100*/10,972

Provider response to alerts (prescribe aspirin or document contraindication).

+

...

...

Ansari, 2003[61]

7

Alerts to prescribe β-blockers for patients with heart failure in primary care.

1/74*/169

Initiated or uptitrated and maintained on β-blockers; patients at target β-blocker dose.

0

Proportion of patients hospitalised or with emergency department visits; deaths.

0

Filippi, 2003[62]

7

Reminders to prescribe acetylsalicylic acid or other antiplatelet agents to diabetic primary care patients.

.../300*/15,343

Antiplatelet drug prescription.

+

...

...

Tamblyn, 2003[65]

7

Alerts to avoid inappropriate prescriptions in geriatric outpatients.

.../107*/12,560

Proportion of new inappropriate prescriptions; discontinuation of pre-existing inappropriate prescriptions.

+

...

...

Weir, 2003[67]

8

Recommendations for appropriate prescribing of anti-platelet and/or anti-coagulant drugs following stroke or transient ischemic attack for in- and out-patients

16*/.../1,952

Number of optimal treatments provided and rank of therapy prescribed.

0

Reduction in ischemic and haemorrhagic vascular events.

0

Zanetti, 2003[68]

8

Alert to redose prophylactic antibiotics during prolonged cardiac surgery.

1/.../447*

Intraoperative redose of antibiotics.

+

Surgical-site infection.

0

Christakis, 2001[73]

5

Recommendations for appropriate prescribing of antibiotics for otitis media in paediatric outpatients.

1/38*/488

Antibiotics prescribed for <10 days.

+

...

...

Rossi, 1997[81]

9

Reminders to modify drug therapy in hypertensive outpatients receiving calcium channel blockers as initial therapy.

1/71/719*

Prescription changes from a calcium channel blocker to another antihypertensive agent.

+

...

...

Rotman, 1996[83]

7

Alerts to prescribe lowest cost drug alternative for adult outpatients.

1/37*/...

Rate of clinically relevant drug interactions.

0

...

...

McDonald, 1980[87]

5

Detection and management of medication-related problems in outpatients.

1/31*/...

Response rate for reminders over five weeks.

+

...

...

Coe, 1977[88]

4

Recommendations for medication management of hypertension in patients attending hypertension clinics.

2/.../116*

...

...

Adequate BP control achieved.

0

McDonald, 1976[89]

2

Recommendations for laboratory tests to detect potential medication-related events in adults attending a diabetes clinic.

1/.../226*

Ordered required tests to monitor drug effects; appropriate response to abnormal measures.

+

...

...

Studies of multi-faceted interventions

Bertoni, 2009[16, 21]

9

Recommendations for screening and treatment of dyslipidaemia in primary care.

59*/.../3,821

Patients with appropriate lipid management at follow-up.

+

...

...

Gilutz, 2009[19]

7

Reminders for monitoring and treatment of dyslipidaemia in primary care patients with known coronary artery disease.

112*/600/7,448

Appropriate initiation, up-titration, or continuation of statin therapy; adequate lipoprotein monitoring.

+

Change in low-density lipoprotein level.

+

Javitt, 2008[27]

6

Patient-specific recommendations for detecting and correcting medical errors in a health maintenance organization setting.

1/1378/49,988*

Resolution for problems identified by care considerations by adding or stopping a drug.

+

...

...

Quinn, 2008[29]

6

Recommendations for management of type 2 diabetes in primary care using remote glucose monitoring.

3/26/30*

Medications intensified; Medication errors identified.

+

Change in HbA1c levels.

+

Van Wyk, 2008[31]

10

On-demand and automatic alerts to screen and treat dyslipidaemia in primary care.

38*/80/92,054

Patients requiring treatment were treated.

Auto, +

On-demand, 0

...

...

Feldstein, 2006b[40]

8

Reminders for monitoring and treatment of osteoporosis care in high-risk women in primary care who experienced a fracture.

15/159/311*

Received only osteoporosis medication within 6 months of study start.

+

...

...

Kuilboer, 2006[44]

10

Recommendations for monitoring and treatment of asthma and COPD in primary care.

32*/40/156,772

Number of prescriptions for respiratory drugs.

0

...

...

Lester, 2006[45, 59]

8

Recommendations for the management of dyslipidaemia in primary care.

1/14/235*

Change in statin prescriptions at 1 and 12 months.

+

Change in low-density lipoprotein levels.

0

Cobos, 2005[49]

10

Recommendations for treatment, monitoring and follow-up for patients with dyslipidaemia in primary care.

42*/.../2,221

Assessed AST/ALT measurements or creatine kinase determinations; use of lipid-lowering drugs in patients with CHD or those without CHD and at high-risk or low-risk.

0

Successful management of cardiovascular risk.

0

Javitt, 2005[52]

6

Recommendations for management of patients whose care deviates from recommended practices in primary care.

.../.../39,462*

Compliance with recommendations to add or discontinue a medication.

+

Hospital admissions, inpatient days, and length of hospital stay.

+

Plaza, 2005[53]

9

Recommendations for cost-effective management of asthma in primary care.

.../20*/198

Prescriptions of oral glucocorticoids

+

St. George Respiratory Questionnaire total score.

+

Sequist, 2005[55]

6

Reminders for management of diabetes and coronary artery disease in primary care.

20*/194/6,243

Receipt of recommended drugs in diabetes (statins, ACE-Is in hypertension) or coronary artery disease (including aspirin, β-blocker or statin use).

0

...

...

Tierney, 2005[56]

9

Recommendations for the management of asthma and chronic obstructive pulmonary disease in adults in primary care.

4/266*/706

Suggestions adhered to for starting, modifying or stopping bronchodilators; medication compliance; patient satisfaction with physicians and pharmacists.

0

SF-36 subscale scores; McMaster Asthma Quality of Life and Chronic Respiratory Disease Questionnaires overall scores; emergency department visits; hospitalizations.

0

Wolfenden, 2005[57]

7

Reminders to provide smoking cessation interventions to patients attending non-cardiac pre-operative clinic.

1/18/210*

Preoperative nicotine replacement therapy offered or prescribed.

+

...

...

Murray, 2004[60]

5

Recommendations for treatment of hypertension in primary care.

4/...*/712

Compliance with all antihypertensive drug suggestions; patient satisfaction.

0

Overall composite quality of life score.

0

Tierney, 2003[66]

10

Recommendations for management of heart disease in primary care.

4*/115/706

Adherence with care suggestions to start low-dose aspirin or antihyperlipidemic drugs, or start or increase an ACE-Is, β-blockers, diuretics, long-acting nitrates, or calcium blockers.

0

Quality of life (score SF-36) at 12 months; quality of life (Chronic heart disease questionnaire overall health status).

0

Eccles, 2002[64, 69]

10

Recommendations for management of asthma or angina in adults in primary care.

62*/.../4,506

Prescription of β-blockers for angina patients.

0

(angina group)

Overall and disease-specific quality of life for angina patients.

0

(angina group)

Flottorp, 2002[63, 70]

9

Recommendations for management of urinary tract infections in women or sore throat in primary care.

142*/.../...

Use of antibiotics for sore throat and urinary tract infections.

+

...

...

Lesourd, 2002[71]

5

Recommendations for hormonal ovarian stimulation for infertile women in a teaching hospital.

3/4/164*

...

...

Pregnancy rate.

0

Selker, 2002[72]

8

Recommendations for thrombolytic and other reperfusion therapy in acute myocardial infarction.

28/.../1,596*

Patients who had ST-segment elevation detected but did not have acute myocardial infarction, including those who received thrombolytic therapy and those who had contraindications; thrombolytic therapy prescribed within one hour of acute myocardial infarction.

0

Death, stroke, or thrombolysis-related bleeding events that required transfusion within 30 days follow-up.

0

Dexter, 2001[74]

10

Reminders for preventive therapies in hospital inpatients.

...*/202/3,416

Hospitalizations with an order for prophylactic heparin or aspirin at discharge (all patients and only eligible patients).

+

...

...

McCowan, 2001[75]

8

Recommendations and reminders for management of asthma in primary care.

...*/46/477

Received prescription for acute asthma exacerbations.

+

Acute exacerbation of asthma.

+

Demakis, 2000[76]

7

Reminders for screening, monitoring, and counselling in accordance with predefined standards of care in ambulatory care.

12*/275/12,989

Adherence to recommendations for international normalised ratio monitoring in warfarin users, anticoagulation in atrial fibrillation, β-blocker following myocardial infarction or change in non-steroidal anti-inflammatory drug therapy following gastrointestinal bleed.

0

...

...

Hetlevik, 1999[7779]

8

Recommendations for diagnosis and management of hypertension, diabetes mellitus, and dyslipidaemia in primary care.

56*/56/3,273

...

...

For hypertension and diabetic patients, change at 21 months in systolic and diastolic BP, serum cholesterol, BMI, proportion of smokers,

CHD risk score (women or men), proportion of patients with cardiovascular inheritance, or HbA1c level (diabetic patients only).

0

Overhage, 1997[80]

8

Recommendations for corollary orders to prevent errors of omission for tests and treatments in general medicine inpatients.

1*/92/2,181

Pharmacist interventions with physicians for significant errors.

+

Days in hospital; maximum serum creatinine level during hospital stay.

0

Overhage, 1996[82]

10

Reminders to comply with 22 US Preventive Services Task Force preventive care measures for hospital inpatients.

1*/78/1,622

Compliance with preventive care guidelines for use of aspirin, oestrogen or calcium, ACE-Is, heparin prophylaxis, and β-blockers.

0

...

...

Tierney, 1993[84]

10

Alerts for drug allergies and drug-drug interactions, and options for cost-effective testing in inpatients.

6*/276/5,219

...

...

Length of hospital stay and resource use after discharge (outpatient visits and readmissions).

0

Mazzuca, 1990[85]

7

Reminders for the management of type 2 diabetes mellitus in outpatients.

4*/114/279

Initiation of oral hypoglycaemic therapy.

0

...

...

McAlister, 1986[86]

7

Recommendations for management of hypertension in primary care.

50/50*/2,231

Patients receiving treatment for hypertension.

0

Diastolic BP ≤90 mmHg on last visit; days with diastolic BP ≤90 mmHg; change in diastolic BP from baseline.

0

Abbreviations: ACE-I = angiotensin converting enzyme inhibitor; BP = blood pressure; CCDSS = computerized clinical decision support system; CHD = coronary heart disease; COPD = chronic obstructive pulmonary disease; LVEF = left ventricular ejection fraction.

*Unit of allocation.

aOutcomes are evaluated for effect based on the following hierarchy, with an effect defined as ≥50% of relevant outcomes showing a statistically significant difference (2p < 0.05):

1. If a single primary outcome is reported, in which all components are applicable, this is the only outcome evaluated.

2. If >1 primary outcome is reported, the ≥50% rule applies and only the primary outcomes are evaluated.

3. If no primary outcomes are reported (or only some of the primary outcome components are relevant) but overall analyses are provided, the overall analyses are evaluated as primary outcomes. Subgroup analyses are not considered.

4. If no primary outcomes or overall analyses are reported, or only some components of the primary outcome are relevant for the application, any reported prespecified outcomes are evaluated.

5. If no clearly prespecified outcomes are reported, any available outcomes are considered.

6. If statistical comparisons are not reported and data are insufficient to conduct analyses, 'effect' is designated as not evaluated (...).

Study characteristics

Thirty-six trials (55%) [17, 18, 20, 2226, 28, 30, 3239, 4143, 4648, 50, 51, 54, 58, 61, 62, 65, 67, 68, 73, 81, 83, 8789] described systems classified as drug therapy-only with the remaining 29 (45%) [16, 19, 21, 27, 29, 31, 40, 44, 45, 49, 52, 53, 5557, 59, 60, 63, 64, 66, 6972, 7480, 82, 8486] describing multi-faceted CCDSSs. Forty-one of 65 included studies (63%) [1659, 63, 68, 70] were published since the previous version of this review. Eleven trials (17%) were published prior to 2000 [7789], 16 (25%) trials [58, 6076] between 2000 and 2004 and 38 (58%) trials [1634, 3446, 4657, 59] after 2004. Most studies (n = 41, 63%) [1621, 23, 24, 26, 28, 3133, 38, 40, 4244, 47, 48, 5557, 60, 61, 63, 6668, 70, 7280, 8288] reported public funding; nine (14%) [29, 34, 35, 45, 46, 49, 50, 52, 53, 59, 71] reported private funding; six (9%) [22, 36, 37, 41, 54, 64, 65, 69] reported public and private funding, and 9 (14%) [25, 27, 30, 39, 51, 58, 62, 81, 89] did not disclose a funding source (see Additional file 3, Table S3). We were able to determine whether improvement occurred with a CCDSS for process of care outcomes in 59 studies [1624, 2638, 4070, 7276, 8083, 8587, 89]; 29 studies reported patient outcomes [19, 25, 26, 29, 33, 38, 39, 43, 45, 48, 49, 5153, 56, 5961, 64, 6669, 71, 72, 75, 7780, 84, 86, 88], and both patient and process outcomes were extracted from 23 (of 29, 79%) reports [19, 26, 29, 33, 38, 43, 45, 48, 49, 5153, 56, 5961, 64, 6669, 72, 75, 80, 86] (Table 1 and see Additional file 4, Table S4). Twenty [32, 33, 38, 39, 43, 48, 5153, 56, 60, 61, 6669, 7173, 75, 80, 84] of 29 (69%) studies reported a patient important outcome rather than an intermediate or surrogate outcome [90].

Study quality

Included trials had a median methodological quality score of 8 (interquartile range [IQR], 2) of a total possible score of 10. Quality assessments for each trial are presented in Additional file 1, Table S1. Most included studies were cluster randomized (n = 44/65, 68%) [1626, 28, 3032, 34, 41, 42, 44, 4649, 53, 55, 56, 58, 6067, 69, 70, 7387], measured an objective outcome or blinded outcome assessments appropriately (n = 64/65, 98%) [1656, 5889], had 80% or greater follow-up of subjects (n = 56, 86%) [1638, 4050, 5254, 5659, 6172, 74, 7679, 8187] and 41 (63%) [16, 18, 2023, 3034, 3651, 53, 54, 5659, 63, 64, 66, 6870, 72, 74, 75, 7782, 84] reported adequate allocation concealment. There was no change in quality score over time (R2 = 0.01, p = 0.53).

CCDSS and study characteristics

Additional file 2, Table S2 describes CCDSS users and Additional file 3, Table S3 describes study settings. A sum of 8,932 providers (median, 80; IQR, 193) used a CCDSS to assist with drug management for a total studied population of 1,246,686 patients (median, 2027; IQR 6960). Most CCDSSs were used by fully-trained physicians (61/65, 94%) [1629, 3135, 3856, 5889] and some by post-graduate medical trainees (19/65, 29%) [20, 23, 32, 35, 55, 56, 60, 66, 73, 74, 76, 8082, 84, 85, 8789]. After physicians, nurses in advanced practice roles (16/25 studies, 25%) [16, 18, 2022, 2527, 33, 35, 41, 42, 57, 58, 68, 73, 81, 87], physician assistants 8/65, 12%) [16, 18, 20, 21, 25, 42, 58, 63, 70, 7779], and pharmacists (8/65, 12%) [30, 3537, 54, 56, 60, 66] were the most common provider types interacting with CCDSSs. Many systems reported use by more than one type of provider. CCDSSs were studied in the United States (n = 44, 68%) [16, 18, 2023, 2630, 32, 33, 3537, 4043, 45, 47, 5052, 5456, 5861, 66, 68, 7274, 76, 8085, 8789], European Union or European Economic Area countries (n = 13, 20%) [31, 34, 38, 39, 44, 46, 49, 53, 6264, 6971, 75, 7779], and Canada (n = 3, 5%) [17, 24, 65, 86], with the remaining five studies (8%) [19, 25, 48, 57, 67] occurring in multiple or other countries.

Outpatient settings were studied more often (n = 55, 85%) [16, 1823, 2641, 4347, 4967, 6971, 73, 7579, 81, 83, 8589] than other settings of care. Studies were conducted in both academic settings (n = 34, 52%) [18, 23, 25, 26, 28, 3335, 38, 39, 42, 46, 48, 51, 5557, 60, 61, 66, 68, 7174, 76, 8085, 8789] and outside academic centres (n = 31, 48%) [16, 17, 1922, 24, 27, 2932, 36, 37, 40, 41, 4345, 47, 49, 50, 5254, 58, 59, 6265, 67, 69, 70, 75, 7779, 86].

As presented in Additional file 2, Table S2, the majority of CCDSS systems in our sample were integrated with an EMR (n = 38/61, 62%) [17, 18, 20, 2326, 28, 31, 32, 34, 40, 42, 4447, 49, 55, 56, 5866, 6870, 73, 74, 7785, 87, 89], delivered feedback via a computer display (n = 44/62, 71%) [17, 18, 20, 2326, 28, 3036, 3840, 42, 44, 4649, 51, 5558, 6066, 6871, 7376, 80, 8284] at the time of care (n = 53/64, 83%) [1621, 2326, 28, 3035, 38, 40, 42, 44, 4651, 53, 5558, 6070, 7285, 8789]. A minority of authors reported testing a CCDSS with a graphical user interface (n = 22/25, 88%) [1618, 20, 21, 2325, 28, 30, 31, 34, 38, 40, 45, 46, 55, 56, 5860, 63, 65, 70, 73, 75, 83], pilot-testing the system before the trial (n = 25/45, 56%) [1719, 24, 28, 29, 3133, 35, 38, 39, 4345, 47, 48, 55, 59, 63, 66, 7072, 74, 75, 7779, 84], or training users on system use (n = 29/52, 56%) [16, 19, 21, 26, 28, 29, 31, 32, 34, 35, 38, 4347, 53, 5660, 62, 6466, 69, 7579, 8385]. Data required by the CCDSS to produce recommendations were most commonly entered via EMR link (n = 32/61, 52%) [17, 18, 20, 24, 26, 28, 31, 32, 34, 40, 4447, 49, 51, 55, 56, 5860, 6266, 6870, 74, 8082, 84, 85, 87, 89], followed by provider entry (n = 23/61, 38%) [16, 21, 23, 25, 30, 31, 34, 35, 38, 39, 46, 53, 56, 63, 64, 66, 67, 6971, 73, 75, 76, 80, 83, 86, 88], study staff (n = 10/61, 16%) [19, 39, 42, 43, 48, 61, 63, 70, 74, 86, 89], and existing staff (n = 8/61, 13%) [36, 37, 68, 7274, 88, 89], although multiple modes of entry were reported in some studies. Nineteen (29%) [17, 18, 20, 2325, 31, 32, 34, 35, 42, 4547, 56, 5860, 80, 8284] studies reported using systems that were integrated with CPOE.

Clinical characteristics

CCDSSs were grouped into one of three categories representing the primary pharmacotherapeutic purpose of the system. Systems designed to optimize drug therapy were tested in 47 (72%) trials [16, 19, 26, 27, 2934, 3840, 4345, 4853, 5558, 6062, 6676, 78, 81, 82, 8588]; systems to prevent adverse drug events accounted for 16 (25%) trials [17, 20, 23, 25, 28, 3537, 41, 42, 47, 54, 65, 80, 84, 89];while the remaining two (3%) trials [18, 83] focused on drug cost management. Patient populations were identified (for each system) and consisted of seven (11%) systems for geriatric patients [17, 23, 25, 35, 37, 42, 65], three (5%) systems for paediatrics [32, 43, 73], four (6%) systems for women's health [36, 40, 70, 71], and 51 (78%) for adults or unspecified general populations. We attempted to identify the main disease state targeted by each system. Sixteen systems (25%) [17, 23, 25, 27, 34, 35, 37, 42, 52, 58, 65, 74, 76, 8284] were employed for multiple conditions. Each of the following disease groupings included three or more systems: cardiovascular disease [26, 33, 51, 60, 61, 66, 69, 72, 81, 86, 88] (n = 11, 17%), diabetes mellitus [29, 30, 50, 55, 62, 78, 85] (n = 7, 11%), respiratory disease [43, 44, 53, 56, 75] (n = 5, 8%), dyslipidaemia [16, 19, 31, 45, 49] (n = 5, 8%) and infectious diseases [32, 48, 68, 70, 73] (n = 5, 8%). Nine of the remaining 16 systems [20, 28, 36, 41, 47, 54, 80, 87, 89] were designed to prevent or detect drug related problems via laboratory monitoring.

CCDSS effectiveness

Thirty-seven trials [16, 18, 19, 2123, 26, 27, 2933, 3538, 40, 41, 43, 45, 48, 50, 5254, 5759, 62, 63, 65, 68, 70, 7375, 80, 81, 87, 89] of 59 (63%) showed improvement in process of care outcomes due to CCDSS use. No significant difference was found between Rx-only (23/33) [18, 22, 23, 26, 30, 32, 33, 3538, 41, 43, 48, 50, 54, 58, 62, 65, 68, 73, 81, 87, 89] and multi-faceted (14/26) [16, 19, 21, 27, 29, 31, 40, 45, 52, 53, 57, 59, 63, 70, 74, 75, 80] CCDSSs for process of care improvement.

Six trials [19, 29, 39, 52, 53, 75] (9% of all trials, 21% of trials measuring a patient outcome) demonstrated improved patient outcomes with CCDSS use compared to usual care without a CCDSS (see Table 1). Four [39, 52, 53, 75] of the six trials demonstrating improved patient outcomes measured patient-important outcomes. No significant difference in improvement was found between drug-only (1/12) [39] and multi-faceted (5/17) [19, 29, 52, 53, 75] CCDSSs.

Results did not significantly vary, for either process of care or patient outcomes, by the type of outcome (primary, pre-specified, or other) selected to determine improvement.

All studies demonstrating improved patient outcomes also showed improvement in measured process of care outcomes.

The proportion of successful trials was not significantly different between cluster trials where units of allocation were mismatched with units of analyses (7/15 for process and 1/5 for patient outcomes) compared with non-cluster trials or cluster trials with an appropriately adjusted analysis (29/43 for process (p = 0.22) and 5/24 for patient outcomes (p = 1)).

Predictors of success

This analysis was limited by incomplete data in many studies and by limited power for multivariate analysis. In univariate analysis, CCDSSs not integrated with an EMR were more likely to improve process of care outcomes, 16/20 (80%) non-integrated systems showed improvement versus 18/35 (51%) improved with EMR linkage (p = 0.03). The same trend was seen with integration of EMR and CCDSS for patient outcomes (6/15 (40%) improved outcomes without EMR link versus (0/13 (0%) with EMR link, p = 0.017). This association between EMR integration and CCDSS failure was not statistically significant via multivariate regression.

Improvement in process of care or patient outcomes was not affected by integration with CPOE, timing or method of decision support delivery, or method of data entry. Improvement in process or patient outcomes did not vary by country, provider type, and outpatient versus other settings of care. Systems trialed outside of academic settings were more likely to improve patient outcomes (5/12 (42%) outside academic settings versus 1/17 (6%) in academic settings, p = 0.03). This finding was not replicated in multivariate analyses. Investigators who developed the system under study were not significantly more likely to see improvement with a CCDSS than investigators studying systems developed by unrelated parties (p = 0.56 for process and p = 1 for patient outcomes). Patient important outcomes were as likely as surrogate outcomes to show improvement with a CCDSS. Post hoc, none of primary disease state, primary patient population, or pharmacotherapeutic purpose predicted success. We found no association between the presence of a sample size calculation and success or between number of trial participants and success. The proportion of studies added in this update demonstrating benefit with CCDSS for process of care and patient outcomes increased compared with studies included in the previous review version, although this trend was not statistically significant.

Costs and practical process related outcomes

Harms

Potential or actual harm resulting from CCDSS use was explicitly discussed in four (6%) [16, 21, 36, 68, 89] included studies (see Additional file 5, Table S5). Two studies reported quantitative data regarding harms. Raebel et al.[36] reported a trial stopped early due to a high rate (40%) of clinically inappropriate reminders generated by the CCDSS. Zanetti et al.[68] reported one inappropriate redose of intra-operative prophylactic antibiotic for every 137 appropriate redose reminders.

Costs

Some information on financial or economic costs associated with CCDSS was reported for 15 (23%) [17, 19, 24, 27, 43, 48, 49, 52, 53, 56, 57, 60, 66, 80, 83, 84] trials (see Additional file 5, Table S5). A formal cost-effectiveness analysis for a patient outcome was performed in only one case [53]. Twelve trials compared direct healthcare costs between CCDSSs and control groups with mixed results: significantly decreased costs were observed in six trials [27, 43, 48, 49, 52, 84], no significant change in five trials [53, 60, 66, 80, 83], and significantly increased costs in one trial [56].

User satisfaction

Fifteen authors reported on user satisfaction with the CCDSS studied (see Additional file 5, Table S5) [18, 19, 29, 30, 33, 39, 55, 57, 63, 64, 67, 69, 70, 75, 7779, 83, 84]. All attempts to measure user satisfaction were conducted via surveys and the properties of the measure used were only discussed in a single trial [55]. Survey response rates ≥50% were found in eight studies. Of these eight, six reported [18, 29, 33, 55, 57, 84] that ≥70% of respondents thought the CCDSS improved care, was useful, or should be continued in use. Satisfaction data from the other two trials [7779, 83] suggested users could not or would not use the CCDSS due to technical or user interface problems. Available data on user satisfaction were too sparse to determine if satisfaction impacted study results.

Discussion

We reviewed 65 RCTs of CCDSSs for drug therapy management reported over a 34-year span. Most trials measured process of care outcomes and results supported the use of CCDSSs to improve these outcomes in a majority of cases (improvement was based on at least 50% of the relevant study outcomes being statistically significantly positive). However, while nearly one-half of (29 studies) included studies measured a patient outcome, only a small proportion demonstrated any direct benefit to patients. While improvement in process outcomes could lead to benefits for patients, no consistent link was observed here. In the absence of data needed for an economic analysis, improved process of care measures alone are not sufficient to recommend adoption of these systems. The success rates we found for processs of care (64%) and clinical outcome measures (21%) are similar to those in our previous review [2] and also a recent umbrella review of systematic reviews of computerized decision support (57% and 30% respectively) by Jaspers et al.[91].

Several possible predictors of CCDSS success were examined. In most cases, these a priori factors did not explain success or failure across included studies. Our previous review [2] concluded that successful trials of CCDSS were more likely to have been conducted by the developers of the system under study. In our current review, no such association was noted. Previously, a significant trend towards increased study quality over time was noted, but not replicated in this update, and we attribute this to a more restrictive inclusion criterion (randomized controlled trials). Counter to our expectations, we found that integration of CCDSSs with EMRs and use in an academic setting was associated with CCDSS failure. This trend was not statistically significant when tested using multi-variate techniques and so we are unable to determine whether this finding represents a true association or is better explained by the lack of power in our multi-variate analysis. We report these findings as hypothesis generating only and suggest they be examined in future.

Compared with the review of Kawamoto et al.[92], we did not find that automatic provision of advice as part of the existing clinical workflow predicted CCDSS success. Because both the current analysis and that of Kawamoto were underpowered to detect such associations, we have refrained from drawing any conclusions in this regard.

Prospective data on the possible harms of CCDSSs are needed to facilitate informed adoption decisions. Only two trials quantitatively reported on harm from CCDSSs [36, 68] with one trial ending early due to increased risk of harm with the CCDSS. We suggest this absence of evidence of harm should not be taken as proof that CCDSSs are safe to employ for drug management in patient care.

Strengths and limitations of review

The results of our review should be interpreted with consideration of methodological strengths and limitations, including steps taken to mitigate the risk of bias. We based our review on the strongest studies available, RCTs. Reviews are necessarily retrospective and we employed multiple methods to limit the introduction of bias, including: duplicate study eligibility assessment, duplicate data abstraction, solicitation of study author feedback on abstracted data, and objective selection of outcomes used to determine improvement. We cannot exclude the possibility that a different method of selecting outcomes from each study to measure improvement could lead to different results, although sensitivity analyses did not suggest this to be the case. Several pre-specified analyses of possible predictors of system success were conducted. Several analyses demonstrated statistically significant results using univariate techniques that were not substantiated using a multi-variate model. Therefore, the few associations we reported between possible predictors of success and improved outcomes with CCDSS should be interpreted with caution.

We have relied upon vote counting as our method of obtaining an estimate of how often CCDSS for drug therapy management improve process or patient outcomes. Significant limitations to this approach as described by Hedges [93] include a tendency to inflate type II error and inadequate incorporation of the effect of unequal study sizes in overall results. The heterogeneity between studies included in our review precluded the use of more robust combination techniques. Formal assessment for publication bias using funnel plots was not possible with the vote-counting technique.

The effectiveness of any CCDSS will be determined in part by the efficacy of the underlying action suggested by the system. Where no benefit was detected with a particular CCDSS, we cannot exclude the possibility that the negative finding is due to a lack of efficacy of the intervention suggested by the system. Measurement of the concordance between decision advice given and followed would be a useful measure to address this issue. These outcomes were included in our analyses of process of care outcomes. It does not necessarily follow, however, that an effective CCDSS that recommends the appropriate prescription of an efficacious intervention will necessarily improve patient care. A multitude of intervening factors (e.g. patient non- or over-adherence or new errors introduced by CCDSS) may mitigate (or exaggerate) estimates of CCDSS effectiveness.

Finally, the systems reviewed constitute a heterogeneous group with differing functionality and clinical intent. While we have attempted to usefully divide the systems for the reader, we acknowledge other divisions were possible.

Implications for practice and research

Because CCDSSs have not been shown to reliably and positively impact patients, and in the absence of useful data on potential harms, costs, and clinician impacts, we cannot recommend the general adoption of CCDSSs for drug therapy management. It is possible that these systems are still evolving and success will improve with time. Clearly further innovation is needed if these systems are to be dependably useful in clinical practice. Rigorous trials of these innovations will be necessary, and we suggest that future research explicitly address patient outcomes, including potential harms, and costs and adverse clinician impacts of CCDSSs. Given the availability of effective non-computerized approaches for promoting safe and effective medication use [5, 94], future studies may wish to incorporate these interventions as active comparators to CCDSSs.

Conclusions

CCDSSs inconsistently improved process of care measures and seldom improved patient outcomes. Lack of clear patient benefit and lack of data on harms and costs preclude a recommendation to adopt CCDSSs for drug therapy management.

Declarations

Acknowledgements

The research was funded by a Canadian Institutes of Health Research Synthesis Grant: Knowledge Translation KRS 91791. The members of the Computerized Clinical Decision Support System (CCDSS) Systematic Review Team included the Principal Investigator, Co-Investigators, Co-Applicants/Senior Management Decision-makers, Co-Applicants/Clinical Service Decision-Makers, and Research Staff. The following were involved in collection and/or organization of data: Jeanette Prorok, MSc, McMaster University; Nathan Souza, MD, MMEd, McMaster University; Brian Hemens, BScPhm, MSc, McMaster University; Robby Nieuwlaat, PhD, McMaster University; Shikha Misra, BHSc, McMaster University; Jasmine Dhaliwal, BHSc, McMaster University; Navdeep Sahota, BHSc, University of Saskatchewan; Anita Ramakrishna, BHSc, McMaster University; Pavel Roshanov, BSc, McMaster University; Tahany Awad, MD, McMaster University. Nicholas Hobson, DiplT, Chris Cotoi, BEng, EMBA, and Rick Parrish, DiplT, at McMaster University provided programming and information technology support.

Authors’ Affiliations

(1)
Health Information Research Unit, Department of Clinical Epidemiology and Biostatistics, McMaster University
(2)
Department of Medicine, McMaster University
(3)
Department of Clinical Epidemiology and Biostatistics, McMaster University
(4)
Hamilton Health Sciences

References

  1. Sim I, Gorman P, Greenes RA, Haynes RB, Kaplan B, Lehmann H, Tang PC: Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc. 2001, 8 (6): 527-534. 10.1136/jamia.2001.0080527.PubMedPubMed CentralGoogle Scholar
  2. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, Sam J, Haynes RB: Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005, 293 (10): 1223-1238. 10.1001/jama.293.10.1223.PubMedGoogle Scholar
  3. Osheroff JA, Pifer EA, Teich JM, Sittig DF, Jenders RA, CDS Expert Review Panel: Improving outcomes with clinical decision support. 2005, Chicago, IL: Healthcare Information & Management Systems SocietyGoogle Scholar
  4. Teich JM, Osheroff JA, Pifer EA, Sittig DF, Jenders RA, CDS Expert Review Panel: Clinical decision support in electronic prescribing: recommendations and an action plan: report of the joint clinical decision support workgroup. J Am Med Inform Assoc. 2005, 12 (4): 365-376. 10.1197/jamia.M1822.PubMedPubMed CentralGoogle Scholar
  5. Corrigan J, Donaldson MS, Kohn LT, Institute of Medicine. Committee on Quality of Healthcare in America: To err is human: building a safer health system. 2000, Washington, D.C.: National Academy PressGoogle Scholar
  6. Berner ES: Clinical decision support systems: State of the Art. 2009, Rockville, Maryland: Agency for Healthcare Research and QualityGoogle Scholar
  7. Touw DJ, Neef C, Thomson AH, Vinks AA, Cost-Effectiveness of Therapeutic Drug Monitoring Committee of the International Association for Therapeutic Drug Monitoring and Clinical Toxicology: Cost-effectiveness of therapeutic drug monitoring: a systematic review. Ther Drug Monit. 2005, 27 (1): 10-17. 10.1097/00007691-200502000-00004.PubMedGoogle Scholar
  8. Hulley SB: Designing clinical research. 2007, Philadelphia, PA: Wolters Kluwer: Lippincott Williams & Wilkins, 3Google Scholar
  9. Haynes RB, Wilczynski NL, Computerized Clinical Decision Support System (CCDSS) Systematic Review Team: Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: methods of a decision-maker-researcher partnership systematic review. Implement Sci. 2010, 5: 12-PubMedPubMed CentralGoogle Scholar
  10. Apkon M, Mattera JA, Lin Z, Herrin J, Bradley EH, Carbone M, Holmboe ES, Gross CP, Selter JG, Rich AS, Krumholz HM: A randomized outpatient trial of a decision-support information technology tool. Arch Intern Med. 2005, 165 (20): 2388-2394. 10.1001/archinte.165.20.2388.PubMedGoogle Scholar
  11. Downs M, Turner S, Bryans M, Wilcock J, Keady J, Levin E, O'Carroll R, Howie K, Iliffe S: Effectiveness of educational interventions in improving detection and management of dementia in primary care: cluster randomised controlled study. BMJ. 2006, 332 (7543): 692-696. 10.1136/bmj.332.7543.692.PubMedPubMed CentralGoogle Scholar
  12. Holbrook A, Thabane L, Keshavjee K, Dolovich L, Bernstein B, Chan D, Troyan S, Foster G, Gerstein H, COMPETE II Investigators: Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial. CMAJ. 2009, 181 (1-2): 37-44. 10.1503/cmaj.081272.PubMedPubMed CentralGoogle Scholar
  13. Thomas JC, Moore A, Qualls PE: The effect on cost of medical care for patients treated with an automated clinical audit system. J Med Syst. 1983, 7 (3): 307-313. 10.1007/BF00993294.PubMedGoogle Scholar
  14. Meigs JB, Cagliero E, Dubey A, Murphy-Sheehy P, Gildesgame C, Chueh H, Barry MJ, Singer DE, Nathan DM: A controlled trial of web-based diabetes disease management: the MGH diabetes primary care improvement project. Diabetes Care. 2003, 26 (3): 750-757. 10.2337/diacare.26.3.750.PubMedGoogle Scholar
  15. Martin DC, Berger ML, Anstatt DT, Wofford J, Warfel D, Turpin RS, Cannuscio CC, Teutsch SM, Mansheim BJ: A randomized controlled open trial of population-based disease and case management in a Medicare Plus Choice health maintenance organization. Prev Chronic Dis. 2004, 1 (4): A05-PubMedPubMed CentralGoogle Scholar
  16. Bertoni AG, Bonds DE, Chen H, Hogan P, Crago L, Rosenberger E, Barham AH, Clinch CR, Goff DC: Impact of a multifaceted intervention on cholesterol management in primary care practices: guideline adherence for heart health randomized trial. Arch Intern Med. 2009, 169 (7): 678-686. 10.1001/archinternmed.2009.44.PubMedPubMed CentralGoogle Scholar
  17. Field TS, Rochon P, Lee M, Gavendo L, Baril JL, Gurwitz JH: Computerized clinical decision support during medication ordering for long-term care residents with renal insufficiency. J Am Med Inform Assoc. 2009, 16 (4): 480-485. 10.1197/jamia.M2981.PubMedPubMed CentralGoogle Scholar
  18. Fortuna RJ, Zhang F, Ross-Degnan D, Campion FX, Finkelstein JA, Kotch JB, Feldstein AC, Smith DH, Simon SR: Reducing the prescribing of heavily marketed medications: a randomized controlled trial. J Gen Intern Med. 2009, 24 (8): 897-903. 10.1007/s11606-009-1013-x.PubMedPubMed CentralGoogle Scholar
  19. Gilutz H, Novack L, Shvartzman P, Zelingher J, Bonneh DY, Henkin Y, Maislos M, Peleg R, Liss Z, Rabinowitz G, Vardy D, Zahger D, Ilia R, Leibermann N, Porath A: Computerized community cholesterol control (4C): meeting the challenge of secondary prevention. Israel Med Assoc J. 2009, 11 (1): 23-29.Google Scholar
  20. Lo HG, Matheny ME, Seger DL, Bates DW, Gandhi TK: Impact of non-interruptive medication laboratory monitoring alerts in ambulatory care. J Am Med Inform Assoc. 2009, 16 (1): 66-71.PubMedPubMed CentralGoogle Scholar
  21. Rosenberger EL, Goff DC, Blackwell CS, Williams DT, Crago OL, Ellis SD, Bertoni AG, Bonds DE: Implementing a palm pilot intervention for primary care providers: lessons learned. Contemp Clin Trials. 2009, 30 (4): 321-325. 10.1016/j.cct.2009.03.009.PubMedPubMed CentralGoogle Scholar
  22. Smith DH, Feldstein AC, Perrin NA, Yang X, Rix MM, Raebel MA, Magid DJ, Simon SR, Soumerai SB: Improving laboratory monitoring of medications: an economic analysis alongside a clinical trial. Am J Manag Care. 2009, 15 (5): 281-289.PubMedGoogle Scholar
  23. Terrell KM, Perkins AJ, Dexter PR, Hui SL, Callahan CM, Miller DK: Computerized decision support to reduce potentially inappropriate prescribing to older emergency department patients: a randomized, controlled trial. J Am Geriatr Soc. 2009, 57 (8): 1388-1394. 10.1111/j.1532-5415.2009.02352.x.PubMedGoogle Scholar
  24. Field TS, Rochon P, Lee M, Gavendo L, Subramanian S, Hoover S, Baril J, Gurwitz J: Costs associated with developing and implementing a computerized clinical decision support system for medication dosing for patients with renal insufficiency in the long-term care setting. J Am Med Inform Assoc. 2008, 15 (4): 466-472. 10.1197/jamia.M2589.PubMedPubMed CentralGoogle Scholar
  25. Gurwitz JH, Field TS, Rochon P, Judge J, Harrold LR, Bell CM, Lee M, White K, LaPrino J, Erramuspe-Mainard J, DeFlorio M, Gavendo L, Baril JL, Reed G, Bates DW: Effect of computerized provider order entry with clinical decision support on adverse drug events in the long-term care setting. J Am Geriatr Soc. 2008, 56 (12): 2225-2233. 10.1111/j.1532-5415.2008.02004.x.PubMedGoogle Scholar
  26. Hicks LS, Sequist TD, Ayanian JZ, Shaykevich S, Fairchild DG, Orav EJ, Bates DW: Impact of computerized decision support on blood pressure management and control: a randomized controlled trial. J Gen Intern Med. 2008, 23 (4): 429-441. 10.1007/s11606-007-0403-1.PubMedPubMed CentralGoogle Scholar
  27. Javitt JC, Rebitzer JB, Reisman L: Information technology and medical missteps: evidence from a randomized trial. J Health Econ. 2008, 27 (3): 585-602. 10.1016/j.jhealeco.2007.10.008.PubMedGoogle Scholar
  28. Matheny ME, Sequist TD, Seger AC, Fiskio JM, Sperling M, Bugbee D, Bates DW, Gandhi TK: A randomized trial of electronic clinical reminders to improve medication laboratory monitoring. J Am Med Inform Assoc. 2008, 15 (4): 424-429. 10.1197/jamia.M2602.PubMedPubMed CentralGoogle Scholar
  29. Quinn CC, Clough SS, Minor JM, Lender D, Okafor MC, Gruber-Baldini A: WellDoc™ mobile diabetes management randomized controlled trial: change in clinical and behavioral outcomes and patient and physician satisfaction. Diabetes Technol Ther. 2008, 10 (3): 160-168. 10.1089/dia.2008.0283.PubMedGoogle Scholar
  30. Reeve JF, Tenni PC, Peterson GM: An electronic prompt in dispensing software to promote clinical interventions by community pharmacists: a randomized controlled trial. Br J Clin Pharmacol. 2008, 65 (3): 377-385. 10.1111/j.1365-2125.2007.03012.x.PubMedGoogle Scholar
  31. van Wyk JT, van Wijk MA, Sturkenboom MC, Mosseveld M, Moorman PW, van der Lei J: Electronic alerts versus on-demand decision support to improve dyslipidemia treatment: a cluster randomized controlled trial. Circulation. 2008, 117 (3): 371-378. 10.1161/CIRCULATIONAHA.107.697201.PubMedGoogle Scholar
  32. Davis RL, Wright J, Chalmers F, Levenson L, Brown JC, Lozano P, Christakis DA: A cluster randomized clinical trial to improve prescribing patterns in ambulatory pediatrics. PLoS Clin Trials. 2007, 2 (5): e25-10.1371/journal.pctr.0020025.PubMedPubMed CentralGoogle Scholar
  33. Heidenreich PA, Gholami P, Sahay A, Massie B, Goldstein MK: Clinical reminders attached to echocardiography reports of patients with reduced left ventricular ejection fraction increase use of beta-blockers: a randomized trial. Circulation. 2007, 115 (22): 2829-2834. 10.1161/CIRCULATIONAHA.106.684753.PubMedGoogle Scholar
  34. Martens JD, van der Weijden T, Severens JL, de Clercq PA, de Bruijn DP, Kester AD, Winkens RA: The effect of computer reminders on GPs' prescribing behaviour: a cluster-randomised trial. Int J Med Inform. 2007, 76 (Suppl 3): S403-S416.PubMedGoogle Scholar
  35. Peterson JF, Rosenbaum BP, Waitman LR, Habermann R, Powers J, Harrell D, Miller RA: Physicians' response to guided geriatric dosing: initial results from a randomized trial. Stud Health Technol Inform. 2007, 129 (Pt 2): 1037-1040.PubMedGoogle Scholar
  36. Raebel MA, Carroll NM, Kelleher JA, Chester EA, Berga S, Magid DJ: Randomized trial to improve prescribing safety during pregnancy. J Am Med Inform Assoc. 2007, 14 (4): 440-450. 10.1197/jamia.M2412.PubMedPubMed CentralGoogle Scholar
  37. Raebel MA, Charles J, Dugan J, Carroll NM, Korner EJ, Brand DW, Magid DJ: Randomized trial to improve prescribing safety in ambulatory elderly patients. J Am Geriatr Soc. 2007, 55 (7): 977-985. 10.1111/j.1532-5415.2007.01202.x.PubMedGoogle Scholar
  38. Thomson RG, Eccles MP, Steen IN, Greenaway J, Stobbart L, Murtagh MJ, May CR: A patient decision aid to support shared decision-making on anti-thrombotic treatment of patients with atrial fibrillation: randomised controlled trial. Qual Saf Healthcare. 2007, 16 (3): 216-223. 10.1136/qshc.2006.018481.Google Scholar
  39. Verstappen SMM, Jacobs JWG, Van der Veen MJ, Heurkens AHM, Schenk Y, Ter Borg EJ, Blaauw AAM, Bijlsma JWJ: Intensive treatment with methotrexate in early rheumatoid arthritis: aiming for remission. Computer Assisted Management in Early Rheumatoid Arthritis (CAMERA, an open-label strategy trial). Ann Rheum Dis. 2007, 66 (11): 1443-1449. 10.1136/ard.2007.071092.PubMedPubMed CentralGoogle Scholar
  40. Feldstein A, Elmer PJ, Smith DH, Herson M, Orwoll E, Chen C, Aickin M, Swain MC: Electronic medical record reminder improves osteoporosis management after a fracture: a randomized, controlled trial. J Am Geriatr Soc. 2006, 54 (3): 450-457. 10.1111/j.1532-5415.2005.00618.x.PubMedGoogle Scholar
  41. Feldstein AC, Smith DH, Perrin N, Yang X, Rix M, Raebel MA, Magid DJ, Simon SR, Soumerai SB: Improved therapeutic monitoring with several interventions: a randomized trial. Arch Intern Med. 2006, 166 (17): 1848-1854. 10.1001/archinte.166.17.1848.PubMedGoogle Scholar
  42. Judge J, Field TS, DeFlorio M, Laprino J, Auger J, Rochon P, Bates DW, Gurwitz JH: Prescribers' responses to alerts during medication ordering in the long term care setting. J Am Med Inform Assoc. 2006, 13 (4): 385-390. 10.1197/jamia.M1945.PubMedPubMed CentralGoogle Scholar
  43. Kattan M, Crain EF, Steinbach S, Visness CM, Walter M, Stout JW, Evans Iii R, Smartt E, Gruchalla RS, Morgan WJ: A randomized clinical trial of clinician feedback to improve quality of care for inner-city children with asthma. Pediatrics. 2006, 117 (6): e1095-e1103. 10.1542/peds.2005-2160.PubMedGoogle Scholar
  44. Kuilboer MM, van Wijk MA, Mosseveld M, van der Does E, de Jongste JC, Overbeek SE, Ponsioen B, van der Lei J: Computed critiquing integrated into daily clinical practice affects physicians' behavior--a randomized clinical trial with AsthmaCritic. Methods Inf Med. 2006, 45 (4): 447-454.PubMedGoogle Scholar
  45. Lester WT, Grant RW, Barnett GO, Chueh HC: Randomized controlled trial of an informatics-based intervention to increase statin prescription for secondary prevention of coronary disease. J Gen Intern Med. 2006, 21 (1): 22-29. 10.1111/j.1525-1497.2005.00268.x.PubMedPubMed CentralGoogle Scholar
  46. Martens JD, van der Aa A, Panis B, van der Weijden T, Winkens RA, Severens JL: Design and evaluation of a computer reminder system to improve prescribing behaviour of GPs. Stud Health Technol Inform. 2006, 124: 617-623.PubMedGoogle Scholar
  47. Palen TE, Raebel M, Lyons E, Magid DM: Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders. Am J Manag Care. 2006, 12 (7): 389-395.PubMedGoogle Scholar
  48. Paul M, Andreassen S, Tacconelli E, Nielsen AD, Almanasreh N, Frank U, Cauda R, Leibovici L, TREAT Study G: Improving empirical antibiotic treatment using TREAT, a computerized decision support system: cluster randomized trial. J Antimicrob Chemother. 2006, 58 (6): 1238-1245. 10.1093/jac/dkl372.PubMedGoogle Scholar
  49. Cobos A, Vilaseca J, Asenjo C, Pedro-Botet J, Sanchez E, Val A, Torremade E, Espinosa C, Bergonon S: Cost effectiveness of a clinical decision support system based on the recommendations of the European Society of Cardiology and other societies for the management of hypercholesterolemia: Report of a cluster-randomized trial. Dis Manag Health Out. 2005, 13 (6): 421-432. 10.2165/00115677-200513060-00007.Google Scholar
  50. Derose SF, Dudl JR, Benson VM, Contreras R, Nakahiro RK, Ziel FH: Point of service reminders for prescribing cardiovascular medications. Am J Manag Care. 2005, 11 (5): 298-304.PubMedGoogle Scholar
  51. Heidenreich PA, Chacko M, Goldstein MK, Atwood JE: ACE inhibitor reminders attached to echocardiography reports of patients with reduced left ventricular ejection fraction. Am J Med. 2005, 118 (9): 1034-1037. 10.1016/j.amjmed.2004.12.028.PubMedGoogle Scholar
  52. Javitt JC, Steinberg G, Locke T, Couch JB, Jacques J, Juster I, Reisman L: Using a claims data-based sentinel system to improve compliance with clinical guidelines: results of a randomized prospective study. Am J Manag Care. 2005, 11 (2): 93-102.PubMedGoogle Scholar
  53. Plaza V, Cobos A, Ignacio-Garcia JM, Molina J, Bergonon S, Garcia-Alonso F, Espinosa C, Grupo Investigador A: [Cost-effectiveness of an intervention based on the Global INitiative for Asthma (GINA) recommendations using a computerized clinical decision support system: a physicians randomized trial]. Med Clin (Barc). 2005, 124 (6): 201-206. 10.1157/13071758.Google Scholar
  54. Raebel MA, Lyons EE, Chester EA, Bodily MA, Kelleher JA, Long CL, Miller C, Magid DJ: Improving laboratory monitoring at initiation of drug therapy in ambulatory care: A randomized trial. Arch Intern Med. 2005, 165 (20): 2395-2401. 10.1001/archinte.165.20.2395.PubMedGoogle Scholar
  55. Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Sperling M, Cook EF, Orav EJ, Fairchild DG, Bates DW: A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. J Am Med Inform Assoc. 2005, 12 (4): 431-437. 10.1197/jamia.M1788.PubMedPubMed CentralGoogle Scholar
  56. Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, Smith FE, Nienaber N, McDonald CJ, Wolinsky FD: Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial. Health Serv Res. 2005, 40 (2): 477-497. 10.1111/j.1475-6773.2005.0t369.x.PubMedPubMed CentralGoogle Scholar
  57. Wolfenden L, Wiggers J, Knight J, Campbell E, Spigelman A, Kerridge R, Moore K: Increasing smoking cessation care in a preoperative clinic: a randomized controlled trial. Prev Med. 2005, 41 (1): 284-290. 10.1016/j.ypmed.2004.11.011.PubMedGoogle Scholar
  58. Krall MA, Traunweiser K, Towery W: Effectiveness of an electronic medical record clinical quality alert prepared by off-line data analysis. Stud Health Technol Inform. 2004, 107 (Pt 1): 135-139.PubMedGoogle Scholar
  59. Lester WT, Grant R, Barnett GO, Chueh H: Facilitated lipid management using interactive e-mail: preliminary results of a randomized controlled trial. Stud Health Technol Inform. 2004, 107 (Pt 1): 232-236.PubMedGoogle Scholar
  60. Murray MD, Harris LE, Overhage JM, Zhou XH, Eckert GJ, Smith FE, Buchanan NN, Wolinsky FD, McDonald CJ, Tierney WM: Failure of computerized treatment suggestions to improve health outcomes of outpatients with uncomplicated hypertension: results of a randomized controlled trial. Pharmacotherapy. 2004, 24 (3): 324-337. 10.1592/phco.24.4.324.33173.PubMedGoogle Scholar
  61. Ansari M, Shlipak MG, Heidenreich PA, Van Ostaeyen D, Pohl EC, Browner WS, Massie BM: Improving guideline adherence: a randomized trial evaluating strategies to increase beta-blocker use in heart failure. Circulation. 2003, 107 (22): 2799-2804. 10.1161/01.CIR.0000070952.08969.5B.PubMedGoogle Scholar
  62. Filippi A, Sabatini A, Badioli L, Samani F, Mazzaglia G, Catapano A, Cricelli C: Effects of an automated electronic reminder in changing the antiplatelet drug-prescribing behavior among Italian general practitioners in diabetic patients: an intervention trial. Diabetes Care. 2003, 26 (5): 1497-1500. 10.2337/diacare.26.5.1497.PubMedGoogle Scholar
  63. Flottorp S, Havelsrud K, Oxman AD: Process evaluation of a cluster randomized trial of tailored interventions to implement guidelines in primary care--why is it so hard to change practice?. Fam Pract. 2003, 20 (3): 333-339. 10.1093/fampra/cmg316.PubMedGoogle Scholar
  64. Rousseau N, McColl E, Newton J, Grimshaw J, Eccles M: Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care. BMJ. 2003, 326 (7384): 314-10.1136/bmj.326.7384.314.PubMedPubMed CentralGoogle Scholar
  65. Tamblyn R, Huang A, Perreault R, Jacques A, Roy D, Hanley J, McLeod P, Laprise R: The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. Can Med Assoc J. 2003, 169 (6): 549-556.Google Scholar
  66. Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, Smith FE, Nienaber N, McDonald CJ, Wolinsky FD: Effects of computerized guidelines for managing heart disease in primary care. J Gen Intern Med. 2003, 18 (12): 967-976. 10.1111/j.1525-1497.2003.30635.x.PubMedPubMed CentralGoogle Scholar
  67. Weir CJ, Lees KR, MacWalter RS, Muir KW, Wallesch CW, McLelland EV, Hendry A, PRISM Study G: Cluster-randomized, controlled trial of computer-based decision support for selecting long-term anti-thrombotic therapy after acute ischaemic stroke. QJM. 2003, 96 (2): 143-153.PubMedGoogle Scholar
  68. Zanetti G, Flanagan HL, Cohn LH, Giardina R, Platt R: Improvement of intraoperative antibiotic prophylaxis in prolonged cardiac surgery by automated alerts in the operating room. Infect Control Hosp Epidemiol. 2003, 24 (1): 13-16. 10.1086/502109.PubMedGoogle Scholar
  69. Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, Purves I: Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002, 325 (7370): 941-10.1136/bmj.325.7370.941.PubMedPubMed CentralGoogle Scholar
  70. Flottorp S, Oxman AD, Havelsrud K, Treweek S, Herrin J: Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat. BMJ. 2002, 325 (7360): 367-10.1136/bmj.325.7360.367.PubMedPubMed CentralGoogle Scholar
  71. Lesourd F, Avril C, Boujennah A, Parinaud J: A computerized decision support system for ovarian stimulation by gonadotropins. Fertil Steril. 2002, 77 (3): 456-460. 10.1016/S0015-0282(01)03231-9.PubMedGoogle Scholar
  72. Selker HP, Beshansky JR, Griffith JL, TPI Trial Investigators: Use of the electrocardiograph-based thrombolytic predictive instrument to assist thrombolytic and reperfusion therapy for acute myocardial infarction. A multicenter, randomized, controlled, clinical effectiveness trial. Ann Intern Med. 2002, 137 (2): 87-95.PubMedGoogle Scholar
  73. Christakis DA, Zimmerman FJ, Wright JA, Garrison MM, Rivara FP, Davis RL: A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children. Pediatrics. 2001, 107 (2): E15-10.1542/peds.107.2.e15.PubMedGoogle Scholar
  74. Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ: A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med. 2001, 345 (13): 965-970. 10.1056/NEJMsa010181.PubMedGoogle Scholar
  75. McCowan C, Neville RG, Ricketts IW, Warner FC, Hoskins G, Thomas GE: Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma. Med Inform Internet. 2001, 26 (3): 191-201. 10.1080/14639230110067890.Google Scholar
  76. Demakis JG, Beauchamp C, Cull WL, Denwood R, Eisen SA, Lofgren R, Nichol K, Woolliscroft J, Henderson WG: Improving residents' compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders. JAMA. 2000, 284 (11): 1411-1416. 10.1001/jama.284.11.1411.PubMedGoogle Scholar
  77. Hetlevik I, Holmen J, Krüger O, Kristensen P, Iversen H, Furuseth K: Implementing clinical guidelines in the treatment of diabetes mellitus in general practice. Evaluation of effort, process, and patient outcome related to implementation of a computer-based decision support system. Int J Technol Assess. 2000, 16 (1): 210-227. 10.1017/S0266462300161185.Google Scholar
  78. Hetlevik I, Holmen J, Krüger O: Implementing clinical guidelines in the treatment of hypertension in general practice. Evaluation of patient outcome related to implementation of a computer-based clinical decision support system. Scand J Prim Healthcare. 1999, 17 (1): 35-40. 10.1080/028134399750002872.Google Scholar
  79. Hetlevik I, Holmen J, Kruger O, Kristensen P, Iversen H: Implementing clinical guidelines in the treatment of hypertension in general practice. Blood Press. 1998, 7 (5-6): 270-276. 10.1080/080370598437114.PubMedGoogle Scholar
  80. Overhage JM, Tierney WM, Zhou XH, McDonald CJ: A randomized trial of "corollary orders" to prevent errors of omission. J Am Med Inform Assoc. 1997, 4 (5): 364-375. 10.1136/jamia.1997.0040364.PubMedPubMed CentralGoogle Scholar
  81. Rossi RA, Every NR: A computerized intervention to decrease the use of calcium channel blockers in hypertension. J Gen Intern Med. 1997, 12 (11): 672-678. 10.1046/j.1525-1497.1997.07140.x.PubMedPubMed CentralGoogle Scholar
  82. Overhage JM, Tierney WM, McDonald CJ: Computer reminders to implement preventive care guidelines for hospitalized patients. Arch Intern Med. 1996, 156 (14): 1551-1556. 10.1001/archinte.156.14.1551.PubMedGoogle Scholar
  83. Rotman BL, Sullivan AN, McDonald TW, Brown BW, DeSmedt P, Goodnature D, Higgins MC, Suermondt HJ, Young C, Owens DK: A randomized controlled trial of a computer-based physician workstation in an outpatient setting: implementation barriers to outcome evaluation. J Am Med Inform Assoc. 1996, 3 (5): 340-348. 10.1136/jamia.1996.97035025.PubMedPubMed CentralGoogle Scholar
  84. Tierney WM, Miller ME, Overhage JM, McDonald CJ: Physician inpatient order writing on microcomputer workstations. Effects on resource utilization. JAMA. 1993, 269 (3): 379-383. 10.1001/jama.269.3.379.PubMedGoogle Scholar
  85. Mazzuca SA, Vinicor F, Einterz RM, Tierney WM, Norton JA, Kalasinski LA: Effects of the clinical environment on physicians' response to postgraduate medical education. Am Educ Res J. 1990, 27 (3): 473-488.Google Scholar
  86. McAlister NH, Covvey HD, Tong C, Lee A, Wigle ED: Randomised controlled trial of computer assisted management of hypertension in primary care. Br Med J (Clin Res Ed). 1986, 293 (6548): 670-674. 10.1136/bmj.293.6548.670.Google Scholar
  87. McDonald CJ, Wilson GA, McCabe GP: Physician response to computer reminders. JAMA. 1980, 244 (14): 1579-1581. 10.1001/jama.244.14.1579.PubMedGoogle Scholar
  88. Coe FL, Norton E, Oparil S, Tatar A, Pullman TN: Treatment of hypertension by computer and physician-a prospective controlled study. J Chronic Dis. 1977, 30 (2): 81-92. 10.1016/0021-9681(77)90077-7.PubMedGoogle Scholar
  89. McDonald CJ: Use of a computer to detect and respond to clinical events: its effect on clinician behavior. Ann Intern Med. 1976, 84 (2): 162-167.Google Scholar
  90. Bucher HC, Kunz R, Cook DJ, Holbrook AM, Guyatt G: Surrogate Outcomes. Users' guides to the medical literature: a manual for evidence-based clinical practice. Edited by: Guyatt G, Rennie D. 2008, Evidence-Based Medicine Working Group. Chicago, IL: McGraw-Hill Professional, 442-2Google Scholar
  91. Jaspers MW, Smeulers M, Vermeulen H, Peute LW: Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings. J Am Med Inform Assoc. 2011, 18 (3): 327-334. 10.1136/amiajnl-2011-000094.PubMedPubMed CentralGoogle Scholar
  92. Kawamoto K, Houlihan CA, Balas EA, Lobach DF: Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005, 330 (7494): 765-10.1136/bmj.38398.500764.8F.PubMedPubMed CentralGoogle Scholar
  93. Hedges LV, Olkin I: Statistical methods for meta-analysis. 1985, Orlando: Academic PressGoogle Scholar
  94. Grimshaw J, Eccles M, Thomas R, MacLennan G, Ramsay C, Fraser C, Vale L: Toward evidence-based quality improvement. Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966-1998. J Gen Intern Med. 2006, 21 (Suppl 2): S14-S20.PubMedPubMed CentralGoogle Scholar

Copyright

© Hemens et al; licensee BioMed Central Ltd. 2011

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.