Computerized clinical decision support systems for drug prescribing and management: A decision-maker-researcher partnership systematic review

Background Computerized clinical decision support systems (CCDSSs) for drug therapy management are designed to promote safe and effective medication use. Evidence documenting the effectiveness of CCDSSs for improving drug therapy is necessary for informed adoption decisions. The objective of this review was to systematically review randomized controlled trials assessing the effects of CCDSSs for drug therapy management on process of care and patient outcomes. We also sought to identify system and study characteristics that predicted benefit. Methods We conducted a decision-maker-researcher partnership systematic review. We updated our earlier reviews (1998, 2005) by searching MEDLINE, EMBASE, EBM Reviews, Inspec, and other databases, and consulting reference lists through January 2010. Authors of 82% of included studies confirmed or supplemented extracted data. We included only randomized controlled trials that evaluated the effect on process of care or patient outcomes of a CCDSS for drug therapy management compared to care provided without a CCDSS. A study was considered to have a positive effect (i.e., CCDSS showed improvement) if at least 50% of the relevant study outcomes were statistically significantly positive. Results Sixty-five studies met our inclusion criteria, including 41 new studies since our previous review. Methodological quality was generally high and unchanged with time. CCDSSs improved process of care performance in 37 of the 59 studies assessing this type of outcome (64%, 57% of all studies). Twenty-nine trials assessed patient outcomes, of which six trials (21%, 9% of all trials) reported improvements. Conclusions CCDSSs inconsistently improved process of care measures and seldomly improved patient outcomes. Lack of clear patient benefit and lack of data on harms and costs preclude a recommendation to adopt CCDSSs for drug therapy management.


Background
Computerized clinical decision support systems (CCDSSs) algorithmically apply an electronic knowledge base to individual patient data to generate and present suggested actions intended to enhance health and healthcare [1][2][3]. CCDSSs for drug therapy management are used to facilitate evidence-informed medication use [4], reduce the incidence of harmful medication errors [5], and improve healthcare system efficiency [2,4,6]. In this review, we considered any CCDSS that provides recommendations to healthcare providers regarding the initiation, modification, monitoring, or discontinuation of drug therapy, based on the patient's characteristics. Systems designed solely to provide advice on the management of narrow therapeutic index drugs based on in vivo monitoring and pharmacokinetic principles    (therapeutic drug monitoring [7]) were excluded because they are in a complementary in-depth review on therapeutic drug monitoring and dosing (submitted to Implementation Science). Variety in the structure and function of CCDSSs complicates methodologically sound investigations and comparisons of these interventions. A CCDSS may be integrated with one or more of electronic medical records (EMR), computerized provider order entry systems (CPOE) or electronic transmission of prescriptions to the point of dispensing. CCDSSs require input of patient data to deliver advice, and this may be accomplished via integration with patient information repositories or by manual entry. Optimally, the knowledge base of a CCDSS used to generate recommendations is evidence-informed, though this may not always be the case. Advice may be delivered to many kinds of providers through a variety of media across diverse settings of care. Systems may be developed 'in house' to meet the requirements of a specific organization or acquired from a commercial vendor. Decision makers, clinicians, and patients should require sound evidence of CCDSS benefits, risks and costs prior to general adoption, as for any health intervention. Randomized controlled trials (RCTs) represent the gold standard for unbiased comparisons of alternative interventions [8].
Our previous review [2] included 24 RCTs of a CCDSS for drug therapy. Of the 13 trials measuring patient-important outcomes, only one detected benefit with a CCDSS. Small study size limited detection of change in patient-important clinical endpoints.
Lack of data on which to base overall conclusions on the effects of CCDSSs together with the increased pace of research in the field prompted this update of our previous review. To aid decision makers and providers, we evaluated the effects of CCDSSs on process of care and patient outcomes via cumulative synthesis of relevant RCTs. This review is one of a series of reviews considering the effects of CCDSSs across multiple application areas (therapeutic drug monitoring and dosing, primary preventive care, diagnostic test ordering, acute care management, and chronic disease management).

Methods
This review was conducted in accord with a published protocol http://www.implementationscience.com/content/5/1/12 [9]. Some trials have been included in more than one review because they were relevant to more  than one CCDSS intervention area. Specific details for the drug therapy review follow.

Research questions
For this review, we were primarily interested in determining: 1) Do CCDSSs improve performance on drugrelated process of care measures or patient outcomes compared to usual care? 2) What features or characteristics of studies or systems are associated with improved process or patient measures? Based partly on our previous review [2], we expected studies demonstrating benefit from CCDSSs would: a) be integrated with an existing EMR or CPOE system (versus a standalone system); b) deliver decision support before or during a patient care encounter where the decision that is being supported was taken (versus supply of decision support at any other time); c) actively suggest treatments or other actions (versus supply general information or access to general information); d) be used in a patient care setting affiliated with an academic institution (versus any other setting); e) have developers of the CCDSS who were also the study investigators (  important outcomes); g) describe higher rates of user satisfaction (versus no or low rates of user satisfaction).

Partnering with decision makers
This review was conducted in partnership with senior hospital managers and clinical leaders with an academic research team in the field of knowledge translation, from healthcare research to clinical practice. Decision makers provided key input as to the kind of data needed about CCDSS to drive effective choices and these needs were incorporated into the research plan where feasible.

Search strategy
The search methods employed have been described in detail elsewhere [9]. Briefly, a comprehensive search (2004 to 2010) of major biomedical databases (MED-LINE, EMBASE, Ovid's EBM Reviews, and Inspec) yielded citations for screening. Pairs of reviewers independently evaluated each citation and abstract. A third reader resolved disagreements where necessary. Interreviewer agreement on study eligibility was measured via unweighted Cohen's kappa (). Studies from our previous reviews were carried forward to this review if they met the inclusion criteria, effectively extending our search from database inception to 2010.

Study selection
Studies were included for review if they described an RCT comparing outcomes for a group of providers or patients using a CCDSS compared with care without the CCDSS. Non-experimental or quasi-experimental investigations were excluded. For inclusion, we required that independent providers or post-graduate trainee (e.g. medical residents) providers be identified as primary users of the CCDSS. The intervention CCDSS was required to provide patient-specific output in the form of assessments, management options, or recommendations to the clinical user. Studies were excluded if the system was used solely by students, only provided summaries of information for patients, only provided feedback on groups of patients without feedback about individual patients, only provided computer-aided instruction, or were used for image analysis. The six CCDSS intervention areas in this series of reviews used a common eligibility screening process [9] to identify reports of trials of CCDSSs for any purpose. Studies were then further screened to determine if the system provided advice regarding drug therapy.

Data extraction
Independent reviewers extracted key data concerning study methods, CCDSS and population characteristics, possible sources of bias, and outcomes in duplicate. Primary authors of each study were asked to review the extracted data for their study and offer comments on the extracted data.

Assessment of study quality
Included studies were evaluated on five dimensions of quality-including concealment of allocation, appropriate unit of allocation, appropriate adjustment for baseline differences, adequate follow-up, and appropriate outcome assessment-to yield a 10-point methods score [9].

Assessment of CCDSS intervention effects Outcome selection and improvement determinations
Each included trial describing a CCDSS that provided advice exclusively or predominantly about drug therapy was classified as drug therapy management only (Rxonly). Systems that gave advice on drug therapy as part of a more complex intervention were categorised as 'multi-faceted' CCDSSs. Improvement was considered to have occurred where 50% or more of the selected outcomes showed a benefit with a CCDSS compared to control. To determine whether improvement occurred, all outcomes were selected from the first of: primary, then pre-specified, then any outcome(s), as defined by study authors (i.e., if a primary outcome was reported for a trial this was used to determine improvement to the exclusion of any other reported outcomes). Where no outcomes were defined as primary, but the study reported a sample size calculation for an outcome, we defined that outcome as primary. These criteria are more specific than those used in our previous review [2]; therefore, the assignment of effect was adjusted for some studies included in the 2005 review. Process of care outcomes for multi-faceted CCDSS studies were selected only if they were clearly drug-related. Multifaceted systems that reported a patient outcome but did not report a drug-related process of care outcome intermediary were excluded as non-responsive to our research questions. Where there were multiple intervention arms, the arm testing the most sophisticated CCDSS was used to determine improvement. Two reviewers, working independently and blinded to study results, classified trials as drug treatment-only or multifaceted, and initially identified the outcomes used to determine improvement, with disagreements resolved by consensus.

Data synthesis and analysis
Data were summarized using descriptive summary measures, including proportions for categorical variables and means (± SD) for continuous variables. For interpretation, a 2-sided p < 0.05 indicated statistical significance. For individual studies we report the measures of association and p -values reported in the studies. We did not attempt a meta-analysis because of differences across studies of participants, settings, disease conditions, interventions, and outcomes. Tests of association between study and CCDSS factors and improved outcomes were tested using the univariate Fisher's exact test. Multivariate analyses were conducted using multinomial logistic regression. All analyses were conducted using SPSS v. 17.
Sensitivity analyses were conducted to determine if the class of outcome selected to judge improvement affected our results. We also identified cluster randomized trials where units of allocation and units of analysis were appropriately matched or mismatched. The proportions of successful trials with matched versus mismatched units were compared.
Summary of trial quality is reported in Additional file 1, Table S1; system characteristics in Additional file 2, Table S2; study characteristics in Additional file 3, Table  S3; outcome data in Additional file 4, Table S4 and  Table 1, and other CCDSS-related outcomes in Additional file 5, Table S5.
Results did not significantly vary, for either process of care or patient outcomes, by the type of outcome (primary, pre-specified, or other) selected to determine improvement.
All studies demonstrating improved patient outcomes also showed improvement in measured process of care outcomes.
The proportion of successful trials was not significantly different between cluster trials where units of allocation were mismatched with units of analyses (7/15 for process and 1/5 for patient outcomes) compared with non-cluster trials or cluster trials with an appropriately adjusted analysis (29/43 for process (p = 0.22) and 5/24 for patient outcomes (p = 1)).

Predictors of success
This analysis was limited by incomplete data in many studies and by limited power for multivariate analysis. In univariate analysis, CCDSSs not integrated with an EMR were more likely to improve process of care outcomes, 16/20 (80%) non-integrated systems showed improvement versus 18/35 (51%) improved with EMR linkage (p = 0.03). The same trend was seen with integration of EMR and CCDSS for patient outcomes (6/15 (40%) improved outcomes without EMR link versus (0/ 13 (0%) with EMR link, p = 0.017). This association between EMR integration and CCDSS failure was not statistically significant via multivariate regression. Improvement in process of care or patient outcomes was not affected by integration with CPOE, timing or method of decision support delivery, or method of data entry. Improvement in process or patient outcomes did not vary by country, provider type, and outpatient versus other settings of care. Systems trialed outside of academic settings were more likely to improve patient outcomes (5/12 (42%) outside academic settings versus 1/17 (6%) in academic settings, p = 0.03). This finding was not replicated in multivariate analyses. Investigators who developed the system under study were not significantly more likely to see improvement with a CCDSS than investigators studying systems developed by unrelated parties (p = 0.56 for process and p = 1 for patient outcomes). Patient important outcomes were as likely as surrogate outcomes to show improvement with a CCDSS. Post hoc, none of primary disease state, primary patient population, or pharmacotherapeutic purpose predicted success. We found no association between the presence of a sample size calculation and success or between number of trial participants and success. The proportion of studies added in this update demonstrating benefit with CCDSS for process of care and patient outcomes increased compared with studies included in the previous review version, although this trend was not statistically significant.

Costs and practical process related outcomes Harms
Potential or actual harm resulting from CCDSS use was explicitly discussed in four (6%) [16,21,36,68,89] included studies (see Additional file 5, Table S5). Two studies reported quantitative data regarding harms. Raebel et al. [36] reported a trial stopped early due to a high rate (40%) of clinically inappropriate reminders generated by the CCDSS. Zanetti et al. [68] reported one inappropriate redose of intra-operative prophylactic antibiotic for every 137 appropriate redose reminders.

User satisfaction
Fifteen authors reported on user satisfaction with the CCDSS studied (see Additional file 5, Table S5) [18,19,29,30,33,39,55,57,63,64,67,69,70,75,[77][78][79]83,84]. All attempts to measure user satisfaction were conducted via surveys and the properties of the measure used were only discussed in a single trial [55]. Survey response rates ≥50% were found in eight studies. Of these eight, six reported [18,29,33,55,57,84] that ≥70% of respondents thought the CCDSS improved care, was useful, or should be continued in use. Satisfaction data from the other two trials [77][78][79]83] suggested users could not or would not use the CCDSS due to technical or user interface problems. Available data on user satisfaction were too sparse to determine if satisfaction impacted study results.

Discussion
We reviewed 65 RCTs of CCDSSs for drug therapy management reported over a 34-year span. Most trials measured process of care outcomes and results supported the use of CCDSSs to improve these outcomes in a majority of cases (improvement was based on at least 50% of the relevant study outcomes being statistically significantly positive). However, while nearly onehalf of (29 studies) included studies measured a patient outcome, only a small proportion demonstrated any direct benefit to patients. While improvement in process outcomes could lead to benefits for patients, no consistent link was observed here. In the absence of data needed for an economic analysis, improved process of care measures alone are not sufficient to recommend adoption of these systems. The success rates we found for processs of care (64%) and clinical outcome measures (21%) are similar to those in our previous review [2] and also a recent umbrella review of systematic reviews of computerized decision support (57% and 30% respectively) by Jaspers et al. [91].
Several possible predictors of CCDSS success were examined. In most cases, these a priori factors did not explain success or failure across included studies. Our previous review [2] concluded that successful trials of CCDSS were more likely to have been conducted by the developers of the system under study. In our current review, no such association was noted. Previously, a significant trend towards increased study quality over time was noted, but not replicated in this update, and we attribute this to a more restrictive inclusion criterion (randomized controlled trials). Counter to our expectations, we found that integration of CCDSSs with EMRs and use in an academic setting was associated with CCDSS failure. This trend was not statistically significant when tested using multi-variate techniques and so we are unable to determine whether this finding represents a true association or is better explained by the lack of power in our multi-variate analysis. We report these findings as hypothesis generating only and suggest they be examined in future.
Compared with the review of Kawamoto et al. [92], we did not find that automatic provision of advice as part of the existing clinical workflow predicted CCDSS success. Because both the current analysis and that of Kawamoto were underpowered to detect such associations, we have refrained from drawing any conclusions in this regard.
Prospective data on the possible harms of CCDSSs are needed to facilitate informed adoption decisions. Only two trials quantitatively reported on harm from CCDSSs [36,68] with one trial ending early due to increased risk of harm with the CCDSS. We suggest this absence of evidence of harm should not be taken as proof that CCDSSs are safe to employ for drug management in patient care.

Strengths and limitations of review
The results of our review should be interpreted with consideration of methodological strengths and limitations, including steps taken to mitigate the risk of bias. We based our review on the strongest studies available, RCTs. Reviews are necessarily retrospective and we employed multiple methods to limit the introduction of bias, including: duplicate study eligibility assessment, duplicate data abstraction, solicitation of study author feedback on abstracted data, and objective selection of outcomes used to determine improvement. We cannot exclude the possibility that a different method of selecting outcomes from each study to measure improvement could lead to different results, although sensitivity analyses did not suggest this to be the case. Several pre-specified analyses of possible predictors of system success were conducted. Several analyses demonstrated statistically significant results using univariate techniques that were not substantiated using a multi-variate model. Therefore, the few associations we reported between possible predictors of success and improved outcomes with CCDSS should be interpreted with caution.
We have relied upon vote counting as our method of obtaining an estimate of how often CCDSS for drug therapy management improve process or patient outcomes. Significant limitations to this approach as described by Hedges [93] include a tendency to inflate type II error and inadequate incorporation of the effect of unequal study sizes in overall results. The heterogeneity between studies included in our review precluded the use of more robust combination techniques. Formal assessment for publication bias using funnel plots was not possible with the vote-counting technique.
The effectiveness of any CCDSS will be determined in part by the efficacy of the underlying action suggested by the system. Where no benefit was detected with a particular CCDSS, we cannot exclude the possibility that the negative finding is due to a lack of efficacy of the intervention suggested by the system. Measurement of the concordance between decision advice given and followed would be a useful measure to address this issue. These outcomes were included in our analyses of process of care outcomes. It does not necessarily follow, however, that an effective CCDSS that recommends the appropriate prescription of an efficacious intervention will necessarily improve patient care. A multitude of intervening factors (e.g. patient non-or over-adherence or new errors introduced by CCDSS) may mitigate (or exaggerate) estimates of CCDSS effectiveness.
Finally, the systems reviewed constitute a heterogeneous group with differing functionality and clinical intent. While we have attempted to usefully divide the systems for the reader, we acknowledge other divisions were possible.

Implications for practice and research
Because CCDSSs have not been shown to reliably and positively impact patients, and in the absence of useful data on potential harms, costs, and clinician impacts, we cannot recommend the general adoption of CCDSSs for drug therapy management. It is possible that these systems are still evolving and success will improve with time. Clearly further innovation is needed if these systems are to be dependably useful in clinical practice. Rigorous trials of these innovations will be necessary, and we suggest that future research explicitly address patient outcomes, including potential harms, and costs and adverse clinician impacts of CCDSSs. Given the availability of effective non-computerized approaches for promoting safe and effective medication use [5,94], future studies may wish to incorporate these interventions as active comparators to CCDSSs.