Computerized clinical decision support systems for acute care management: A decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes

Background Acute medical care often demands timely, accurate decisions in complex situations. Computerized clinical decision support systems (CCDSSs) have many features that could help. However, as for any medical intervention, claims that CCDSSs improve care processes and patient outcomes need to be rigorously assessed. The objective of this review was to systematically review the effects of CCDSSs on process of care and patient outcomes for acute medical care. Methods We conducted a decision-maker-researcher partnership systematic review. MEDLINE, EMBASE, Evidence-Based Medicine Reviews databases (Cochrane Database of Systematic Reviews, DARE, ACP Journal Club, and others), and the Inspec bibliographic database were searched to January 2010, in all languages, for randomized controlled trials (RCTs) of CCDSSs in all clinical areas. We included RCTs that evaluated the effect on process of care or patient outcomes of a CCDSS used for acute medical care compared with care provided without a CCDSS. A study was considered to have a positive effect (i.e., CCDSS showed improvement) if at least 50% of the relevant study outcomes were statistically significantly positive. Results Thirty-six studies met our inclusion criteria for acute medical care. The CCDSS improved process of care in 63% (22/35) of studies, including 64% (9/14) of medication dosing assistants, 82% (9/11) of management assistants using alerts/reminders, 38% (3/8) of management assistants using guidelines/algorithms, and 67% (2/3) of diagnostic assistants. Twenty studies evaluated patient outcomes, of which three (15%) reported improvements, all of which were medication dosing assistants. Conclusion The majority of CCDSSs demonstrated improvements in process of care, but patient outcomes were less likely to be evaluated and far less likely to show positive results.


Background
Computerized clinical decision support systems (CCDSSs) are information systems intended to improve clinical decision-making. CCDSSs match individual patient data to a computerized knowledge base that uses software algorithms to generate patient-specific recommendations that are delivered to healthcare practitioners [1][2][3].
This review, acute medical care, is one of a series of six on specific interventions of CCDSSs, including primary preventive care, chronic disease management, diagnostic test ordering, drug prescribing and management, and therapeutic drug monitoring and dosing. The review process involved senior healthcare managers in setting priorities and co-sponsoring the review process with an academic review team, and engagement of key clinical leaders in each review to establish review questions, guide data extraction needed for clinical application, and draw conclusions from a practical clinical perspective [4].
Expectations are high for the utility of CCDSSs in acute care because acute care in hospitals and emergency rooms is the most intensive and expensive part of the healthcare system on a per patient basis, but many concerns and problems have been identified [5]. As with any healthcare intervention, CCDSSs purporting to improve patient care or outcomes should be rigorously evaluated before being routinely implemented in clinical practice [6]. This systematic review focuses on the use of CCDSSs for management of medical problems in acute care settings and summarizes the most rigorous evidence to date concerning the effects of CCDSSs in acute medical care. An example of such a CCDSS includes advice for paramedics responding to emergency calls.

Research question
Do CCDSSs improve process of care or patient outcomes for acute medical care?

Partnering with decision makers
The review was conducted using a partnership model [4] with 2 main groups: decision makers from local health institutions and research staff of the Health Information Research Unit (HIRU) at McMaster University. There were two types of decision makers-senior health managers of Hamilton Health Sciences (a large academic health sciences centre) provided overall guidance and endorsement, and a clinical service leader (RL, a pediatrician) provided specific guidance for acute care management. HIRU research staff and students were responsible for completing the literature search, and appraising, extracting, and synthesizing the data. The goal of the partnership model was to maximize knowledge translation with respect to potential local CCDSS implementation.

Search strategy
We reassessed all citations in our most recent review [3] and retrieved new citations from that review's September 2004 closing date to 6 January 2010, in all languages, by employing a comprehensive search strategy of MEDLINE, EMBASE, Evidence-Based Medicine Reviews databases (Cochrane Database of Systematic Reviews, DARE, ACP Journal Club, and others), and the Inspec bibliographic database. Pairs of reviewers independently evaluated each citation and abstract to determine the eligibility of all studies identified in our search. Disagreements were resolved by a third reviewer or by consensus. Inter-reviewer agreement on study eligibility was measured using the unweighted Cohen's kappa (), and was excellent ( = 0.93; 95% confidence interval [CI], 0.91 to 0.94) overall. A panel of reviewers-including a physician, a pharmacist, and two individuals trained in health research methodsreviewed eligible studies and assigned them to appropriate care area(s). Acute care referred to episodic health conditions that could be possibly cured or stabilised in less than six months. Figure 1 summarizes the study selection process, including specifics for acute care management.
Several studies addressed two or more clinical care areas; the review for each care area focused only on the study outcomes that were most relevant for that area. Most study overlaps for acute care were with therapeutic drug monitoring and drug prescribing.
Six studies were excluded from acute care after the initial selection. Three studies met initial criteria but were later excluded for confounding of healthcare provider across treatment groups (e.g., pharmacist using CCDSS versus physician giving usual care) [7][8][9]. Three studies that met review criteria did not report relevant data for acute care conditions [10][11][12].

Study selection
Studies were included if they met all of the following five criteria: evaluated a CCDSS used for acute care; used an randomized controlled trial (RCT) design where patient care with a CCDSS was compared to patient care without a CCDSS; assessed effects among healthcare professionals in clinical practice or post-graduate training; provided patient-specific information in the form of assessments (management options or probabilities) or recommendations to the clinicians, who remained responsible for actual decisions; and measured clinical performance (a measure of process of care) or patient outcomes (including any aspect of patient well-being). Studies were excluded if they provided only summaries of patient information, feedback on groups of patients without individual assessment, or only computer-aided instruction; used simulated patients; or used CCDSSs for image analysis.

Data extraction
Pairs of reviewers independently extracted the following data from all eligible studies: study setting, study methods, CCDSS characteristics, patient characteristics, and outcomes. Disagreements were resolved by a third reviewer or by consensus. We attempted to contact primary authors of all 36 included studies and 28 authors (78%) replied and confirmed data, including six who had previously replied and confirmed data in our most recent review [3].

Assessment of study quality
Methodological quality was evaluated using a 10-point scale consisting of five potential sources of bias, and based on an extension of the Jadad scale [13]. A score of 10 on the scale indicated the highest study quality [4].

Assessment of CCDSS intervention effects
Studies with multiple treatment arms were counted as a positive study if any of the treatment arms showed a benefit over the control arm. Outcomes were considered primary if reported by the author as 'primary' or 'main' outcomes. If no primary outcomes were reported and a power statement was provided, then the outcome on which the power statement was based was considered primary. The use of effect sizes was judged to be inappropriate because of the high degree of heterogeneity in almost every aspect of the individual studies. Effects for each CCDSS were evaluated based on relevant outcomes showing a statistically significant difference (2p<0.05). Effects were identified as statistically significantly positive (+) or negative (-), or no effect (0), based on the following predefined hierarchy of outcomes: 1. If a single primary outcome was reported, in which all components were applicable to acute medical care, this was the only outcome evaluated.  2. If >1 primary outcome was reported, only the applicable primary outcomes were evaluated, and judged positive if ≥50% were statistically positive.
3. If no primary outcomes were reported (or only some of the primary outcome components were relevant) but overall analyses were provided, the overall analyses were evaluated as primary outcomes. Subgroup analyses were not considered.
4. If no primary outcomes or overall analyses were reported, or only some components of the primary outcome were relevant for the care area, any reported applicable pre-specified outcomes were evaluated.
5. If no clearly pre-specified outcomes were reported, any available relevant outcomes were considered.
6. If statistical comparisons were not reported, 'effect' was designated as not evaluated (denoted as ...).
These criteria are more specific than those used in our previous review; therefore, the assignment of effect was adjusted for some studies included in the earlier review.

Data synthesis and analysis
We summarized data and p-values reported in individual studies. CCDSS characteristics were analyzed and interpreted with the study as the unit of analysis. Data were summarised using descriptive summary measures, including proportions for categorical variables and means (± standard deviation [SD]) for continuous variables. All analyses were carried out using SPSS v.15. A 2-sided p < 0.05 indicated statistical significance.
A sensitivity analysis was conducted to assess the possibility of biased results in studies with a mismatch between the unit of allocation (e.g., clinicians) and the unit of analysis (e.g., individual patients without adjustment for clustering). Success rates comparing studies with matched and mismatched analyses were compared using chi-square for comparisons. No differences in reported success were found for either process of care outcomes (Pearson X 2 = 2.70, 2p = 0.10) or patient outcomes (Pearson X 2 = 0.39, 2p = 0.53). Accordingly, results have been reported without distinction for mismatch.
Summary of trial quality is reported in Additional file 1, Table S1; system characteristics in Additional file 2, Table S2; study characteristics in Additional file 3, Table S3; outcome data in Additional file 4, Table S4 and  Table 1; and other CCDSS-related outcomes in Additional file 5, Table S5.
The studies of highest quality in this group all produced improvements in process of care outcomes. Overhage et al. tested a CCDSS that generated corollary orders to prevent errors of omission for any of 87 target tests and treatments in hospital inpatients [26]. In comparison to a computerized order entry system alone, compliance with corollary orders was increased in the CCDSS group and the number of pharmacist interventions with physicians for significant errors was decreased. Another high-quality study, Dexter et al., gave reminders for preventive therapies in hospital inpatients and showed an increase in the proportion of eligible hospitalized patients who received the targeted preventive therapy [19]. Terrell et al. assessed a CCDSS that provided alerts to avoid inappropriate prescriptions in geriatric outpatients during discharge from emergency care [48]. Inappropriate medication prescriptions decreased in the CCDSS group when compared to usual care. Kroth et al. had the largest patient population in this group (N = 90,162) and tested a CCDSS which helped improve the accuracy of temperature capture by nurses for non-critical care hospital patients [39]. The study reported a decrease in the number of (presumed erroneous) low temperatures recorded by nurses in the CCDSS group compared to usual care. Kuperman et al. tested a CCDSS that notified health providers when critical laboratory results for all medical and surgical inpatients were ready [23]. In comparison to usual care, the CCDSS group reduced the time from recording the alert to ordering the appropriate treatment. Zanetti et al. provided alerts for redosing of prophylactic antibiotics during prolonged cardiac surgery and showed an increase in the number of intraoperative redoses compared to usual care [47]. The CCDSS in White et al. identified signs and risk factors for digoxin intoxication for inpatients [31]. The trial reported an increase in the number of physician actions related to the alerts in the CCDSS group compared to usual care. Rood et al. developed a guideline for tight glycaemic control in intensive care unit (ICU) patients and compared a CCDSS version to the paper-based system [34]. Use of the CCDSS resulted in stricter adherence to the guideline, both in terms of timing of glucose measurements and use of advised insulin doses. This resulted in a small improvement in patient glycaemic control; however, the improvement was judged to be not clinically important.

Management assistants -guidelines and algorithms
Nine studies [20,33,37,38,[40][41][42][43][44]46] examined a management assistant employing guidelines and algorithms-these CCDSSs generated recommendations for the management of acute health issues using guidelines or algorithms, such as evidence-based electronic prescribing in paediatric care [42]. Of the eight studies that assessed process of care outcomes, three (38%) demonstrated improvements [37,38,40,42], and none of the four studies that assessed patient outcomes showed an improvement.
Process improvements occurred in a multicentre study of high methodological quality by Paul et al. [40]. The CCDSS assisted with choice of empiric antibiotic treatment in hospital inpatients and improved appropriate antibiotic therapy in comparison to usual care. Davis et al. assessed appropriate prescribing for upper respiratory tract infections in paediatric outpatients [42]. Compared to usual care, the CCDSS increased prescriptions that were consistent with evidence-based recommendations. Rothschild et al. tested a CCDSS that produced recommendations for non-emergent inpatient transfusion orders, and showed improvement in guideline adherence as measured by the percentage of appropriate and inappropriate transfusion orders [37,38]. The methodologically sound study by Kuilboer et al. had the largest patient population of all the acute care trials (N = 156,772), but did not report any improvements in process of care for monitoring and treatment of asthma and chronic obstructive pulmonary disease (COPD) in primary care [41].

Diagnostic assistants
Three studies tested diagnostic assistants [15,35,45]these CCDSSs provided advice for the diagnosis of acute health conditions, such as acute small bowel obstruction in surgical inpatients [15]. All studies assessed process of care outcomes with two (67%) showing improvements with the CCDSS [35,45]. Roukema et al. tested a CCDSS that provided advice for the diagnostic management of children with fever without an apparent source in the emergency department (ED) and showed an increase in test ordering [35]. Stengel et al. examined a CCDSS that assisted electronic documentation of diagnosis and findings in patients admitted to orthopaedic wards [45]. In comparison to standard paper forms, the CCDSS demonstrated success in improving diagnosis per patient.
Of the two studies examining patient outcomes, neither demonstrated an improvement.

Medication dosing assistants
Fourteen studies evaluated medication dosing assistants, providing recommendations specific to drug dosing adjustments, such as insulin dosing or dosing advice for warfarin initiation [14,[16][17][18]21,22,24,27,28,30,32,36,49,50]. These CCDSSs showed improvements in process of care outcomes in 9 of 14 studies (64%) [14,21,22,27,28,32,36,49,50], improvements in patient outcomes in 3 of 10 studies (30%) [32,49,50], and a negative effect on patient outcomes in 1 of 10 (10%) studies [49]. Many studies in the Medication Dosing Assistants section overlap with studies in the therapeutic drug monitoring and thus, are not the primary focus of this review. A more in-depth analysis of these studies is provided in the therapeutic drug monitoring and dosing review (submitted to IS for consideration of publication as part of this series of six reviews).

Discussion
Our systematic review identified 36 RCTs of CCDSSs for acute care. The trials were diverse in CCDSS design, clinical settings, clinical problems, and measured effects. Study quality scores increased over time, but that may be due to an improvement in the reporting of trials. Most studies evaluated process of care effects, with 63% showing benefit (benefit was based on at least 50% of the relevant study outcomes being statistically significantly positive) [14,19,[21][22][23][26][27][28]31,32,[34][35][36][36][37][38][39][40]42,45,[47][48][49][50]. Few examined meaningful patient outcomes, and none showed significant reductions in major patient morbidity or mortality, although some found small reductions in length of hospital stay [16,26,32,40,50]. Lack of findings for patient-important outcomes may be largely an issue of study design, especially the size of the study. Most studies involved few participants, suggesting that they were preliminary in nature, attempting to establish whether the CCDSS could change the process of care, as a prelude to larger studies assessing whether lives could be saved.
Some studies did demonstrate substantial effects on the process of care. For example, the multicentre trial by Paul et al. used a causal probabilistic network and local susceptibility data to develop a CCDSS called the TREAT system, which suggested empiric antibiotic regimens based on basic data entered by practitioners during the work-up of patients with new infections [40]. When compared to pathogens isolated further in the course of disease, the TREAT recommendations significantly bettered the physician-only prescriptions (odds ratio of 1.48). Costs were also shown to be reduced, and length of stay was reduced by one day.
In the case of reminders and alerts, where the effect on the process of care is a strong surrogate for patient outcomes, it could also be argued that evidence of effect on patient outcomes is not needed. For example, Terrell et al. developed a group of alerts within a computerized physician order entry system to warn emergency physicians when they were about to prescribe potentially dangerous medications (as defined by externally validated criteria) to elderly patients on discharge from the department [48]. The system carried a message explaining why the medication could be dangerous, who had made the recommendation, and links to further explanatory information. Perhaps most importantly, it also suggested safer substitute therapies for each warning. Physicians in the computer-assisted group prescribed fewer inappropriate medications than physicians with no access to the alerts.
It is difficult to make general recommendations regarding the broad applicability and effectiveness of CCDSSs in acute care settings given the current literature and heterogeneity of the individual studies. Several practical details identified as pertinent for extraction by the decision-makers (such as implementation details and costs) were not reported in sufficient detail or with adequate consistency across studies to summarize. There are certainly encouraging trends witnessed by a number of recent high-quality studies demonstrating positive process of care outcomes. Important effects on patient outcomes have yet to be convincingly demonstrated, however, and purchasers and decision makers are advised to take this into consideration.
We did not complete a formal analysis of the factors associated with success across the trials. However, confirming findings from our previous review [3], when study authors were also the developers of the CCDSSS under assessment, the findings were more likely to be favourable for the CCDSS. This could be due to any number of factors including, for example, greater attention to customization for local settings, choosing outcomes more likely to be influenced by CCDSSs, influence of developers on enthusiasm for use of the system, or bias in the analysis of findings, which has been documented for commercial trials of pharmaceuticals [51]. As such, the included trials are potentially more likely to overestimate the size of the effect and increase the risk of a type I error in this review.

Limitations
A number of studies did not report relevant data or had insufficient data to conduct the appropriate analyses. For example, we were unable to evaluate the effect size for process of care or patient outcomes for several of the studies. As well, our strict inclusion criteria that included only RCTs focuses on only the most scientifically sound studies and would miss, for example, more effective CCDSSSs that were not as rigorously tested. The heterogeneity between studies in CCDSS features and outcome measures was too great to justify a meta-analysis to pool effect sizes. Instead, the overall results of the trials were reported by taking the number of trials with statistically significant results and dividing them by the total number of trials, a method known as vote-counting, which is limited by giving equal weight to each study, regardless of individual merit and size. A study was considered to have a positive effect (i.e., CCDSS showed improvement) if at least 50% of the relevant study outcomes were statistically significantly positive. Many of the studies were small, increasing the risk of type 2 (false negative) error. On the other hand, it is likely that publication bias exists in this field, as shown in many others, such that the number of 'negative' trials is underestimated from the published literature. The majority of studies were conducted in the US and academic medical institutions, where the nature of the clinical landscape could have affected the application and results from the CCDSSs, reducing their generalizability to other settings. Last, but perhaps most important, very few studies evaluated patient-important outcomes.

Future research directions
Future research should focus on evaluating the effect of CCDSSs on patient outcomes, providing full details of the CCDSS to help establish the relationship between CCDSS characteristics and CCDSS success, and ensuring studies of high methodological quality. Fortunately, with recent initiatives on the adoption of electronic medical records to achieve meaningful enhancements of healthcare [52], there are many opportunities for good measurement and assessment of CCDSSs in acute care. The high quality studies to date show that it is feasible to rigorously evaluate CCDSSs; the findings so far underscore that existing CCDSSs have not been shown to improve patient-important outcomes.