Computerized clinical decision support systems for chronic disease management: A decision-maker-researcher partnership systematic review

Background The use of computerized clinical decision support systems (CCDSSs) may improve chronic disease management, which requires recurrent visits to multiple health professionals, ongoing disease and treatment monitoring, and patient behavior modification. The objective of this review was to determine if CCDSSs improve the processes of chronic care (such as diagnosis, treatment, and monitoring of disease) and associated patient outcomes (such as effects on biomarkers and clinical exacerbations). Methods We conducted a decision-maker-researcher partnership systematic review. We searched MEDLINE, EMBASE, Ovid's EBM Reviews database, Inspec, and reference lists for potentially eligible articles published up to January 2010. We included randomized controlled trials that compared the use of CCDSSs to usual practice or non-CCDSS controls. Trials were eligible if at least one component of the CCDSS was designed to support chronic disease management. We considered studies 'positive' if they showed a statistically significant improvement in at least 50% of relevant outcomes. Results Of 55 included trials, 87% (n = 48) measured system impact on the process of care and 52% (n = 25) of those demonstrated statistically significant improvements. Sixty-five percent (36/55) of trials measured impact on, typically, non-major (surrogate) patient outcomes, and 31% (n = 11) of those demonstrated benefits. Factors of interest to decision makers, such as cost, user satisfaction, system interface and feature sets, unique design and deployment characteristics, and effects on user workflow were rarely investigated or reported. Conclusions A small majority (just over half) of CCDSSs improved care processes in chronic disease management and some improved patient health. Policy makers, healthcare administrators, and practitioners should be aware that the evidence of CCDSS effectiveness is limited, especially with respect to the small number and size of studies measuring patient outcomes.


Background
Chronic conditions present patients, practitioners, and healthcare systems with some unique demands, including recurrent visits, adherence to complex care plans, long-term disease and treatment monitoring, behavior modification, and patient self-management. For the many patients with multiple co-morbidities [1], overlapping or diverging care plans may further complicate these processes.
Computerized clinical decision support systems (CCDSSs) may help practitioners meet the requirements of chronic care. These systems analyze a patient's characteristics to provide tailored recommendations for diagnosis, treatment, patient education, adequate follow-up, and timely monitoring of disease indicators. For example, Holbrook et al. [2,3] gave providers and diabetic patients access to a web-based system that offered care advice, allowed monitoring of diabetes risk factors, and tracked key care targets. As with any health intervention, however, rigorous testing is warranted to determine whether CCDSSs improve chronic care processes and patient outcomes.
In our previous review of the effects of CCDSSs [4], we analyzed 100 randomized and non-randomized studies published until September 2004, 40 of which assessed the effects of CCDSSs on disease management. Of these 40 studies, 37 measured processes of care of which 62% (23) showed an improvement, and 27 measured patient outcomes of which 19% (5) showed an improvement. The quality of the studies varied widely, but improved over time.
Many new randomized controlled trials (RCTs) have been published in this field since our previous work, potentially documenting important advances. Recognizing that the management of chronic disease has unique characteristics, we wished to review the impact of CCDSSs on the quality and effectiveness of chronic care. We had the opportunity to include the perspectives of senior hospital managers and front-line healthcare practitioners to ensure that relevant data were extracted and summarized-a level of stakeholder engagement that has not been included in other reviews [5][6][7][8].

Methods
We previously published the details of our review protocol, openly accessible at http://www.implementationscience.com/content/5/1/12 [9]. These methods are briefly summarized here, along with details specific to this review of CCDSSs for chronic disease management.

Research question
Do CCDSSs improve chronic disease management processes or patient outcomes?

Partnering with decision makers
We conducted this review in partnership with individuals responsible for implementing CCDSSs in our region [9]. Decision makers, both managers and clinicians, met with the review team periodically to discuss direction and specific details for the data extraction, analysis, presentation and interpretation of results.

Search strategy
Full details of our search strategy are in our review protocol [9]. In summary, we searched MEDLINE, EMBASE, Ovid's Evidence-Based Medicine Reviews, and Inspec until 6 January 2010, and reviewed the reference lists of included RCTs and relevant systematic reviews. We screened articles for eligibility in two stages: a duplicate, independent review of titles and abstracts followed by a duplicate, independent, full-text review of potentially eligible articles, with a third reviewer resolving disagreements.

Study selection
We selected RCTs of a CCDSS used by a health care provider for management of chronic conditions, published up to 6 January 2010 in any language that measured CCDSS impact on processes of care or patient outcomes. We included RCTs in any language that compared patient care with a CCDSS to routine care without a CCDSS and evaluated clinical performance (i.e., a measure of process of care) or a patient outcome. Additionally, to be included in the review, the CCDSS had to provide patient-specific advice that was reviewed by a healthcare practitioner before any clinical action. Studies were excluded if the system was used solely by students, only provided summaries of patient information, provided feedback on groups of patients without individual assessment, only provided computer-aided instruction, or was used for image analysis. Trials included in our previous review [4] were included if they were eligible. Trials of CCDSSs for managing narrow therapeutic index medications used in some chronic conditions (such as warfarin in atrial fibrillation [10]) were not included in this review, but are discussed in our review for therapeutic drug monitoring and dosing.

Data extraction
To meet the needs of our management and clinical partners, we extracted study characteristics (e.g., study design, size, setting, authorship, funding, and year of publication) and system characteristics (e.g., integration with other systems, user interface elements, methods of data entry and delivery of recommendations, target users, and implementation details such as pilot testing and user training). Disagreements were resolved by a third reviewer or by consensus. We contacted primary authors to provide missing data and to assess the accuracy of the extracted data; 78% (43/55) provided input. For the remaining trials, a trained reviewer assessed the extraction form against the full-text to confirm accuracy.

Assessment of study quality
Using a 10-point scale, pairs of reviewers independently evaluated the selected trials on five dimensions of quality, including concealment of allocation, appropriate unit of allocation, appropriate adjustment for baseline differences, appropriate blinding of assessment, and adequate follow-up [9]. We used a 2-tailed Mann-Whitney U test to compare methodologic scores between trials published before the year 2000 and those published later to determine if trial quality has improved with time.

Assessment of CCDSS intervention effects
We assessed the effectiveness of CCDSSs in each trial for improving process of care and patient outcomes. We defined process outcomes as changes in care activities such as diagnosis, treatment, and monitoring of disease. Examples of patient outcomes included changes in blood pressure, clinical events and health-related quality of life. We judged a CCDSS effective if it produced a statistically significant (p < 0.05) improvement in a primary chronic disease outcome or in ≥50% of multiple relevant pre-specified outcomes. We considered primary any outcome that trial reports described as 'primary' or 'main.' If authors did not designate a primary outcome, we considered the outcome used to calculate the trial's sample size to be primary, if reported. When there were no pre-specified outcomes, the system was considered effective if it produced an improvement in ≥50% of all reported chronic disease outcomes. Our assessment criteria are more specific than those used in our 2005 review [4]; therefore, the assignment of effect was adjusted for some trials included in the review.

Data synthesis and analysis
We summarized data using proportions, medians, and ranges. Denominators vary in some proportions because not all trials reported relevant information. All analyses were carried out using SPSS, version 15.0. We did not attempt a meta-analysis because of study-level differences in participants, clinical settings, disease conditions, interventions, and outcomes measured.

Results
Summary of trial quality is reported in Additional file 1, Table S1; system characteristics in Additional file 2, Table S2; study characteristics in Additional file 3, Table  S3; outcome data in Table 1 and Additional file 4, Table  S4; and other CCDSS-related outcomes in Additional file 5, Table S5.
Several recent trials were conducted in primary community clinics whereas most previous trials were conducted in hospitals. For example, in two trials conducted across multiple practices, CCDSSs provided patient-specific reminders during visits and notified at-risk patients of their care targets and upcoming appointments [2,3,11,18]. Both trials demonstrated improvements in composite process measures comprising timely completion of foot and eye exams, and monitoring of blood pressure, HbA 1c , lipoproteins, and renal function. Both trials also showed improvements in corresponding composite patient outcomes.
In contrast to diabetes systems, however, hypertension systems showed little or no patient benefit. Of the nine trials that reported patient outcomes, such as blood pressure and health-related quality of life [34][35][36][37][38][39][41][42][43][44][45], only one found benefit [42][43][44]. This multi-component system improved patients' perceived health status by giving suggestions for the management of hypertension, obesity, and renal disease. The trial, however, was of poor quality (methods score 4), and the nature of the intervention prevented isolating effects related to hypertension.
Three trials measured effects on process of care and demonstrated improvements in lipid monitoring and treatment [57][58][59][60][61], but only one of three trials measuring patient outcomes found a benefit [59]. This CCDSS generated patient-specific reminders that were mailed to primary care physicians and nurses; highlighted the patient's risk factors, lipoprotein values, and current medications; and recommended initiation or adjustment of lipid-lowering treatment when appropriate. The trial detected improvements in blood lipid monitoring and treatment management, as well as relative reductions in patients' LDL cholesterol.
Two of five trials measuring patient outcomes found an impact [52,56]. One system delivered asthma recommendations in primary care, made prognostic predictions by matching patients to similar known cases, and allowed users to print self-management plans for their patients [56]. The trial demonstrated a reduction in acute asthma exacerbations and patient-initiated primary care visits. Another system delivered guideline recommendations to general practitioners and pneumologists, and proved to be more cost effective at improving quality of life than usual asthma care [52].
Three asthma systems also gave advice for management of COPD [48,49,51,53]. All of these measured process of care but detected no effects. One trial also measured patient outcomes but did not show benefit [53].

Cardiac care
Systems in four methodologically strong trials (median score, 9.5; ranging from 8 to 10) focused on heart failure [65,66], cardiac rehabilitation [63,64], ischemic heart disease [67], and angina [54,55]. All measured process of care using adherence to guideline recommendations but only one found benefit [63,64]. The CCDSS for cardiac rehabilitation used electronic medical records and needs assessment data to generate recommendations for exercise training, education, lifestyle change, and stress management [63,64]. The trial demonstrated improved guideline adherence, but patient outcomes were not studied. The other three trials measured effects on quality of life as a patient outcome, but none found benefit [54,55,[65][66][67].

Other care
We did not group the remaining 12 trials due to their diverse primary indications. They focused on urinary incontinence [78], cancer [75], osteoporosis [74], renal disease [70], functional deficits [77], obesity [68,69], dementia [73], rheumatoid arthritis [72], advance directives [76], and various non-specific indications [71,79]. Most trials found improvements in care process but only two demonstrated benefit to patients: one reduced urinary incontinence in nursing home patients [78], and the other improved likelihood of remission in patients with early rheumatoid arthritis through CCDSS-guided management of methotrexate [72].
In addition to process of care and patient outcomes, we looked for effects on user satisfaction and workflow (see Additional file 5, Table S5). Only seven trials reported a formal effort of assessing user satisfaction: 3 found users satisfied overall [19,28,63,64], one found them unsatisfied [54,55], and the remaining three showed mixed results [2,3,[31][32][33]56]. The authors of five other studies commented that users were satisfied in informal evaluations [13,18,36,59,72].
Two trials made formal attempts to measure systems' impact on user workflow and reported mixed results [23,63,64].

Discussion
This review was done in partnership with key decision makers to summarize the effectiveness of clinical decision support technology for the management of chronic conditions. We considered studies 'positive' if they showed a statistically significant improvement in at least 50% of relevant outcomes. CCDSSs often improved the process of patient care. When assessed, effects on any patient outcomes were rarely found, but may have been underestimated: 56% of trials reporting these outcomes declared them primary [11][12][13][14][15][16][17][18][19][20]22,29,34,35,38,50,52,56,[59][60][61][62]67,70,72], and the remaining trials may not have been large enough or long enough to detect such outcomes. No study showed convincing evidence of benefit for major patient outcomes.
Nevertheless, results from recent diabetes management trials are encouraging. Several of these systems were deployed in general community practice and those that engaged both patients and providers were consistently effective. These systems may become increasingly popular with the advent of patient-controlled electronic medical records. Systems addressing several conditions, including but not limited to diabetes, generally improved care but only one measured patient outcomes [31][32][33] (no effect). In dyslipidemia, systems improved lipid monitoring and treatment, but only one reduced blood lipids [59]. The few dyslipidemia trials were recent and may represent a promising area for future research.
Conversely, most trials in hypertension measured patient outcomes and almost never found benefits, and only some showed improvements in the process of care. Asthma and COPD systems mostly failed to show effectiveness, despite being tested in recent, high-quality trials. The small collection of trials in heart failure, ischemic heart disease, cardiac rehabilitation, and angina also rarely show effects, with improvement only in rehabilitation processes. The remaining systems, too diverse to group, often improved care processes but were seldom found to benefit patients.
While systems in diabetes appear to achieve success with respect to patient outcomes more often than systems in asthma and hypertension, we did not pre-specify this comparison and, given the play of chance and many possible confounders, we cannot confidently assert that the pattern is real. It is plausible that the effectiveness of CCDSS recommendations at improving patient outcomes for some indications is limited by the absence of high-quality clinical evidence in that area. Even the most scientifically sound recommendations, however, will fail to improve health outcomes if patients do not adhere to prescribed treatments-a very common problem [80]. Unfortunately, our suggestions regarding the discrepancy remain purely speculative because studies did not explore reasons for failure, and we do not have enough trials to test these hypotheses reliably.
The growing use of CCDSSs and their potential for benefit and harm highlight the importance of evaluating these systems in well-conducted randomized clinical trials. The increase in number and quality of trials is encouraging, but results remain mixed, and few trials investigated the mechanisms behind their findings. Careful description of study and system design in trial reports, as well as assessments of effectiveness and acceptability of system features, would support progress in this area.
CCDSSs may represent a cost-effective way of improving chronic disease outcomes. However, the economic effects of systems are not readily assessed based on available data. The costs of design, local implementation, ongoing maintenance, and user support can be high, and may be further elevated by the unique nature of chronic care. This warrants cost-effectiveness analyses, but only four trials [14][15][16][17]52,65,66,75] reported such data and little cost data of any kind are available across studies. If cost savings exist, however, current results suggest that they are modest.
The benefits we can expect from the use of computerized decision support are not clear. Policy makers promoting the use of CCDSS, as well as healthcare administrators and practitioners considering local implementation, should be aware that the evidence of CCDSS effectiveness is limited, especially with respect to the small number and size of studies of patient outcomes. Further, evidence of benefit comes mainly from a few 'trail blazer' institutions with much in-house informatics expertise, evaluating home-grown systems developed over many years. As a result, trials in this review may not represent the effects in less technically endowed settings or from commercially available systems, the capabilities of which have been shown to vary greatly [81].
Our review has some potential limitations. Great heterogeneity in CCDSS design, purpose, and targets for evaluation prevented us from conducting a meta-analysis. Instead, we used a binary measure of effect, where we considered studies 'positive' if they showed a statistically significant improvement in at least 50% of relevant outcomes. Thus, some of the studies we considered to show no effect found improvement on a minority of secondary or non-prespecified outcomes. These findings could be real but could also be due to post hoc unplanned analyses and multiple testing. Readers should refer to the Methods section for a more detailed account of our effect assessment.
We were unable to assess the risk of publication bias in this literature. Given that most systems were studied by their own developers, we suspect that publication bias is likely, and even our findings of modest effects may overestimate the true likelihood of seeing benefit from CCDSSs.
Our method of summarizing the evidence by vote counting inflates the risk of Type 2 error [82] and should generally be approached with caution. However, our results remain essentially unchanged from our 2005 review [4] and are comparable to another major review conducted by Kawamoto and colleagues [83], and a recent 'umbrella ' review of high-quality systematic reviews of CCDSSs in hospital settings [84]. Another recent review of reminder systems [5] (a subset of CCDSS) summarized evidence by effect size meta-analysis and qualified the impact of these interventions as falling below the thresholds of clinical importance. Given the similar conclusions of these other systematic reviews and the risk of publication bias in the CCDSS literature, we have little reason to believe that our methods underestimate the benefit from these systems.
Finally, we observed improvements in the quality of trials over time but this trend may have resulted from better reporting in more recent studies.