Skip to main content

Implementing an evidence-based computerized decision support system linked to electronic health records to improve care for cancer patients: the ONCO-CODES study protocol for a randomized controlled trial



Computerized decision support systems (CDSSs) are computer programs that provide doctors with person-specific, actionable recommendations, or management options that are intelligently filtered or presented at appropriate times to enhance health care. CDSSs might be integrated with patient electronic health records (EHRs) and evidence-based knowledge.


The Computerized DEcision Support in ONCOlogy (ONCO-CODES) trial is a pragmatic, parallel group, randomized controlled study with 1:1 allocation ratio. The trial is designed to evaluate the effectiveness on clinical practice and quality of care of a multi-specialty collection of patient-specific reminders generated by a CDSS in the IRCCS Istituto Scientifico Romagnolo per lo Studio e la Cura dei Tumori (IRST) hospital. We hypothesize that the intervention can increase clinician adherence to guidelines and, eventually, improve the quality of care offered to cancer patients. The primary outcome is the rate at which the issues reported by the reminders are resolved, aggregating specialty and primary care reminders. We will include all the patients admitted to hospital services. All analyses will follow the intention-to-treat principle.


The results of our study will contribute to the current understanding of the effectiveness of CDSSs in cancer hospitals, thereby informing healthcare policy about the potential role of CDSS use. Furthermore, the study will inform whether CDSS may facilitate the integration of primary care in cancer settings, known to be usually limited. The increasing use of and familiarity with advanced technology among new generations of physicians may support integrated approaches to be tested in pragmatic studies determining the optimal interface between primary and oncology care.

Trial registration, NCT02645357

Peer Review reports


Background and rationale

Most hospitals collect huge amounts of electronic administrative and clinical data. Without returning these data to clinicians, collected e-information about healthcare services fails to inform clinicians [1]. Administrative and demographic information, diagnoses, treatments, prescription drugs, laboratory tests, hospitalizations, and patient insurances, are cumulated. Doctors are primarily involved in the collection of clinical data. Despite doctors invest energies and time collecting patient information during visits, this information, at best, comes back after some times (e.g., at the end of the year), in a report aggregating data from different patients and doctors. So, the feedback is not pertinent to a specific patient (i.e., non-selective) but applies to an average patient visited in the past (i.e. asynchronous) being not that helpful [2]. Computer applications that regularly and effortlessly track key clinical and administrative data and select the information that applies to a single patient may support real-time clinical decision-making conveying on time messages.

One of the major innovations in this field is computerized decision support systems (CDSSs) that are fully integrated with electronic health records (EHRs) and evidence-based knowledge [3]. CDSSs are information technology-based software that provides clinicians, staff, patients, or other individuals with person-specific, actionable recommendations, or management options that are intelligently filtered or presented at appropriate times to enhance health and health care [4, 5].

Current research investigates CDSSs’ potential to assist with problems raised in clinical practice, decrease the rate of medication errors, increase the adherence of clinicians to guideline- or protocol-based care, and, ultimately, to improve the overall efficiency and quality of healthcare delivery systems [619].

Few randomized controlled trials (RCTs) have evaluated the impact of CDSSs on health care [5]. Most CDSSs supported few specific clinical decisions restricted to specialized care (e.g., vital parameter monitoring and critical care in intensive care units).

The opportunity to improve patient care, by making medical knowledge readily available to physicians at the point of care, represents one of the main incentives for investing on development and evaluation of these sophisticated information systems.

Despite evidence shows that primary care helps prevent illness and death, regardless of whether the care is supplied in primary or secondary care settings, there is limited research on integrating primary and oncology care [20, 21]. Cancer patients are often old, have comorbidities, and face long diseases and treatments. They would benefit from greater supply of primary care when cared for conditions other than primary [22]. Few CDSSs commercially available are capable to support a wide range of medical conditions and support decisions such as drugs and diagnostic test ordering, encompassing oncology, other specialties and primary care. These tools can potentially support the provision of integrated healthcare services by hospital doctors who are primarily accountable for addressing specialty diseases (i.e., cancer) and developing a continuum between primary and specialty care.


Our study aims to evaluate the effectiveness of patient-specific, point-of-care reminders generated by the Medilogy Decision Support System (MediDSS) [23] on clinical practice and on the improvement of quality of care in cancer patients. We hypothesize that the intervention can increase clinician adherence to recommendations targeting decisions both in primary and secondary settings. Adherence to international guidelines, eventually, can improve the quality of care offered to cancer patients and survivors.


This protocol is reported in accordance with the SPIRIT 2013 guidance for content of clinical trial protocols [24, 25].

Trial design

The Computerized DEcision Support in ONCOlogy (ONCO-CODES) trial is a pragmatic, parallel group, randomized controlled study with 1:1 allocation ratio. The flow diagram of the study can be found in Fig. 1.

Fig. 1
figure 1

Trial flow chart

Study setting

The study will involve the medical staff of the day hospital service and of the oncologic medicine departments of the IRST Istituto Scientifico Romagnolo per lo Studio e la Cura dei Tumori IRCCS (thereafter indicated as IRST), which is a research center specialized in innovative care in the oncologic field and operational since 2007. The hospital is placed in the Emilia-Romagna region, in Italy, has 36 bed places, and is the hub of several “spoke” ambulatory services. The catchment population of the whole network is about 1,130,000 people, while the high specialization level of the hospital service attracts patients from all Northern Italy [26].

The IRST counts more than 4900 admissions per year. In addition to the in-patient wards which admit 1600 patients per year, the day hospital service performs over 3300 admissions annually.

The IRST has a medical staff of 140 doctors for a total of 500 health professionals. Since 2007, the hospital has been electronically tracking all clinical and administrative information through an EHR system based on the “Cartella clinica oncologica” platform (developed by a local software house, Log80) [27].

Eligibility criteria and recruitment

As a pragmatic clinical trial [28, 29], ONCO-CODES seeks to investigate the effectiveness of the computerized reminders in everyday clinical practice with diverse patients and varying conditions. Thus, we will enroll all patients admitted to the oncology in-patient departments and the day hospital services of the IRST during the study period, without applying any exclusion criteria.


We selected the Medilogy Decision Support System (MediDSS) after a comparative assessment of available editorial products using a predefined set of essential criteria [30, 31]. MediDSS is a product by Medilogy, an Italian developer of scientific software and medical technology. Medilogy translated and adapted Evidence-based medicine electronic decision support (EBMeDS) [32], a CDSS developed by Duodecim Medical Publications Ltd., a company owned by the Finnish Medical Society Duodecim. EBMeDS can be described as a set of rules (scripts) based on EBM guidelines and applied to structured health data. MediDSS further includes knowledge from Swedish, Finnish, INteraction X-referencing (SFINX), a drug-drug interaction database containing concise evidence-based information about the harms and benefits of about 18,000 drug interactions and adverse events [33].

MediDSS may be used as stand-alone application or may integrate structured patient data from EHR to generate patient-specific reminders, therapeutic suggestions, and diagnosis-specific links to full-text guidelines. Reminders are automatically generated and displayed on the monitors of clinicians when they open a patient’s EHR, enter a new diagnosis, prescribe a drug, or when new laboratory test results are available. Reminders were formed using international evidence-based guidelines and subsequently approved by an international panel of experts.

Our study will use international reminders (n = 262), which cover a large number of health conditions across specialties and are derived from the EBMeDS, and local reminders (n = 39), a series of tailored messages designed by the hospital staff. Moreover, the CDSS provides information about more than 10,000 drug–drug interactions or other drug-related problems, giving advice on how to handle them [34]. For instance, in a diabetic patient, a diagnosis of renal failure, or a laboratory result of creatinine increase, can trigger an alert message to reduce the drug dosage (e.g., cisplatin) based on the patient’s glomerular filtration rate (GFR).There are also several reminders that may help oncologists in the holistic care of the patient. Examples are: starting and maintaining bisphosphonate therapy when a myeloma or metastatic cancer is in the patient’s diagnosis list; alert on platelets monitoring in patients who have recently started treatment with heparins.

The reminders system will be “activated” in the EHR of patients allocated to the intervention group, while the system will be “silenced” in the EHR of those allocated to the control group. Since during care of control group patients, the generated reminders will not be shown to the physicians, control is the standard clinical practice without reminders in use. However, the best evidence for usual care will be available to the physicians at all times during the trial, if the doctor wishes to actively search for clinical practice guidelines or other information sources. All participating physicians will be informed on the availability and use of the MediDSS system.

Stepped wedge implementation

The intervention is a new technology: its integration in the current hospital system requires the configuration and customization of the software. To allow security controls and successful implementation, the CDSS will be sequentially rolled out to participants over a number of time periods. We anticipate that the number of periods will be limited (i.e., two or three periods). Over an initial period, all participants will receive the intervention. The order in which participants will receive the intervention is not determined at random, but will be determined by selecting physicians prone to provide constructive feedback to the implementation team. The RCT adopts a stepped wedge implementation of the intervention, but not a stepped wedge design [35]. Sequential roll-out of the intervention will not be considered a pilot phase of the trial, but a part of the whole RCT.

Selection and development of local reminders

In order to encourage the participation of the hospital staff within the study, we invited hospital representatives to assess the priority needs of the hospital wards. Following consultation, an IRST-specific set of reminders has been implemented.

Due to the increasing costs for the national health system of certain kinds of drugs, i.e., high-cost monoclonal antibodies, the Italian Medicines Agency developed managed entry agreements based on a risk-sharing policy. According to it, innovative oncology drugs are the subjects of risk-sharing schemes [36]. The agreements require manufacturers to pay-back part of the price of the drug for each patient who fails to respond to the new treatment. Hospitals track patient eligibility and monitor the use of drugs included in the financial scheme and their outcomes. Evidence collection of treatment completion or interruption owing to progression or toxicity has to follow rigid scheme and timeline. The hospital then applies to the national health system for pay-back. Late request for risk sharing or inaccurate monitoring of progression lead to partial losses in the reimbursement to the hospital, with negative financial impact. To support efficient documentation of treatment failures, a set of reminders was developed to optimize the timing of clinical-instrumental evaluations in accordance with the procedures provided by the risk-sharing schedule.

The local hospital expert group identified the following drugs as of special interest: abiraterone, afatinib, aflibercept, axitinib, azacitidine, bevacizumab, bortezomib, bosutinib, cetuximab, crizotinib, dabrafenib, dasatinib, decitabine, enzalutamide, eribulin, erlotinib, everolimus, gefitinib, ipilimumab, lapatinib, lenalidomide, nilotinib, ofatumumab, paclitaxel, panitumumab, pazopanib, pemetrexed, pertuzumab, plerixafor, ponatinib, sorafenib, sunitinib, temserolimus, trabectidine, trastuzumab, vemurafenib, vinflunina, and vismodegib. The IRST-specific set of reminders was developed taking into consideration the conditions that have to be met for drug full reimbursement, including the schedule for the diagnostic and laboratory analyses required from the Italian Medicines Agency to monitor the drug’s efficacy and safety [37].

In case of changes or updates in the agreements, the reminders will be modified accordingly, in order to guarantee the CDSS reminders to be always up-to-date.

An example of an algorithm is provided in Table 1.

Table 1 Algorithm for the correct prescription and the reimbursement by the national health system of a monoclonal antibody

Figures 2 and 3 show a snapshot of the activation button and of the actual reminders.

Fig. 2
figure 2

Activation button

Fig. 3
figure 3


Qualitative integration

The implementation of the intervention is a key aspect of this RCT: the results will be valid if physicians will use MediDSS in their clinical activities. Healthcare service studies on CDSSs, however, suggested that the mere provision of such technology does not guarantee its uptake. In fact, even if a CDSS is readily available within a hospital, there is a strong tendency to ignore its recommendations, not trusting the majority of its alerts [38]. Our RCT is informed by qualitative interviews aimed to detect how the CDSS fits in into clinical practice by diverse health professionals involved in oncology patient care. The interviews are a part of a larger cross-sectional study, which involves three Italian hospitals [39]. The interviews will explore barriers and facilitators that may hinder the use of a CDSS in specialty settings, including technical (e.g., poor usability or knowledge of system), individual (e.g., negative perception of CDSS or EBM, lack of motivation), group or organizational (e.g., structural or administrative constraints), and cultural factors (e.g., adverse social norms).

The qualitative assessment will inform the trial intervention, enabling possible improvements in the integrity of the intervention and associated compliance. We will collect feedbacks about usability, possible errors, or inaccuracies of the information and recommendations provided. We will offer the best possible solutions to clinicians and hospital staff to overcome these problems, modifying the CDSS software if required and possible. We cannot anticipate the types and level of changes that will be proposed to improve the technology. Physicians will volunteer for interviews, so it is possible we will likely to include disproportionately more of those with strong feelings or requests about the system.

Study outcomes

Active reminders represent potential problems in patient management: the fewer the number of active reminders persisting in a patient’s EHR, the better the health care provided. The primary endpoint is a process outcome, i.e., the rate at which the issues reported by the reminders are resolved (resolution rates). Reminders will be considered resolved when the doctor acts to remove the cause that motivated the reminder. The primary analysis will consider all reminders, irrespective of disease. The CDSS will identify updates or changes in the patient status, and related reminders, by monitoring any changes in the patient EHR.

Secondary outcomes will explore resolution rates for reminders’ subgroup types as well as clinical outcomes. They are the resolution rates of (i) the reminders targeting primary care, (ii) the reminders targeting oncology care, (iii) the reminders targeting drugs that are the subjects of risk-sharing schemes, (iv) the reminders for venous thromboembolism (VTE) prevention, (v) in-hospital morbidity for VTE-related causes, (vi) VTE-related mortality, and (vii) in-hospital all causes mortality. VTE condition is a well-known problem in oncologic care, being a preventable yet still neglected disease. Research has shown the use of CDSSs to improve the assessment of patients’ risk for VTE, facilitate appropriate administration of prophylaxis interventions, and reduce the rate of symptomatic VTE in hospitalized patients [4043]. We are interested in exploring the effectiveness of VTE reminders when they are combined together with several other types of reminders.

We will also explore any difference between the intervention and control groups regarding clinical characteristics, such as the site of tumor, physician, and inpatient/outpatient settings.

Sample size

We calculated the sample size on the basis of the primary outcome. A previous systematic review assessing the effects of computer reminders delivered to clinicians at the point-of-care on healthcare processes and outcomes found a median improvement of 4.2 % in process adherence across all reported process outcomes [44]. Accordingly, assuming resolution rates of 5% in the intervention group versus 2 % in the control group (due to a possible group contamination), we calculated that a sample of 1306 reminders will be necessary to detect the difference between the two groups (power = 0.80; α = 0.05, two-sided; 1:1 allocation). Because estimates for intracluster correlation are not available, we increased the required sample size (by 10 %) to 1436 reminders to account for clustering by patient.

Moreover, based on a prior study evaluating EBMeDS, which recorded an average of 0.30 reminders per individuals triggered at baseline [45], we determined that a total number of approximately 4800 patients (2400 per group) need to be enrolled. This figure corresponds to a recruitment period of about 12 months for the oncology departments and the day hospital services of the IRST.

Allocation and blinding

Anonymous patient identification (ID) numbers in the EHR system will be the unit of randomization. An individual external to the study group will generate the anonymous IDs using a formula based on patients’ unique fiscal code numbers.

We will randomly assign patients to either the control or experimental group with a 1:1 allocation. We will follow a computer-generated randomization schedule stratified by gender and age (0–30, 31–60, 61–80, and >80 years) using permuted blocks of random sizes [46]. Patients will be randomized immediately after the first launch of their EHR (entry of demographic data by physicians at hospital admission), and the allocation will be maintained through successive admissions.

Patients and study investigators (i.e., researchers, statisticians, information technology specialists, and hospital representatives) will be blinded to the allocation of participants. We will maintain the blinding up to the dataset disclosure. On the other hand, blinding of physicians is not feasible due to the nature of the intervention: the physician will know that a patient has been allocated to the intervention group if an automatic, patient-specific reminder is displayed on the screen.

Data collection

The data collection for this study will follow the standard data collection procedures of the IRST. We will collect demographic (i.e., gender, age) and administrative (i.e., anonymous patient ID, admission and discharge dates, diagnoses the following data from the EHR archive on a regular basis) data from the EHR archive on a daily basis. Information on reminders, including all scripts that have been activated in a patient’s record, will also be collected daily, but during the night, so as not to disturb or slow down the use of the patient EHR.

Statistical methods

For the primary outcome (i.e., resolution rates), the reminder will serve as the unit of analysis, and the patient the clustering factor. The patient will be the unit of analysis for the secondary outcomes (i.e., length of stay and in-hospital mortality). All analyses will follow the intention-to-treat principle: patients will be analyzed in the group to which they have been randomized. Descriptive statistics will be presented as means ± standard deviations (SD), medians and interquartile ranges (IQR), or percentages when appropriate. We will compare continuous variables using the Student’s t test when normally distributed, and the non-parametric two-sample Wilcoxon rank-sum (Mann-Whitney) test when they are not normally distributed. We will compare categorical variables using the chi-squared test or the Fisher’s exact test, as appropriate. To model the resolution rates of the reminders, we will run a random effects logistic regression analysis, accounting for clustering by patient [47].

For hypothesis testing, we will consider a probability level of less than 0.05 as statistically significant. All statistical tests will be two-sided. We will use the Stata software to perform all statistical analyses (Stata Corp., College Station, TX, USA).

Data monitoring

Data monitoring will inform the ONCO-CODES trial conduct, identifying the potential need for adjustments:

  1. (I)

    Sample size recalculation: Because the sample size calculation utilizes several assumptions, we will analyze the first batch of data collected and adjust the estimated sample size, if necessary, at the end of the sequential roll-out of the intervention. The 12-month recruitment period may also be adjusted, accordingly.

  2. (II)

    Interim analysis: We will perform an interim analysis on the primary endpoint after 50 % of the patients have been randomized, after 50 % of the expected events have occurred, or after 6 months of the study’s initiation (the assumed half-life of the trial), whichever occurs first. An independent statistician that is blind to the patient allocation will perform the analysis. This analysis will inform whether the intervention has been proven for efficacy (beyond reasonable doubt). We will subsequently decide whether (or not) it is necessary to modify the study or prematurely terminate it, if necessary.

  3. (III)

    End of trial: The end of trial will occur 30 days after the randomization of the last EHR. We will submit an end of trial notification and final report to the competent Ethical Committee, the IRST, and to the Sponsor.


We do not anticipate any harms (or other unintended effects) to study participants. Intervention and control groups will differ in the presence (intervention) or absence (control) of automatic reminders displayed on physicians’ monitors. Patients assigned to the control group will receive usual care without the reminders. Nevertheless, we will consult an External Advisory Board in the event that the discontinuation of the study becomes an option due to unforeseeable reasons.

Ethical and regulatory considerations

This study is conducted in accordance with the principles of the Declaration of Helsinki (October 2013) [48]. As the ONCO-CODES trial has a cluster design (several reminders, the unit of analysis, may derive from the same EHR, the unit of randomization), we followed the Ottawa statement to identify research participants and apply ethical and regulatory protections [49, 50]. The intervention (electronic CDSS reminders) does not directly target patients, but physicians who can be considered as the participants of the study. The risks associated with the participation of physicians in the ONCO-CODES trial are negligible. Physicians will be fully informed about the involvement of the IRST in the ONCO-CODES trial and trained to use the intervention. Requiring the signed consent of each physician is not feasible and will impact on the validity and generalizability of study results. Some have argued that healthcare professionals have an obligation to participate in health system or knowledge translation research [51, 52]. We consider that the waiver of signed consents will not adversely affect the rights or welfare of the research participants.

The Ethical Committee of IRST IRCCS Area Vasta Romagna approved the ONCO-CODES protocol.

Protocol amendments

Any changes to the research protocol that may impact on the study conduct (e.g., changes in study design, eligibility criteria, study outcomes, sample size, study procedures, or significant administrative aspects) will require a formal amendment of the protocol. We will communicate any such amendments to the trial registry ( and notify the health authorities in accordance with the Italian regulations. We will further seek the approval of the Ethical Committee for any amendments to the protocol.


The trial staff will ensure the maintenance of participants’ anonymity. The participants will be identified only by their initials and anonymous patient ID number. Depersonalized data will be extracted from the EHR. All documents will be stored securely and accessible only by the trial investigators and authorized personnel.

Clinical data collected during the study will only be accessible to the staff at IRST, thus complying with the current medical practice of the hospitals. The trial investigators external to the hospital (statistician, data manager, information technology personnel, etc.) will not access to any information at the patients level.

The ONCO-CODES trial will comply with the Italian Data Protection Act, which requires data to be anonymized as soon as it is practical to do so [53].

Dissemination policy

The trial results will be posted on as well as published in an open access medical journal.

We will further disseminate the study results to the health professionals of the IRST involved in the study. Datasets will be made available for research purposes upon request after the end of the study.


The results of our study will contribute to the current understanding of the effectiveness of CDSSs on cancer hospitals, thereby informing healthcare policy about the potential role of CDSS use in supporting integrated approaches between primary and specialty care. The increasing use of and familiarity with advanced technology among new generations of physicians call for pragmatic studies determining the optimal interface between primary and oncology care [54].

Another trial, named CODES trial ( NCT02577198), is actually active [55]. It implements the same CDSS of the ONCO-CODES trial. However, its focus is on general hospitals and internal medicine patients. Therefore, it prioritizes an accurate implementation of the full spectrum of international reminders, covering multiple fields of knowledge and practice, rather than focusing on the impact of local and international reminders on a specific target population, i.e., cancer patients.

Strengths and limitations

The ONCO-CODES trial has several strengths. First, in questions of effectiveness, the randomized controlled trial is the most appropriate research design. Second, we introduced a parallel qualitative study to elucidate why the CDSS does or does fit in clinical practice. The qualitative interviews will enable us to collect different attitudes and feedback, exploring the maturity of the technology. If the technology is not mature, it is likely that the results will be inconclusive and there will be no differences between the intervention and the control. It is always difficult to identify a right time to conduct a summative evaluation of a complex technology, but CDSSs seem to have more rather than less promise. So, the time is appropriate to pragmatically evaluate their effectiveness in large scale trials, accepting the risk that the technology under study has moved on by the time the trial is completed.

We preferred a pragmatic design under conditions that mimics the actual use of CDSSs in practice to increase the generalizability of the results as well as allows a more accurate estimation of the intervention’s true effectiveness. We must note the methodological limitation that physicians will not be blinded to the treatment allocation. When a patient-specific reminder is automatically displayed on the monitors, the physician will know that the particular patient belongs to the intervention group. We are aware that the unit of allocation (i.e., patient) and the lack of physician blinding can lead to possible group contamination: the doctors will be alternatively exposed to intervention and control group patients, applying the knowledge from a reminder generated for a patient allocated to the intervention group to a patient allocated to the control group. This possible learning effect (contamination of knowledge) can decrease the trial effect and lead to a more conservative effect estimate (i.e., towards the null). Randomization at the physician level, however, does not eliminate the possibility of contamination as physicians can care for patients across different wards; this level of randomization would, moreover, increase the organizational complexity of the study. A second factor that may decrease the trial effect is the adoption of an intention-to-treat analysis: not all physicians may adhere to the reminders within the study. Noncompliance is problematic because it results in a smaller difference between the intervention and comparison groups than may exists, thereby diluting the real impact of an intervention. Indeed, if the results will be positive, we will be confident that the CDSS is very likely to modify the resolution rates of the reminders, since most design factors work to dilute the effects.


This is a pragmatic trial that is powered to find small (but important) effects of a CDSS on process outcomes. While a trial to assess true clinical endpoints would be ideal, the size and timeframe of such a trial would challenge its feasibility. Positive results in terms of improved adherence to reminders indicates that the CDSS intervention tested in this trial should help specialty physicians to close the gap between their intended and actual behaviors with respect to the primary and secondary care they provide for their patients. If the intervention works only in one setting (e.g., specialty care), it could be further tested trying more intensive approaches to facilitate interoperable EHR accessible to patients and health professionals at multiple facilities.

Trial status

The implementation phase of the study was completed in October 2015, when the CDSS (MediDSS) was fully integrated with the hospital’s EHR (Cartella clinica oncologica). Subject recruitment and data collection began in November 2015 in the IRST hospital.



Computerized decision support system


Evidence-based medicine


Evidence-based medicine electronic decision support


Electronic health record


Glomerular filtration rate


Interquartile range


Medilogy Decision Support System


COmputerized DEcision Support in ONCOlogy


Randomized controlled trial


Standard deviation


Venous thromboembolism


  1. Moja L, Kwag KH. Point of care information services: a platform for self-directed continuing medical education for front line decision makers. Postgrad Med J. 2015;91:83–91.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Foy R, Eccles MP, Jamtvedt G, Young J, Grimshaw JM, Baker R. What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review. BMC Health Serv Res. 2005;5:50.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Moja L, Banzi R. Navigators for medicine: evolution of online point-of-care evidence-based services. Int J ClinPract. 2011;65:6–11.

    CAS  Google Scholar 

  4. Osheroff J. Improving medication use and outcomes with clinical decision support: a step-by-step guide. The Healthcare Information and Management Systems Society: Chicago, IL; 2009.

    Google Scholar 

  5. Berner E. Clinical decision support systems: state of the Art. AHRQ Publication No. 09-0069-EF. Agency for Healthcare Research and Quality: Rockville, MD; 2009.

    Google Scholar 

  6. Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, Samsa G, Hasselblad V, Williams JW, Musty MD, Wing L, Kendrick AS, Sanders GD, Lobach D. Effect of clinical decision-support systems: a systematic review. Ann Intern Med. 2012;157:29–43.

    Article  PubMed  Google Scholar 

  7. Ash JS, McCormack JL, Sittig DF, Wright A, McMullen C, Bates DW. Standard practices for computerized clinical decision support in community hospitals: a national survey. J Am Med Inform Assoc. 2012;19:980–7.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Griffey RT, Lo HG, Burdick E, Keohane C, Bates DW. Guided medication dosing for elderly emergency patients using real-time, computerized decision support. J Am Med Inform Assoc. 2012;19:86–93.

    Article  PubMed  Google Scholar 

  9. Romano MJ, Stafford RS. Electronic health records and clinical decision support systems: impact on national ambulatory care quality. Arch Intern Med. 2011;171:897–903.

    PubMed  PubMed Central  Google Scholar 

  10. Terrell KM, Perkins AJ, Dexter PR, Hui SL, Callahan CM, Miller DK. Computerized decision support to reduce potentially inappropriate prescribing to older emergency department patients: a randomized, controlled trial. J Am Geriatr Soc. 2009;57:1388–94.

    Article  PubMed  Google Scholar 

  11. Holbrook A, Thabane L, Keshavjee K, Dolovich L, Bernstein B, Chan D, Troyan S, Foster G, Gerstein H, COMPETE II Investigators. Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial. CMAJ. 2009;181:37–44.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Bonnabry P, Despont-Gros C, Grauser D, Casez P, Despond M, Pugin D, Rivara-Mangeat C, Koch M, Vial M, Iten A, Lovis C. A risk analysis method to evaluate the impact of a computerized provider order entry system on patient safety. J Am Med Inform Assoc. 2008;15:453–60.

    Article  PubMed  PubMed Central  Google Scholar 

  13. de Lusignan S, Chan T. The development of primary care information technology in the United kingdom. J Ambul Care Manage. 2008;31:201–10.

    Article  PubMed  Google Scholar 

  14. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, Morton SC, Shekelle PG. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144:742–52.

    Article  PubMed  Google Scholar 

  15. Feldstein AC, Smith DH, Perrin N, Yang X, Rix M, Raebel MA, Magid DJ, Simon SR, Soumerai SB. Improved therapeutic monitoring with several interventions: a randomized trial. Arch Intern Med. 2006;166:1848–54.

    Article  PubMed  Google Scholar 

  16. Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Sperling M, Cook EF, Orav EJ, Fairchild DG, Bates DW. A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. J Am Med Inform Assoc. 2005;12:431–7.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003;163:1409–16.

    Article  PubMed  Google Scholar 

  18. Tamblyn R, Huang A, Perreault R, Jacques A, Roy D, Hanley J, McLeod P, Laprise R. The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. CMAJ. 2003;169:549–56.

    PubMed  PubMed Central  Google Scholar 

  19. Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ. A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med. 2001;345:965–70.

    Article  CAS  PubMed  Google Scholar 

  20. Shi L, Macinko J, Starfield B, Politzer R, Wulu J, Xu J. Primary care, social inequalities, and all-cause, heart disease, and cancer mortality in U.S. counties, 1990. Am J Public Health. 2005a;95:674–80.

  21. Engstrom S, Foldevi M, Borgquist L. Is general practice effective? A systematic literature review. Scand J Prim Health Care. 2001;19:131–44.

    Article  CAS  PubMed  Google Scholar 

  22. Klabunde CN, Ambs A, Keating NL, He Y, Doucette WR, Tisnado D, Clauser S, Kahn KL. The role of primary care physicians in cancer care. J Gen Intern Med. 2009;24(9):1029–36.

    Article  PubMed  PubMed Central  Google Scholar 

  23. MedilogyS.r.l., Medilogy Decision Support System (MediDSS). Accessed 10 Nov 2016.

  24. Chan AW, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krleza-Jeric K, Hrobjartsson A, Mann H, Dickersin K, Berlin J, Dore C, Parulekar W, Summerskill W, Groves T, Schulz K, Sox H, Rockhold FW, Rennie D, Moher D. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013;158:200–7.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Chan AW, Tetzlaff JM, Gøtzsche PC, Altman DG, Mann H, Berlin JA, Dickersin K, Hrobjartsson A, Schulz KF, Parulekar WR, Krleza-Jeric K, Laupacis A, Moher D. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ. 2013;346:e7586.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Istituto Scientifico Romagnolo per lo Studio e la Cura dei Tumori, Homepage. Accessed 10 Nov 2016.

  27. Log80, Cartella clinica oncologica. Accessed 10 Nov 2016.

  28. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20:637–48.

    Article  CAS  PubMed  Google Scholar 

  29. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, Oxman AD, Moher D, CONSORT group. Pragmatic Trials in Healthcare (Practihc) group. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:a2390.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Banzi R, Cinquini M, Liberati A, Moschetti I, Pecoraro V, Tagliabue L, Moja L. Speed of updating online evidence based point of care summaries: prospective cohort analysis. BMJ. 2011;343:d5856.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Banzi R, Liberati A, Moschetti I, Tagliabue L, Moja L. A review of online evidence-based practice point-of-care information summary providers. J Med Internet Res. 2010;12:e26.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Duodecim Medical Publications, Evidence-based medicine electronic decision support (EBMeDS). Accessed 10 Nov 2016.

  33. Böttiger Y, Laine K, Andersson ML, Korhonen T, Molin B, Ovesjö ML, Tirkkonen T, Rane A, Gustafsson LL, Eiermann B. SFINX-a drug-drug interaction database designed for clinical decision support systems. Eur J ClinPharmacol. 2009;65:627–33.

    Article  Google Scholar 

  34. Andersson ML, Böttiger Y, Bastholm-Rahmner P, Ovesjö ML, Vég A, Eiermann B. Evaluation of usage patterns and user perception of the drug–drug interaction database SFINX. Int J Med Inform. 2015;84(5):327–33.

    Article  PubMed  Google Scholar 

  35. Ageno W, Squizzato A, Ambrosini F, Dentali F, Marchesi C, Mera V, Steidl L, Venco A. Thrombosis prophylaxis in medical patients: a retrospective review of clinical practice patterns. Haematologica. 2002;87:746–50.

    PubMed  Google Scholar 

  36. Trojniak MP, Palozzo AC. New drugs—how much are they worth: the Italian experience. EJOP. 2013;7(2):17–25.

    Google Scholar 

  37. AIFA – Agenzia Italiana del Farmaco, Registri e piani terapeutici. Accessed 10 Nov 2016.

  38. Moxey A, Robertson J, Newby D, Hains I, Williamson M, Pearson SA. Computerized clinical decision support for prescribing: provision does not guarantee uptake. J Am Med Inform Assoc. 2010;17:25–33.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Moja L, Liberati E, Galuppo L, Gorli M, Maraldi M, Nanni O, Rigon G, Ruggieri P, Ruggiero F, Scaratti G, Vaona A, Kwag K. Barriers and facilitators to the uptake of computerized clinical decision support systems in specialty hospitals: protocol for a qualitative cross-sectional study. Implement Sci. 2014;9:105.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Baroletti S, Munz K, Sonis J, Fanikos J, Fiumara K, Paterno M, Goldhaber SZ. Electronic alerts for hospitalized high-VTE risk patients not receiving prophylaxis: a cohort study. J Thromb Thrombolysis. 2008;25:146–50.

    Article  PubMed  Google Scholar 

  41. Beeler PE, Kucher N, Blaser J. Sustained impact of electronic alerts on rate of prophylaxis against venous thromboembolism. Thromb Haemost. 2011;106:734–8.

    Article  CAS  PubMed  Google Scholar 

  42. Durieux P, Nizard R, Ravaud P, Mounier N, Lepage E. A clinical decision support system for prevention of venous thromboembolism: effect on physician behavior. JAMA. 2000;283:2816–21.

    Article  CAS  PubMed  Google Scholar 

  43. Kucher N, Koo S, Quiroz R, Cooper JM, Paterno MD, Soukonnikov B, Goldhaber SZ. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med. 2005;352:969–77.

    Article  CAS  PubMed  Google Scholar 

  44. Shojania KG, Jennings A, Mayhew A, Ramsay CR, Eccles MP, Grimshaw J. The effects of on-screen, point of care computer reminders on processes and outcomes of care. Cochrane Database of Systematic Reviews 2009, Issue 3. Art. No.: CD001096.

  45. Kortteisto T, Raitanen J, Komulainen J, Kunnamo I, Makela M, Rissanen P, Kaila M, EBMeDS (Evidence-Based Medicine electronic Decision Support) Study Group. Patient-specific computer-based decision support in primary healthcare—a randomized trial. Implement Sci. 2014;9:15.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Lachin JM, Matts JP, Wei LJ. Randomization in clinical trials: Conclusions and recommendations. Controlled Clinical Trials. 1998;9(4):365–74.

  47. Twisk JWR. Applied longitudinal data analysis for epidemiology: a practical guide. 2nd ed. Cambridge: Cambridge University Press; 2013.

    Book  Google Scholar 

  48. World Medical Association Declaration of Helsinki - Ethical Principles for Medical Research Involving Human Subjects Accessed 10 Nov 2016.

  49. Taljaard M, Weijer C, Grimshaw JM, Eccles MP, Ottawa Ethics of Cluster Randomised Trials Consensus Group. The Ottawa Statement on the ethical design and conduct of cluster randomised trials: precis for researchers and research ethics committees. BMJ. 2013;346:f2838.

    Article  PubMed  Google Scholar 

  50. Weijer C, Grimshaw JM, Eccles MP, McRae AD, White A, Brehaut JC, Taljaard M, Ottawa ethics of cluster randomized trials consensus group. The Ottawa statement on the ethical design and conduct of cluster randomized trials. PLoS Med. 2012;9:e1001346.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Hutton JL. Are distinctive ethical principle required for cluster randomized trials? Stat Med. 2001;20:473–88.

    Article  CAS  PubMed  Google Scholar 

  52. Winkens RA, Knottnerus JA, Kester AD, Grol RP, Pop P. Fitting a routine health-care activity into a randomized trial: an experiment possible without informed consent? J Clin Epidemiol. 1997;50:435–9.

    Article  CAS  PubMed  Google Scholar 

  53. Autorità italiana di garanzia sulla privacy, Legislazione italiana sulla privacy. Accessed 10 Nov 2016.

  54. Aiello Bowles EJ, Tuzzio L, Wiese CJ, Kirlin B, Greene SM, Clauser SB. etal. Understanding high-quality cancer care: a summary of expert perspectives. Cancer. 2008;112(4):934–42.

    Article  PubMed  Google Scholar 

  55. Moja L, Polo Friz H, Capobussi M, Kwag KH, Banzi R, Ruggiero F, González-Lorenzo M, Liberati EG, Mangia M, Nyberg P, KunnamoI, Cimminiello C, Vighi G, Grimshaw J, Bonovas S. Implementing an evidence-based computerized decision support system to improve patient care in a general hospital: the CODES study protocol for a randomized controlled trial. Implement Sci. 2016;11:89.

Download references


We want to thank Tiina Kortteisto, Tuomas Koskela, Minna Kaila, and Marjukka Mäkelä for their comments on the research project.

Availability of data and materials

Data sets will be made available for research purposes upon request after the end of the study.

Authors’ contributions

LM is the principal investigator of the COmputerized DEcision Support system in ONCOlogy (ONCO-CODES) trial. All authors contributed to the conception and design of the study, and/or to the implementation of the intervention, providing important intellectual contents. RB, SB, MC, KK, and LM drafted the manuscript. All authors contributed to the editing and final approval of the protocol.

Competing interests

The University of Milan provides evidence services for clinical knowledge resources and publishers. Ilkka Kunnamo is the founder and leader of the EBMeDS system for CDSS, published by the Finnish Medical Society Duodecim. Peter Nyberg is a full-time employee of Duodecim. Massimo Mangia is the Chief Executive Officer of Medilogy.

Consent for publication

Not applicable.

Ethics approval and consent to participate

The Ethical Committee of IRST IRCCS Area Vasta Romagna approved the ONCO-CODES protocol: (file: IRST- aut.-IRST 100.23 ONCO CODES, approved on June 11th, 2015).

Physicians are the participants of the study and are fully informed about the involvement of the IRST in the ONCO-CODES trial and trained to use the intervention. Individual signed consent is not feasible and could impact on the validity and generalizability of the results. The Ethical Committee considered that the waiver of signed consents does not adversely affect the rights or welfare of the research participants.

Sources of financial, material, and other support

The EBMeDS system is developed by Duodecim Medical Publications Ltd. [32], a Finnish company owned by the Finnish Medical Society Duodecim. It is distributed in Italy by Medilogy S.r.l. [23]. Neither Duodecim Medical Publications Ltd nor Medilogy S.r.l. finance the ONCO-CODES trial.

The research project is financially supported by the Italian Ministry of Health (GR-2009-1606736) and by Regione Lombardia (D.R.L. IX/4340 26/10/2012). The funding sources have no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Lorenzo Moja.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Moja, L., Passardi, A., Capobussi, M. et al. Implementing an evidence-based computerized decision support system linked to electronic health records to improve care for cancer patients: the ONCO-CODES study protocol for a randomized controlled trial. Implementation Sci 11, 153 (2016).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: