This article has Open Peer Review reports available.
Evaluating the uptake and effects of the computerized decision support system NHGDoc on quality of primary care: protocol for a large-scale cluster randomized controlled trial
© Lugtenberg et al.; licensee BioMed Central Ltd. 2014
Received: 15 July 2014
Accepted: 19 September 2014
Published: 17 October 2014
Computerized decision support systems (CDSSs) are increasingly used to improve quality of care. There is evidence for moderate to large effects from randomized controlled trials (RCTs), but evidence on their effectiveness when implemented at a national level is lacking. In the Netherlands, the Dutch College of General Practitioners (NHG) initiated their successful guideline program already 30 years ago. NHGDoc, a CDSS based on these NHG guidelines, covering multiple disease areas for general practice, was developed in 2006 with the aim to improve quality of primary care. In this paper, a protocol is presented to evaluate the uptake and effects of NHGDoc.
A cluster RCT will be conducted among 120 general practices in the Netherlands. Eligible general practices will be randomized to receive either the regular NHGDoc decision support modules (control arm) or the regular modules plus an additional module on heart failure (intervention arm). The heart failure module consists of patient-specific alerts concerning the treatment of patients with heart failure. The effect evaluation will focus on performance indicators (e.g., prescription behavior) as well as on patient outcomes (e.g., hospital admissions) relevant in the domain of heart failure. Additionally, a process evaluation will be conducted to gain insight into the barriers and facilitators that affect the uptake and impact of NHGDoc.
Results of this study will provide insight in the uptake and impact of a multiple-domain covering CDSS for primary care implemented by a national guideline organization to improve the quality of primary care. Whereas the trial focuses on a specific domain of care-heart failure-conclusions of this study will shed light on the functioning of CDSSs covering multiple disease areas for primary care, particularly as this study also explores the factors contributing to the system's uptake and effectiveness.
Clinical trials NCT01773057
Clinical practice guidelines aim to improve quality of care, but their implementation in practice remains a challenge. Despite considerable efforts in improving guideline implementation, several reviews have shown that guidelines are only moderately effective in changing clinical practice ,. A study in the US showed that only about half of the patients received care as recommended in the guidelines . Similarly, in other countries, adherence to the guidelines among physicians is found to be suboptimal as well ,.
Computerized decision support systems (CDSSs) are one of the tools that can be used to improve the uptake of guidelines in practice. By linking characteristics of individual patients to a computerized medical knowledge base, they can provide patient-specific recommendations to healthcare providers during patient care . To the extent that CDSSs are guideline-driven, i.e., if the content is directly derived from clinical practice guidelines, they have the potential to increase physicians’ adherence to guidelines and ultimately lead to improved patient outcomes.
Whereas CDSSs are increasingly being used in various healthcare settings, evidence is mainly retrieved from small-scale academic-driven experiments among selected groups of innovators. Definitive evidence on their effectiveness in large-scale practice-driven use of CDSSs remains to be established. Some studies have shown that CDSSs can improve medical practice, e.g., -. In a recent series of six systematic reviews , it was found that CDSSs improved the process of medical care in 52%-64% of studies across all six reviews. However, only 15%-31% of those reviews that evaluated patient outcomes showed a positive impact on patients’ health -.
Moreover, CDSSs seem to have added value as a tool for improving quality of care when focusing on specific behaviors (e.g., drug dosing) , within well-defined areas of care. A large share of medical care, however, is delivered by primary care practitioners (PCPs), particularly in the Netherlands . PCPs work in generic settings and are confronted with a variety of diseases. This setting necessitates CDSSs covering multiple disease areas. Thus far, little is known on the effectiveness of CDSSs in settings in which PCPs are exposed to various alerts within multiple domains of care.
In the Netherlands, the Dutch College of GPs (NHG) initiated their successful guideline program already 30 years ago . In 2006, NHGDoc, a CDSS based on these NHG guidelines and covering multiple disease areas for general practice, was developed as a collaborative effort between the Dutch College of GPs (NHG)  and ExpertDoc , a private enterprise. The content of NHGDoc is directly derived from the NHG guidelines-the national prevailing guidelines for general practice. NHGDoc is gradually being implemented at a national level and is currently-at the onset of the trial-integrated into two out of eight electronic health record systems (EHRSs) used in Dutch general practice, covering about 25% of all general practices in the Netherlands.
To assess the effects of the CDSS NHGDoc on relevant performance indicators for the process of medical care
To assess the effects of the CDSS NHGDoc on patient outcomes
To gain insight into the determinants that affect the uptake and impact of NHGDoc
A two arm-cluster (before and after) RCT will be conducted with a follow-up period of 12 months. Randomization will take place at a practice level, i.e., general practices will be randomized to the control arm or the intervention arm rather than individual PCPs. This will be the preferred unit of allocation as the intervention (i.e., the NHGDoc module on heart failure) can only be implemented at the level of the practices. Moreover, this is done to avoid possible contamination between PCPs working within the same practice yet being allocated to different study groups .
The study will be conducted within the Dutch primary care setting. In the Netherlands, there are approximately 5,000 general practices in which nearly 11,000 GPs are delivering care . Almost all GPs are members of the Dutch College of General Practitioners (NHG). Since the late 1980s, the NHG has produced over 100 national clinical practice guidelines for general practice . These NHG guidelines cover the vast majority of conditions and diseases frequently seen in general practice. Currently, eight of these guidelines have been integrated into NHGDoc, i.e., cardiovascular risk management, asthma/COPD, diabetes mellitus type II, thyroid disorders, viral hepatitis and other liver diseases, atrial fibrillation, and subfertility.
Additional to GPs, approximately 3,000 practice nurses (PNs) are delivering care in approximately 60% of the Dutch general practices. They are mainly responsible for regular checks of the chronically ill such as cardiovascular and asthma/COPD patients .
NHGDoc was developed in 2006 as a collaborative effort between the NHG  and ExpertDoc . As the NHG is the owner of the system and is an organization highly appreciated among GPs, it was decided to introduce the name `NHGDoc’ to the CDSS. NHGDoc is at the time of recruiting practices for the study integrated into two (MicroHIS X and Promedico-ASP) out of eight EHRSs used in Dutch general practice. These two systems cover about 25% of all Dutch general practices.
All general practices in the Netherlands that use either the EHRS MicroHIS X or Promedico-ASP and thus have NHGDoc at their disposal, will be invited to participate in the study (n = approximately 1,100). To invite general practices to participate in the study, a comprehensive recruitment plan will be executed. All general practices will be approached by ordinary post mail signed by the NHG as well as an email signed by the EHRS providers (MicroHIS X and Promedico-ASP) and their user associations (Orego and Atlas). Additionally, announcements about the evaluation study will be placed in several relevant Dutch journals, websites, newsletters, and through social media (i.e., Facebook, Twitter, and LinkedIn). In all messages, reference will be made to the website of the evaluation study (www.nhgdoc-evaluatie.nl) which presents information about the study as well as an online registration form for general practices. To enable blinding of participants and contrast between groups, the study is purposefully presented as a before-after study without a concurrent control group, rather than as a before-after RCT. Therefore, the participants will not be aware of the fact that they will be allocated at random to two different conditions.
In order to participate in the study, each general practice should fill out the online registration form. The applicant is asked to consent on behalf of all practice staff (GPs and PNs) that could potentially use NHGDoc. This is necessary as the intervention and data collection can only be implemented at the level of the practices. We will, however, be able to distinguish between different users in analyzing the study results.
Intervention and comparisons
Control arm: regular NHGDoc decision support
General practices assigned to the control arm receive the regular NHGDoc decision support. These practices receive decision support with respect to all modules (NHG guidelines) that-at the onset of the trial-have been integrated into NHGDoc (see `Description of NHGDoc and its regular modules (control arm)’ section).
Description of NHGDoc and its regular modules (control arm)
NHGDoc is a CDSS integrated within the EHRS and based on the NHG guidelines, the prevailing guidelines for general practice in the Netherlands. It provides GPs and PNs evidence-based and patient-specific advices during consultation in terms of patient data registration, drug prescription, and management.
At the time of the onset of the trial, NHGDoc covered the following NHGDoc modules: cardiovascular risk management, asthma/COPD, diabetes mellitus type II, thyroid disorders, viral hepatitis and other liver diseases, atrial fibrillation, and subfertility. For each NHG guideline, key recommendations have been selected based on relevance of disease burden, room for improvement, and possibility to translate or normalize the recommendation into if-then rules. This selection of key recommendations is approved by representative experts of the guideline committees. Subsequently, the selected key recommendations are digitized into NHGDoc.
When the GP or PN opens a patient file in the EHRS, anonymous patient and performance data are sent to the NHGDoc server. The patient and performance data are compared to the digitized guideline recommendations and in the case of a discrepancy between current and advised care, an alert will be sent back to the GP or PN.
Intervention arm: regular modules plus additional module on heart failure
General practices allocated to the intervention arm will receive the same decision support modules as the control arm (see `Description of NHGDoc and its regular modules (control arm)’ section), extended with the NHGDoc module on heart failure, which will be activated at the onset of the trial (see `Basic elements/key recommendations of the NHGDoc module on heart failure (intervention arm)’ section). The reason we chose heart failure to be the subject of the trial was first of all of pragmatic nature; heart failure was one of the few modules that were to be implemented into NHGDoc in the year of the onset of the trial. In addition, we chose heart failure as it is a relatively common condition for which relatively large improvements (in terms of increasing or decreasing mortality and morbidity) are to be expected if healthcare providers adhere to the treatment guidelines, compared to other conditions -.
Basic elements/key recommendations of the NHGDoc module on heart failure (intervention arm)
- 1.Alerts on heart failure in terms of registering the following patient data:
clearance of creatinine;
blood pressure (systolic and diastolic).
- 2.Alerts on heart failure in terms of prescribing (or adjusting the dose of) the following drugs:
For example: Stop prescribing ACE-inhibitor because of the reduced clearance of creatinine and the increased serum potassium and check these values.
- 3.Alerts on heart failure in terms of (paying attention to) the following management aspects such as:
stop prescribing medication such as calcium antagonist, thiazolidinedione, disopyramide or a combination of ACE-inhibitor and AII antagonists;
consider adjusting the dosage of medication when hypertension is increasing;
consider use of NSAID;
consider weight policy, e.g., discuss weight reduction and consider increasing the dosage of diureticum if weight increases with two kilogram or more within a short period of time;
consider temporary restriction of fluids;
advice to give up smoking;
consider sleeping research and;
consider referral to a cardiologist or nephrologist.
To select outcome measures for our evaluation study, we first identified previously validated indicators for heart failure . Subsequently, we consulted an expert panel of general practitioners as well as specialists in the area of heart failure to select the outcome measures that are clinically most relevant for our study and that are suitable for quantitative evaluation. Outcomes will be measured during the year before the start of the intervention (baseline) and in the first year following the start of the intervention (follow-up). This resulted in the following primary and secondary outcome measures:
Primary outcome measures (per practice, per year):
➢ Prescribing of ACE-inhibitors/angiotensin II
Percentage of consultations in which a patient with heart failure was prescribed ACE-inhibitors/angiotensin II.
➢ Prescribing of beta-blockers
Percentage of consultations in which a patient with heart failure was prescribed beta-blockers.
➢ Prescribing of diuretics
Percentage of consultations in which a patient with heart failure was prescribed diuretics.
Secondary outcome measures:
➢ Use of the system NHGDoc
Number of requests sent and alerts received for all patients; number of requests sent and alerts received for heart failure patients; percentage of opened alerts compared to the number received alerts; percentage of opened alerts compared to the number received alerts for heart failure patients; mean delivery time for an alert; number of practices that switched any module off; for each module, the number/percentage of practices with at least one user who switched the module off; and number of practices with at least one user who switched the module heart failure off (intervention group only).
➢ Patient data registration
Percentage of consultations in which for a patient with heart failure, relevant patient data have been registered: clearance of creatinine (<12 mnd); (serum) creatinine (<12 mnd); (serum) potassium (<12 mnd); (serum) sodium (<12 mnd); patient weight (<6 mnd); heart (frequency) (<6 mnd); blood pressure (systolic and diastolic) (<6 mnd).
➢ Hospital admissions
Number of hospital admissions of patients with heart failure.
➢ All cause mortality
Number of patients with heart failure that died in the hospital.
The minimum required number of general practices was based on the two main primary outcome measures: prescribing of ACE-inhibitors/angiotensin II and prescribing of beta-blockers. The anticipated improvement was based on literature , as well as on consultation of experts in the field (ACE-inhibitor change from 63%-69%; beta-blockers from 59%-65%). As for beta-blockers, to detect this difference, we will need at least 122 general practices completing the trial with 80% power at a type I error risk (α) of 5%, taking an estimated intra-cluster correlation of 0.1 into account . With this number of practices, the study will also be able to detect the difference for prescribing of ACE-inhibitors (minimum number of practices =116). We do not, however, expect to be able to detect an effect on hospital admissions and hospital mortality with this number of practices.
General practices and the staff working within these practices will be blinded to group allocation. They will not be informed about being allocated to either the intervention group or the control group (see `Participants’ section). They are even unaware of participating in an intervention trial with a concurrent control group. However, as the intervention arm will, at the onset of the trial, receive decision support on heart failure (directly available to all intervention practices at one preset moment in time) without the usual communication about the implementation of a new module, these practices may suspect heart failure to be the subject/topic of the trial. Therefore, the extent to which the blinding was really successful will be checked in a survey, which is part of the process evaluation of the trial.
Patients will also be blinded to group allocation as they are not aware and are not likely to become aware of the trial (design). Given the fact that patients are not being exposed to experimental interventions and that all patient data used in our evaluation study are anonymized, we are not obliged to inform patients about the use of these data, nor do we have to offer them an opt-out option. And finally, researchers will also be blinded to the group allocation as the randomization variables will be labeled with the letters A and B (rather than I and C) for the control and intervention group by a third independent researcher.
To investigate the effectiveness of NHGDoc, data will be collected from two sources: the NHGDoc server and Dutch Hospital Data.
For the largest part of this study, existing data collection methods from the NHGDoc alert service that have been developed by ExpertDoc to generate CDSS alerts for NHGDoc will be used. These are all automated data extractions from electronic patient records available in the EHRS. The NHGDoc alert service only uses anonymized patient data, and data are only streamed and not stored. For this evaluation study, however, it is necessary to save the anonymous data for the follow-up period of a year. Therefore, ExpertDoc, the company that manages the data, will send the NHGDoc data on a monthly basis, secured with a log in code, to IQ healthcare. NHGDoc data include all alerts, and the anonymous patient data that generate the alerts.
Dutch Hospital Data
To be able to detect the effects of NHGDoc on patient outcomes, we also use data from Dutch Hospital Data (DHD), a foundation that collects and manages data of Dutch hospitals. Based on the practice codes (practice AGB codes) of the participating practices in both arms, DHD selects relevant data with respect to hospital admissions and mortality of patients that are registered with ICD-code I50 of the ICD 10 and sends them, after anonymizing the dataset, to IQ healthcare.
To assess the effects of NHGDoc in terms of improving the quality of primary care, the scores on outcomes measures during the follow-up period will be compared between both study arms adjusted for baseline findings. To take into account potential clustering of effects within practices, multilevel models will be used.
The primary outcome measures are the percentage of consultations in which a patient with heart failure is prescribed 1. ACE-inhibitors/angiotensin II, 2. beta-blockers, and 3. diuretics. To assess the effects of NHGDoc on each of these variables, multilevel linear regression analysis will be performed. To assess the effects of NHGDoc on mortality, logistic regression analysis will be used, adjusting for severity of illness at hospital admission and comorbidity as measured by the Charlson Comorbidity Index. Both patient-level and practice-level intercept estimates will be used to account for potential correlation of measurements within patients and within general practices. We do not use the provider level estimates in our analyses as the difference between number of practices and providers will be too small due to solo practices. Moreover, patients may visit different GPs within the same practice during the year making this distinction less relevant as well.
Aside from the effect evaluation, a process evaluation will be conducted to gain insight into the barriers and facilitators that affect the system's uptake and impact . The actual exposure of the PCPs to the intervention as well as their experience with the intervention may have affected the results of the study and will be investigated. Results of the process evaluation will be used to improve (the implementation of) NHGDoc.
Data will be collected among participants in the intervention and control arm in three different ways. First, some variables measuring exposure to the intervention (e.g., not functioning of the server, deliberately turning off the heart failure module or particular areas within this module) will be collected by the NHGDoc server.
Second, three focus groups will be conducted among NHGDoc users (eight to ten GPs and PNs per group) to gain insight into their experiences with CDSSs in general and NHGDoc in particular. In each focus group, barriers and facilitators will be discussed using a topic list. Participants will be recruited by sending a direct email and reminders to all participating practices. If necessary, additional announcements will be placed on relevant websites and through social media (i.e., Facebook, Twitter, and LinkedIn).
And finally, an electronic questionnaire will be sent to all participating practices at the end of the study period to triangulate the findings of the study. Aside from quantitatively assessing the perceived barriers and facilitators to using CDSSs in general and NHGDoc in particular, attention will be paid to the exposure to and experiences with the specific heart failure module. Also, the intended `blinding’ in terms of the content of the intervention as well as the group allocation will be checked among participants.
Exposure to and experiences with NHGDoc will be analyzed descriptively. Data from the focus groups will be audiotaped, verbatim transcribed, and independently analyzed by two researchers using content analysis with the software program AtlasTi7.0.
In the Netherlands, studies involving human subjects need to undergo a medical ethics review if they are subject to the Medical Research Involving Human Subjects Act (WMO). The study protocol was assessed by the Medical Ethics Committee (CMO) (of district Nijmegen/Radboud) and declared that no further ethical approval was required.
Moreover, the Dutch College of General Practitioners (NHG) set up a privacy college for the use of data from NHGDoc in order to monitor whether all data related to NHGDoc are used appropriately. Therefore, the study protocol of our evaluation study was checked and approved by this college. Also, the members from the college have the option throughout the trial to check on whether the data were encrypted and delivered as agreed on in the protocol.
At the time of manuscript submission, the trial is closed to recruitment and follow-up. No data cleaning or analysis has been executed prior to the submission of this manuscript.
This paper describes the protocol of a cluster RCT, which aims to evaluate the effectiveness of a multiple-domain covering CDSS for primary care implemented by a national organization. By means of a large-scale 1-year trial, it will be investigated whether providing guideline-driven and patient-specific advices to healthcare providers during consultation leads to improved process of medical care and ultimately to improved patient outcomes. Additionally, insight will be gained into the barriers and facilitators perceived by users that may affect NHGDoc's uptake and impact.
The effectiveness of CDSSs in primary care has been evaluated in previous studies -. However, the CDSS being subject of this trial exhibits some unique features. First, NHGDoc is a guideline-driven CDSS for primary care, which is being implemented by a national organization. Whereas most CDSSs incorporate evidence-based recommendations, in a few of them, the whole content is directly derived from a set of national prevailing clinical practice guidelines developed by a national independent and respected organization. Moreover, the system NHGDoc is also owned and implemented by this independent professional organization. A CDSS such as NHGDoc that is owned and implemented by an independent professional national organization might be more effective compared to CDSSs for which this link is not established.
Another distinguishing feature of NHGDoc is its focus on the generic setting of primary care. Most CDSSs have been developed to target specific behaviors (e.g., preventive measures such as vaccinations, appropriate drug dosing) within well-defined areas of care (e.g., intensive care unit). NHGDoc is developed for a generic setting and integrates multiple alerts within multiple domains of care or disease areas. Although CDSSs have proven to be successful in focusing on specific behaviors within well-defined areas of care, very little is known , on how systems designed for and used in a generic primary care setting will function. This trial will shed some light on this issue.
Although NHGDoc is a multiple-domain covering CDSS, the design of the study does not allow us to differentiate in the effects between the various domains of care. The intervention in our trial consists solely of decision support in the area of heart failure. Nevertheless, our study does provide insight into the extent that decision support on heart failure is effective in a setting in which decision support is provided for several domains of care. Moreover, the process evaluation in our study will also provide insight in terms of the uptake and effectiveness of decision support as part of a multiple domain-covering CDSSs as well as its contributing/factors.
A limitation of our study design is that we do not have a `real’ control group, consisting of healthcare providers not using NHGDoc at all. Although the control arm does not receive decision support in the area of heart failure, it does receive decision support with respect to the regular NHGDoc modules that had already been implemented at the onset of the trial. This may suggest a selection bias with only GPs and PNs with a positive attitude towards CDSSs participating in the trial. However, both the control and intervention arm consist of general practices that have NHGDoc at their disposal because it is integrated into their EHRS, rather than that they have deliberately chosen for it. As a consequence, it is unlikely that only highly motivated GPs and PNs using NHGDoc frequently participate in our study. Moreover, if there is any bias in terms of motivation among participating GPs, we expect it to be equally distributed between the intervention and control arm, as a result of the randomization process of the trial.
This study will shed light on the uptake and effectiveness of CDSSs covering multiple disease areas in primary care, which is owned and implemented by a national guideline organization. By also exploring factors contributing to the system's uptake and effectiveness, insight will be provided in how such systems should be designed and implemented in order to function well in a generic primary care setting. Results will not only be of interest to developers and implementers of CDSSs but also to guideline implementers using CDSSs to improve the uptake of guidelines and ultimately patient outcomes in primary care.
We thank the Dutch College of GPs (NHG) for providing financial support. The funders had no role in study design, data collection and proposed analysis, preparation of the manuscript, or decision to submit the manuscript for publication.
- Grol R: Improving the quality of medical care: building bridges among professional pride, payer profit, and patient satisfaction. JAMA. 2001, 286: 2578-2585. 10.1001/jama.286.20.2578.View ArticlePubMedGoogle Scholar
- Lugtenberg M, Burgers JS, Westert GP: Effects of evidence-based clinical practice guidelines on quality of care: a systematic review. Qual Saf Health Care. 2009, 18: 385-392. 10.1136/qshc.2008.028043.View ArticlePubMedGoogle Scholar
- McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, Kerr EA: The quality of health care delivered to adults in the United States. N Engl J Med. 2003, 348: 2635-2645. 10.1056/NEJMsa022615.View ArticlePubMedGoogle Scholar
- Braspenning J, Schellevis F, Grol R: Tweede Nationale Studie naar ziekten en verrichtingen in de huisartspraktijk. Kwaliteit huisartsenzorg belicht. 2004, WOK/ NIVEL, Nijmegen/ UtrechtGoogle Scholar
- Zeymer U, James S, Berkenboom G, Mohacsi A, Iñiguez A, Coufal Z, Sartral M, Paget M-A, Norrbacka K, Ferrieres J, Bakhai A: Differences in the use of guideline-recommended therapies among 14 European countries in patients with acute coronary syndromes undergoing PCI. Eur J Prev Cardiol. 2013, 20: 218-228. 10.1177/2047487312437060.View ArticlePubMedGoogle Scholar
- Hunt DL, Haynes RB, Hanna SE, Smith K: Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998, 280: 1339-1346. 10.1001/jama.280.15.1339.View ArticlePubMedGoogle Scholar
- Johnston ME, Langton KB, Haynes RB, Mathieu A: Effects of computer-based clinical decision support systems on clinician performance and patient outcome: a critical appraisal of research. Ann Intern Med. 1994, 120: 135-142. 10.7326/0003-4819-120-2-199401150-00007.View ArticlePubMedGoogle Scholar
- Garg AX, Adhikari NKJ, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, Sam J, Haynes RB: Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005, 293: 1223-1238. 10.1001/jama.293.10.1223.View ArticlePubMedGoogle Scholar
- Kawamoto K, Houlihan CA, Balas EA, Lobach DF: Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005, 330: 765-10.1136/bmj.38398.500764.8F.View ArticlePubMedPubMed CentralGoogle Scholar
- Roshanov PS, Fernandes N, Wilczynski JM, Hemens BJ, You JJ, Handler SM, Nieuwlaat R, Souza NM, Beyene J, Spall HGCV, Garg AX, Haynes RB: Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ. 2013, 346: f657-10.1136/bmj.f657. doi:http://dx.doi.org/10.1136/bmj.f657View ArticlePubMedGoogle Scholar
- Hemens B, Holbrook A, Tonkin M, Mackay J, Weise-Kelly L, Navarro T, Wilczynski N, Brian Haynes R: Computerized clinical decision support systems for drug prescribing and management: a decision-maker-researcher partnership systematic review. Implement Sci. 2011, 6: 89-10.1186/1748-5908-6-89.View ArticlePubMedPubMed CentralGoogle Scholar
- Roshanov P, Misra S, Gerstein H, Garg A, Sebaldt R, Mackay J, Weise-Kelly L, Navarro T, Wilczynski N, Haynes RB: Computerized clinical decision support systems for chronic disease management: a decision-maker-researcher partnership systematic review. Implement Sci. 2011, 6: 92-10.1186/1748-5908-6-92.View ArticlePubMedPubMed CentralGoogle Scholar
- Roshanov P, You J, Dhaliwal J, Koff D, Mackay J, Weise-Kelly L, Navarro T, Wilczynski N, Brian Haynes R: Can computerized clinical decision support systems improve practitioners' diagnostic test ordering behavior? A decision-maker-researcher partnership systematic review. Implement Sci. 2011, 6: 88-10.1186/1748-5908-6-88.View ArticlePubMedPubMed CentralGoogle Scholar
- Nieuwlaat R, Connolly S, Mackay J, Weise-Kelly L, Navarro T, Wilczynski N, Brian Haynes R: Computerized clinical decision support systems for therapeutic drug monitoring and dosing: a decision-maker-researcher partnership systematic review. Implement Sci. 2011, 6: 90-10.1186/1748-5908-6-90.View ArticlePubMedPubMed CentralGoogle Scholar
- Sahota N, Lloyd R, Ramakrishna A, Mackay J, Prorok J, Weise-Kelly L, Navarro T, Wilczynski N, Brian Haynes R: Computerized clinical decision support systems for acute care management: a decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes. Implement Sci. 2011, 6: 91-10.1186/1748-5908-6-91.View ArticlePubMedPubMed CentralGoogle Scholar
- Souza N, Sebaldt R, Mackay J, Prorok J, Weise-Kelly L, Navarro T, Wilczynski N, Haynes RB: Computerized clinical decision support systems for primary preventive care: a decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes. Implement Sci. 2011, 6: 87-10.1186/1748-5908-6-87.View ArticlePubMedPubMed CentralGoogle Scholar
- Haynes RB, Wilczynski N: Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: methods of a decision-maker-researcher partnership systematic review. Implement Sci. 2010, 5: 12-10.1186/1748-5908-5-12.View ArticlePubMedPubMed CentralGoogle Scholar
- Schäfer W, Kroneman M, Boerma W, van den Berg M, Westert G, Devillé W, Van Ginneken E: The Netherlands: health system review. Health Systems in Transition. 2010, 12: 1-229.Google Scholar
- Nederlands Huisartsen Genootschap (NHG); nhg.artsennet.nl Accessed Sep 2014Google Scholar
- ExpertDoc; expertdoc.nl Accessed Sep 2014Google Scholar
- Kerry SM, Bland JM: Analysis of a trial randomised in clusters. BMJ. 1998, 316: 54-10.1136/bmj.316.7124.54.View ArticlePubMedPubMed CentralGoogle Scholar
- Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG: CONSORT explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010, 2010: 340-Google Scholar
- Schulz KF, Altman DG, Moher D: CONSORT statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010, 2010: 340-Google Scholar
- Campbell MK, Elbourne DR, Altman DG: CONSORT statement: extension to cluster randomised trials. BMJ. 2004, 328: 702-708. 10.1136/bmj.328.7441.702.View ArticlePubMedPubMed CentralGoogle Scholar
- Van Hassel DTP, Kenens RJ: Cijfers uit de registratie van huisartsen - peiling 2012. 2013, NIVEL, UtrechtGoogle Scholar
- van der Velden L, Hansen J, Hingstman L, Verheij R: Praktijkondersteuning huisartsen. Wetenschapsdag 15 mei 2008, Rotterdam. 2008, Nederlands Huisartsen Genootschap, Utrecht, 19-20.Google Scholar
- Wennekes L, Van Lieshout J, Mulder J: Medicatie bij hartfalen. Huisarts Wet. 2010, 53: 355-10.1007/s12445-010-0163-8.View ArticleGoogle Scholar
- Bosch M, Wensing M, Bakx JC, Van Der Weijden T, Hoes AW, Grol RPTM: Current treatment of chronic heart failure in primary care; still room for improvement. J Eval Clin Pract. 2010, 16: 644-650.PubMedGoogle Scholar
- Davies MK, Gibbs CR, Lip GYH: Management: diuretics, ACE inhibitors, and nitrates. BMJ. 2000, 320: 428-431. 10.1136/bmj.320.7232.428.View ArticlePubMedPubMed CentralGoogle Scholar
- Pitt B, Zannad F, Remme WJ, Cody R, Castaigne A, Perez A, Palensky J, Wittes J: The effect of spironolactone on morbidity and mortality in patients with severe heart failure. N Engl J Med. 1999, 341: 709-717. 10.1056/NEJM199909023411001.View ArticlePubMedGoogle Scholar
- Mant J, Al-Mohammad A, Swain S, Laramée P: Management of chronic heart failure in adults: synopsis of the national institute for health and clinical excellence guideline. Ann Intern Med. 2011, 155: 252-259. 10.7326/0003-4819-155-4-201108160-00009.View ArticlePubMedGoogle Scholar
- Hoes HW, Voors AA, Rutten FH, Van Lieshout J, Jansen PGH, Walma EP: NHG-Standaard Hartfalen (M51), tweede herziening. Huisarts Wet. 2010, 53: 368-389.Google Scholar
- Voerman G, Van Lieshout J, Maassen I, Calsbeek H, Braspenning J: Publieke indicatoren eerstelijnszorg voor patiënten met chronisch hartfalen. 2009, IQ healthcare, NijmegenGoogle Scholar
- Health Services Research Unit, University of Aberdeen / Chief Scientist Office. www.abdn.ac.uk/hsru/research/delivery/behaviour/methodological-research.Accessed July 2014
- Hulscher MEJL, Van Drenth BB, Van der Wouden JC, Mokkink HGA, Van Weel C, Grol RPTM: Changing preventive practice: a controlled trial on the effects of outreach visits to organise prevention of cardiovascular disease. Qual Health Care. 1997, 6: 19-24. 10.1136/qshc.6.1.19.View ArticlePubMedPubMed CentralGoogle Scholar
- Bindels R, Hasman A, Van Wersch JWJ, Talmon J, Winkens RAG: Evaluation of an automated test ordering and feedback system for general practitioners in daily practice. Int J Med Inform. 2004, 73: 705-712. 10.1016/j.ijmedinf.2004.06.001.View ArticlePubMedGoogle Scholar
- Martens J, van der Weijden T, Severens J, De Clercq P, De Bruijn D, Kester A, Winkens R: The effect of computer reminders on GPs' prescribing behaviour: a cluster-randomised trial. Int J Med Inform. 2007, 76: S403-S416. 10.1016/j.ijmedinf.2007.04.005.View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.