This article has Open Peer Review reports available.
A multifaceted feedback strategy alone does not improve the adherence to organizational guideline-based standards: a cluster randomized trial in intensive care
© de Vos et al. 2015
Received: 27 November 2014
Accepted: 30 June 2015
Published: 8 July 2015
Organizational data such as bed occupancy rate and nurse-to-patient ratio are related to clinical outcomes and to the efficient use of intensive care unit (ICU) resources. Standards for these performance indicators are provided in guidelines. We studied the effects of a multifaceted feedback strategy to improve the adherence to these standards.
In a cluster randomized controlled study design the intervention ICUs received extensive monthly feedback reports, they received outreach visits and initiated a quality improvement team. The control ICUs received limited quarterly feedback reports only. We collected primary data prospectively within the setting of a Dutch national ICU registry over a 14-month study period. The target indicators were bed occupancy rate (aiming at 80 % or below) and nurse-to-patient ratio (aiming at 0.5 or higher). Data were collected per 8-h nursing shift. Logistic regression analysis was performed. For both study end points, the odds ratios (OR) for improvements at follow-up versus at baseline were calculated separately for control and intervention ICUs.
We analyzed data on 67,237 nursing shifts. The bed occupancy rate did not improve in the intervention group compared to baseline (adjusted OR 0.88; 95 % confidence interval (CI), 0.62–1.27) or compared to control group (OR 0.67; 95 % CI 0.39–1.15). The nurse-to-patient ratio did not improve (OR 0.72; 95 % CI 0.41–1.26 compared to baseline and OR 0.65; 95 % CI 0.35–1.19 compared to control group).
A multifaceted feedback intervention did not improve the adherence to guideline-based standards on the organizational issues bed occupancy rate and nurse-to-patient ratio in the ICU. The reasons may be a limited confidence in data quality, the lack of practical tools for improvement, and the relatively short follow-up.
KeywordsMultifaceted feedback Critical care Quality improvement Bed occupancy Nurse-to-patient ratio
Organizational aspects are associated with clinical outcomes in the intensive care setting. A low nurse-to-patient ratio is associated with unfavorable outcomes [1–5]. In addition, several studies show that a high bed occupancy rate is associated with an increased mortality and also with an increased refusal rate for new appropriately referred patients [6–8]. The bed occupancy rate and the nurse-to-patient ratio are not only relevant clinical issues but also important management tools as they are related to efficiency and costs of intensive care departments . Based on these arguments, national and international societies published guidelines that include recommendations on bed occupancy rates and nurse-to patient ratios [10, 11]. The adherence to these guidelines intends to improve the efficiency and efficacy of intensive care treatment. It is, however, well known that the adherence to guidelines in clinical practice is, in general, limited and improvement in the adherence to guidelines is hard to achieve. Several factors that limit guideline adherence can be distinguished . Performance feedback is a well-established tool to facilitate guideline implementation and to improve the adherence to guideline recommendations. It was shown that feedback reports combined with other strategies as part of a multifaceted intervention were more successful than feedback reports alone [13, 14]. In addition, performance feedback was especially effective in improving the clinical processes of care , but data on organizational aspects is lacking. In the intensive care setting, several studies addressed the impact of multifaceted feedback on clinical processes [15–17]. However, no studies concerning multifaceted feedback for organizational standards in the critical care setting are available. Based on these arguments, we conducted a cluster randomized controlled trial (RCT) among Dutch intensive care units (ICUs) to study the effect of a multifaceted feedback intervention on the adherence levels to guideline-based organizational standards, in particular bed occupancy rates and nurse-to-patient ratios.
Context: organizational standards for intensive care in The Netherlands
All Dutch ICUs are closed-format and the majority has an intensivist on call 24 h a day . In 2006, the Dutch Society for Intensive Care (NVIC) developed an evidence-based guideline. This guideline focused on the organizational aspects of intensive care and incorporated several standards . One of these standards is that the bed occupancy rate during a shift should not exceed 80 %. Another standard is that within a shift, no more than two patients should be assigned to one ICU nurse (i.e., a nurse-to-patient ratio of at least 0.5). Both standards were based on evidence that showed crossing these thresholds leads to unfavorable patient outcomes measured as mortality or complication rates.
To ensure that the guideline’s target users, the intensivists, agreed with the standards, the guideline was formally endorsed in a general assembly of the society members. In addition, subsequent to guideline publication, two performance indicators were introduced to monitor adherence to the two standards, one related to bed occupancy and one regarding nurse-to-patient ratio . These indicators were part of a set of 11 structure, process, and outcome indicators . Data of these indicators were collected in the Dutch National Intensive Care Evaluation (NICE) quality registration . ICUs participating in this national registry received a standard quarterly benchmark report on all indicators, including the adherence to these two standards, as a part of the regular service.
Multifaceted feedback intervention (InFoQi program)
The ICUs in the control group did not receive any of these interventions except the quarterly standard benchmark reports (Fig. 1, right side).
In a clustered RCT, the hypothesis was tested that ICUs receiving the multifaceted feedback intervention improved their adherence to the two organizational standards more than ICUs receiving standard feedback reports only. Dutch ICUs were randomized to either the intervention or the control arm . The Institutional Review Board of the Academic Medical Center (Amsterdam, The Netherlands)—in accordance with Dutch and European legislation—deemed formal IRB approval and patient consent was not necessary because the intervention did not directly involve patient care. The trial was registered at Current Controlled Trials (www.controlled-trials.com; ISRCTN50542146). The study results are reported according the CONSORT statement for cluster randomized trials .
In- and exclusion criteria
ICUs were eligible to participate in the study if they were planning to participate in the Dutch NICE, which included monthly submission of indicator data to the database. ICUs were excluded when they were not able to allocate resources to perform necessary study activities such as establishing a local multidisciplinary QI team. The medical managers of all participating ICUs signed a consent form to formalize leadership support.
The primary study endpoints were defined as the following: (1) Adherence to the standard for ICU bed occupancy rate. The ICU bed occupancy rate was defined as the proportion of shifts with a bed occupancy rate 80 % or below (with lower proportions implying higher adherence levels). We calculated bed occupancy rate as the maximum number of patients simultaneously treated on any moment during an 8-h nursing shift divided by the number of operational beds in that same shift. A bed was defined as “operational” when monitoring and ventilation equipment as well as nursing and medical staff is available. (2) Adherence to the nurse-to-patient ratio of 0.5 or above. A higher proportion of nursing shifts with a ratio above 0.5 implied higher adherence levels. The nurse-to-patient ratio was calculated by the maximum number of patients simultaneously present on any moment during an 8-h nursing shift divided by the number of registered qualified ICU nurses in that same shift; student-nurses were not included in this calculation.
Randomization and blinding
Allocation was based on a stratified randomization with a block size of four. ICUs were stratified by patient volume (more/less than the national median number of ventilated, non-cardiac surgery admissions) and whether they had participated (yes/no) in a pilot study to evaluate feasibility of indicator data collection . An independent researcher—blinded to group assignment—generated the allocation list using a computer-generated randomization scheme. It was impossible to conceal study arm allocation from participating ICUs or from those providing the reports or outreach visits due to the nature of the intervention.
The total study period lasted 16 months starting at randomization to 3 months after sending the last feedback report. These 16 months consisted of a 2-month run-in period, which was the period between randomization and the first outreach visit, directly followed by 14 months of follow-up. The study period for control ICUs was equally defined as a run-in period for the first 2 months after randomization with a follow-up period of 14 months. The data were collected locally by the participating ICUs and uploaded monthly to the national database. The data were commonly recorded per nursing shift using dedicated software. The data included admission and discharge times and dates of individual ICU admissions, the number of operational beds, and the number of qualified nurses per shift. The ICU bed occupancy rates and nurse-to-patient ratios per shift were centrally calculated from the uploaded raw data. The NICE registry assured the quality of submitted data by means of periodical on-site data quality audits and automated data range and consistency checks [21, 26]. Data on QI activities in the intervention ICUs were extracted from the local QI action plans as formulated during outreach visits.
Immediately after inclusion in the study, we interviewed one representative per participating ICU by phone to record whether their unit had any QI infrastructure available (e.g., a quality manager), and how they handled the standard quarterly feedback reports at study baseline. We repeated this interview at the end of follow-up.
The unit of analysis was the individual nursing shift. Each ICU had three nursing shifts per day. The effect of the intervention on the proportion of shifts with a bed occupancy rate below 80 % was analyzed with logistic regression analyses. We used generalized estimation equations with an autoregressive correlation structure to account for clustering of shift occupancy observations within ICUs. The same procedure was followed to analyze the proportion of shifts with a nurse-to-patient ratio above 0.5. The change in proportions was analyzed by testing for the effects of group (intervention versus control), study period (baseline versus follow-up), and the interaction between group and study period.
For both study end points, the odds ratios (OR) for improvements at follow-up versus at baseline were calculated separately for control and intervention ICUs. In addition, the ratio of these ORs for improvement at follow-up (OR intervention/OR control) was calculated . This ratio of ORs should be interpreted as the OR of changes at follow-up in the intervention group adjusted for changes at follow-up in the control group. A ratio of ORs > 1 implied that improvement at follow-up is larger in the intervention group than in the control group.
In all analyses, we adjusted for covariates defined as type of shift (day, evening, and night) and seasonal fluctuation in admission levels by including calendar month dummies for both indicators. In addition, to adjust for differences in organizational structure between the study arms, we used two covariates on the ICU level. In the analyses for the bed occupancy rate, these two covariates were availability of an emergency bed (yes/no) and hospital type (academic or teaching/non-teaching). For the nurse-to-patient ratio, the covariates we added were the proportion of newly admitted mechanical ventilated patients and hospital type.
To account for correlation between the two study endpoints, we performed a secondary analysis in which we added nurse-to-patient ratio as an additional covariate in the analysis for the bed occupancy rate, while also adding bed occupancy rate as a covariate in the analysis for the nurse-to-patient ratio.
Sample size calculation
To determine the minimally required number of ICUs completing the trial, we used an analysis of the NICE registry data from 18 ICUs in 2008. First, we ranked these 18 ICUs by the percentage of shifts with a bed occupancy rate exceeding 80 % (i.e., failure rate). We than applied the achievable benchmarks of care method to determine what would be an achievable improvement by calculating the difference in failure rate of ICUs in the upper half of the ranking (23 %) and the failure rate among all units (44 %) . This amounted to an absolute reduction of 21 %, corresponding to a relative reduction of 48 %. Calculations based on the Binomial distribution showed that we needed 23 ICUs to detect this difference with 80 % power at a type I error risk (α) of 5 %. In this calculation, we took into account an intra-cluster correlation of 0.28, which we estimated from the 2008 national data. We used SPSS version 16.0 for all statistical analyses.
Baseline characteristics of participating ICUs and included shifts in the InFoQI program
ICU level characteristics
Number included in analysis
Academic or teaching hospital
Cardiac surgery center
Emergency bed available
Participated in indicator pilot study
Shift level characteristics
Number included in analysis
Bed occupancy rate
Number of operational bedsa
Bed occupancy ratea
Percentage of shifts adhering to standard for bed occupancy rateb
Number of qualified nursesa
Percentage of shifts adhering to standard for nurse-to-patient ratio
Effect of the multifaceted feedback intervention
Results of the multifaceted intervention program on study end points
Crude effect of intervention compared to baselineb
Adjustedc effect of intervention compared to baseline
Crude effect of intervention compared to control groupd
Adjustedc,d effect of intervention compared to control group
Main effect (95% CI)
Main effect (95% CI)
Main effect (95% CI)
Main effect (95% CI)
Proportion of shifts with bed occupancy rate below 80 %
Proportion of shifts with nurse-to-patient ratio above 0.5
The proportion of shifts with a bed occupancy rate below 80 % is higher in follow-up compared to baseline for the control ICUs, but the ratio of change during follow-up between intervention and control ICUs was not statistically significant (ratio of ORs, 0.67; 95 % CI, 0.39–1.15).
The proportion of shifts with a nurse-to-patient ratio below 0.5 showed a non-significant reduction during follow-up compared to baseline for intervention ICUs (OR, 0.72; 95 % CI, 0.41–1.26). The ratio of change in intervention versus control ICUs was not statistically significant (ratio of ORs, 0.65; 95 % CI, 0.35–1.19).
The secondary analyses in which the additional covariates were added did not change the final conclusion.
Quality improvement activities
Type and examples of planned QI actions as formulated during outreach visits and the number of QI plans (n = 13) in which they appeared
Type of action
Understanding low or high variation in performance
Verifying unusually high or low occupancy rates in specific shifts
Develop standard registration procedure for combined intensive care—recovery nursing shiftsa
Investigate low performance
Investigate effect of daily operating-room schedule on bed occupancy rate
Investigate the number of student-nurses in shifts with a ratio below the lower thresholdb
Adjust (process of) resource planning
Introduce labeling of the “best patient” in each shift who could be discharged in case of an emergency admission
Take on additional part-time nurses or share nursing capacity with medium care to increase nursing team’s versatility
During the study period in the control group, two ICUs established a local QI team, while in one ICU, this was discontinued. In addition, four control ICUs reported to have started using the standard quarterly benchmark reports.
This study shows that the intervention with extended feedback reports, QI teams, and educational outreach visits did not lead to a significant improvement in the adherence to guideline-based standards for bed occupancy rates and nurse-to-patient ratios in the Dutch intensive care setting compared to standard feedback reports alone. To our knowledge, this is the first study that evaluated the effect of a multifaceted feedback intervention on the adherence to organizational guidelines in the critical care setting. Our findings are in contrast with improvements shown in adherence to clinical guideline standards after implementation of a multifaceted feedback approach [15–17]. Apparently, organizational improvements are more difficult to achieve than clinical improvements. This discrepancy may have several reasons such as study design, difficulties in changing organizational indicators, and the type of intervention.
The lack of effect may in part be explained by weaknesses in the study design. First, the control group also received feedback on adherence levels, which diluted the contrast between the two study arms. This lack of contrast could not be avoided because all ICUs in our study already participated in the national registry with regular performance feedback in some form. Another limitation is that we had to exclude six of the ICUs from our analysis due to absent or incomplete data, which may have limited the power of the study. This relatively large drop out may be due to the time-consuming and manual recording of operational beds and qualified nurses (three times daily) in ICUs without a patient data management system. However, the power analysis indicates that sufficient ICUs in our analyses remained.
Strength of our study is the use of a cluster randomized design. In addition, we investigated the potential achievable improvement prior to the study for both study end points by using actual data and a validated method .
Organizational versus clinical indicators
It is shown that the bed occupancy rate and nurse-to-patient ratio are difficult to change. These organizational issues are difficult to influence by nature, as they need change in the number of admitted patients or the number of nurses. Changing the admission rate may be difficult because a large proportion of admissions are unplanned. In order to increase the number of nurses, it is necessary to search for new employees in a short supply market or to achieve flexible availability of nursing staff. In addition, to find money for this investigation, most ICUs are hampered by budgetary cuts in health care. It is also true that the two endpoints may co-interfere as an increasing number of nurses may increase the number of operational beds (providing that the unit can increase its beds and apparatus) and thereby improve both bed occupancy rate and nurse-to-patient ratio.
Intervention and quality initiatives
The multifaceted intervention did not include practical tools to achieve improvements in the study endpoints. These practical implications had to be designed by the QI teams their selves. These tools might, for instance, have been extended information on patient admission and discharge logistics or tools to achieve flexible availability of nurses. However, our data show that most of the initiatives from the QI teams aimed to “increase the confidence in data,” which does not directly target the adherence to the guideline standards. This finding is in line with those of Bradley et al. . They described that efforts in QI were most effective when physicians perceived that the data were credible, and this data credibility often took time to develop in a hospital . In our prospectively performed barrier analysis, we found that the confidence in data was a barrier that had to be solved before this study could be able to achieve an improvement. Indeed, the process evaluation of our study did show an increase in confidence in indicator data quality , but the participating ICUs were apparently stuck in improving data quality and were not able to take a step beyond this issue. This finding suggests that the 14-month time frame of this study was too short to achieve a measurable effect. This is confirmed by the finding that the increase in confidence in the data took several months to occur. Consecutive initiatives after data improvement were, however, not often undertaken. In this respect, it is unknown whether the intensivists and nursing staff had enough latitude to change the relevant processes. In addition to the relatively short follow-up, as stated above, the feedback of organizational data may also be more difficult to explain. It may be true that recipients need more background information to correctly interpret data on their organizational performance . The organizational indicators used in this study are based on evidence-based standards, but conflicting interests may have influenced the interpretation of the data and the initiation of QI activities. Economic targets and quality improvement targets may not be inline. For example, bed occupancy rates over 80 % can be economically beneficial but can have negative effects for the accessibility of intensive care facilities. Finally, another explanation for the observed outcomes could be that the ICUs in the intervention group showed better results at baseline compared to the control group, which results in limited room for improvement in the follow-up period or due to regression to the mean.
Extrapolation and future steps
From this study, it can be learned that an improvement in organizational aspects of a guideline in the intensive care setting is not easy to achieve in a 12-month episode. For organizational indicators, in contrast to clinical indicators, a distinct implementation approach might be necessary and more time might be needed. Although our study has been conducted within the domain of intensive care, our conclusions might be true in other settings as well. Also, it is unknown whether our results are true for other countries. Future studies are needed to gain insight on these questions.
Bed occupancy rate and nurse-to-patient ratio are important hallmarks for an efficient use of critical care resources and they are associated with clinical outcomes. Improvement of the adherence to guideline-based standards on these organizational issues in critical care was studied using a cluster randomized trial. A multifaceted feedback intervention using extensive feedback data, a local quality improvement team, and an educational outreach team appeared to be ineffective. The ICUs appeared to have limited confidence in the correctness of the feedback data. The ICUs appeared to be unable to take appropriate steps beyond improvement of data quality. Moreover, this study suggests that efforts to improve bed occupancy rate and nurse-to-patient ratio may require an intervention with practical tools for improvement and a longer follow-up.
The authors thank all personnel from the ICUs that participated in the study. We also thank Eric van der Zwan and Twan Koetsier for their assistance with data management tasks. We thank the Dutch Society of Intensive Care (NVIC) for their support.
This study was funded by the institutes’ own resources.
- Aiken LH, Clarke SP, Sloane DM, Sochalski J, Silber JH. Hospital nurse staffing and patient mortality, nurse burnout, and job dissatisfaction. JAMA. 2002;288:1987–93.View ArticlePubMedGoogle Scholar
- Amaravadi RK, Dimick JB, Pronovost PJ, Lipsett PA. ICU nurse-to-patient ratio is associated with complications and resource use after esophagectomy. Intensive Care Med. 2000;26:1857–62.View ArticlePubMedGoogle Scholar
- Cho S-H, Hwang JH, Kim J. Nurse staffing and patient mortality in intensive care units. Nurs Res. 2008;57:322–30.View ArticlePubMedGoogle Scholar
- Garfield M, Jeffrey R, Ridley S. An assessment of the staffing level required for a high-dependency unit. Anaesthesia. 2000;55:137–43.View ArticlePubMedGoogle Scholar
- Dimick JB, Swoboda SM, Pronovost PJ, Lipsett PA. Effect of nurse-to-patient ratio in the intensive care unit on pulmonary complications and resource use after hepatectomy. Am J Crit Care. 2001;10:376–82.PubMedGoogle Scholar
- Iapichino G, Pezzi A, Borotto E, Mistraletti G, Meroni M, Corbella D. Performance determinants and flexible ICU organisation. Minerva Anestesiol. 2005;71:273–80.PubMedGoogle Scholar
- Metcalfe MA, Sloggett A, McPherson K. Mortality among appropriately referred patients refused admission to intensive-care units. Lancet. 1997;350:7–11.View ArticlePubMedGoogle Scholar
- Howie AJ, Ridley SA. Bed occupancy and incidence of methicillin-resistant Staphylococcus aureus infection in an intensive care unit. Anaesthesia. 2008;63:1070–3.View ArticlePubMedGoogle Scholar
- Zilberberg MD, Shorr AF. Prolonged acute mechanical ventilation and hospital bed utilization in 2020 in the United States: implications for budgets, plant and personnel planning. BMC Health Serv Res. 2008;8:242.View ArticlePubMedPubMed CentralGoogle Scholar
- Dutch Society for Anesthesiology. Richtlijn organisatie en werkwijze op intensive care- afdelingen voor volwassenen in Nederland [Guideline organisation and working processes of ICUs for adults in The Netherlands]. Van Zuiden Communications: Alphen aan de Rijn; 2006.Google Scholar
- Valentin A, Ferdinande P. ESICM working group on quality improvement. Recommendations on basic requirements for intensive care units: structural and organizational aspects. Intensive Care Med. 2011;37:1575–87.View ArticlePubMedGoogle Scholar
- Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PC, et al. Why don’t physicians follow clinical practice guidelines? JAMA. 1999;282:1458–65.View ArticlePubMedGoogle Scholar
- Vers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6, CD000259.Google Scholar
- van der Veer SN, de Keizer NF, Ravelli ACJ, Tenkink S, Jager KJ. Improving quality of care. A systematic review on how medical registries provide information feedback to health care providers. Int J Med Inform. 2010;79:305–23.View ArticlePubMedGoogle Scholar
- Scales DC, Dainty K, Hales B, Pinto R, Fowler RA, Adhikari NKJ, et al. A multifaceted intervention for quality improvement in a network of intensive care units: a cluster randomized trial. JAMA. 2011;305:363–72.View ArticlePubMedGoogle Scholar
- Garrouste-Orgeas M, Soufir L, Tabah A, Schwebel C, Vesin A, Adrie C, et al. A multifaceted program for improving quality of care in intensive care units: IATROREF study. Crit Care Med. 2012;40:468–76.View ArticlePubMedGoogle Scholar
- Doig GS, Simpson F, Finfer S, Delaney A, Davies AR, Mitchell I, et al. Effect of evidence-based feeding guidelines on mortality of critically ill adults: a cluster randomized controlled trial. JAMA. 2008;300:2731–41.View ArticlePubMedGoogle Scholar
- Carlson RW, Weiland DE, Srivathsan K. Does a full-time, 24-hour intensivist improve care and efficiency? Crit Care Clin. 1996;12:525–51.View ArticlePubMedGoogle Scholar
- de Vos M, Graafmans W, Keesman E, Westert G, van der Voort PH. Quality measurement at intensive care units: which indicators should we use? J Crit Care. 2007;22:267–74.View ArticlePubMedGoogle Scholar
- van der Voort PHJ, van der Veer SN, de Vos MLG. The use of indicators to improve the quality of intensive care: theoretical aspects and experiences from the Dutch intensive care registry. Acta Anaesthesiol Scand. 2012;56:1084–91.View ArticlePubMedGoogle Scholar
- Arts D, de Keizer N, Scheffer G-J, de Jonge E. Quality of data collected for severity of illness scores in the Dutch National Intensive Care Evaluation (NICE) registry. Intensive Care Med. 2002;28:656–9.View ArticlePubMedGoogle Scholar
- De Vos ML, Van der Veer SN, Graafmans WC, de Keizer NF, Jager KJ, Westert GP, et al. Implementing quality indicators in ICUs: exploring barriers to and facilitators of behaviour change. Implement Sci. 2010;5:52.View ArticlePubMedPubMed CentralGoogle Scholar
- de Vos M, Graafmans W, Kooistra M, Meijboom B, Van Der Voort P, Westert G. Using quality indicators to improve hospital care: a review of the literature. Int J Qual Health Care. 2009;21:119–29.View ArticlePubMedGoogle Scholar
- van der Veer SN, de Vos ML, Jager KJ, van der Voort PH, Peek N, Westert GP, et al. Evaluating the effectiveness of a tailored multifaceted performance feedback intervention to improve the quality of care: protocol for a cluster randomized trial in intensive care. Implement Sci. 2011;6:119.View ArticlePubMedPubMed CentralGoogle Scholar
- Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. Ann Intern Med. 2001;134:657–62.View ArticlePubMedGoogle Scholar
- Arts DGT, De Keizer NF, Scheffer G-J. Defining and improving data quality in medical registries: a literature review, case study, and generic framework. J Am Med Inform Assoc. 2002;9:600–11.View ArticlePubMedPubMed CentralGoogle Scholar
- Madden JM, Graves AJ, Zhang F, Adams AS, Briesacher BA, Ross-Degnan D, et al. Cost-related medication nonadherence and spending on basic needs following implementation of Medicare Part D. JAMA. 2008;299:1922–8.View ArticlePubMedPubMed CentralGoogle Scholar
- Kiefe CI, Weissman NW, Allison JJ, Farmer R, Weaver M, Williams OD. Identifying achievable benchmarks of care: concepts and methodology. Int J Qual Health Care. 1998;10:443–7.View ArticlePubMedGoogle Scholar
- Bradley EH, Holmboe ES, Mattera JA, Roumanis SA, Radford MJ, Krumholz HM. Data feedback efforts in quality improvement: lessons learned from US hospitals. Qual Saf Health Care. 2004;13:26–31.View ArticlePubMedPubMed CentralGoogle Scholar
- De Vos MLG, van der Veer SN, Graafmans W, de Jager KJ, de Keizer NF, Westert G, et al. Process evaluation of a tailored multifaceted feedback program to improve the quality of intensive care by using quality indicators. BMJ Quality and Safety; 2013;22:233–41.View ArticleGoogle Scholar
- Pringle M, Wilson T, Grol R. Measuring “goodness” in individuals and healthcare systems. BMJ. 2002;325:704–7.View ArticlePubMedPubMed CentralGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.