Exploring the black box of quality improvement collaboratives: modelling relations between conditions, applied changes and outcomes
Implementation Science volume 4, Article number: 74 (2009)
Despite the popularity of quality improvement collaboratives (QICs) in different healthcare settings, relatively little is known about the implementation process. The objective of the current study is to learn more about relations between relevant conditions for successful implementation of QICs, applied changes, perceived successes, and actual outcomes.
Twenty-four Dutch hospitals participated in a dissemination programme based on QICs. A questionnaire was sent to 237 leaders of teams who joined 18 different QICs to measure changes in working methods and activities, overall perceived success, team organisation, and supportive conditions. Actual outcomes were extracted from a database with team performance indicator data. Multi-level analyses were conducted to test a number of hypothesised relations within the cross-classified hierarchical structure in which teams are nested within QICs and hospitals.
Organisational and external change agent support is related positively to the number of changed working methods and activities that, if increased, lead to higher perceived success and indicator outcomes scores. Direct and indirect positive relations between conditions and perceived success could be confirmed. Relations between conditions and actual outcomes are weak. Multi-level analyses reveal significant differences in organisational support between hospitals. The relation between perceived successes and actual outcomes is present at QIC level but not at team level.
Several of the expected relations between conditions, applied changes and outcomes, and perceived successes could be verified. However, because QICs vary in topic, approach, complexity, and promised advantages, further research is required: first, to understand why some QIC innovations fit better within the context of the units where they are implemented; second, to assess the influence of perceived success and actual outcomes on the further dissemination of projects over new patient groups.
In the last decade, many countries have initiated quality improvement collaboratives (QICs) in healthcare settings. QICs bring together 'groups of practitioners from different healthcare organisations to work in a structured way to improve one aspect of the quality of their service. It involves them in a series of meetings to learn about best practices in the area chosen, about quality methods and change ideas, and to share experiences of making changes in their own local setting' . Another important feature of collaboratives is the use of continuous quality improvement methods to realise changes. Continuous quality improvement is a proactive philosophy of quality management featuring multi-disciplinary teamwork, team empowerment, an iterative approach to problem solving, and ongoing measurement [2, 3]. QICs are presented as 'arguably the healthcare delivery industry's most important response to quality and safety gaps', representing substantial investments of time, effort, and funding . Nevertheless, the problem is that despite its popularity, the evidence for QIC effectiveness is positive but limited [3–5]. Effects cannot be predicted with great certainty . Therefore researchers urge for more investigation into the different types of QICs and their effectiveness, as well as linking QIC practices explicitly to organisational and change management theory [1, 4, 7–9]. Or, as stated by Cretin et al., it is important to open the 'black box' of QIC implementation .
The current study intends to contribute to a better understanding of the processes and outcomes of QIC implementation in the context of a change programme for 24 Dutch hospitals based on 18 QICs. This programme--a multi-level quality collaborative--is aimed at organisational development and the dissemination of healthcare innovations . It is the third pillar of 'Better Faster', a programme embedded in a broader national policy mix involving an increase in managed competition and transparency, a new reimbursement system based on standardised output pricing, and an intensified role for public actors (like the Healthcare Inspectorate), patient representatives, and healthcare insurers in monitoring the quality and safety of care (see Appendix 1) [10–14]. The multi-level quality collaborative is based on the implementation of different breakthrough collaboratives in the areas of patient safety and logistics. The patient safety targets involve pressure ulcers, medication safety, and postoperative wound infections. Logistics teams deal with operating theatre productivity, throughput times, length of in-hospital stay, and access time for outpatient appointments (for details see Table 1).
This study aims to answer two questions: to what extent do expected relationships between conditions, applied changes, and outcomes of QIC-implementation exist; and can differences in conditions and outcomes be explained by the fact that the teams belong to different QICs and hospitals?
This study focuses on relations between relevant conditions for successful QIC implementation, on changes in working methods and activities, and on patient-related outcomes. In opposite order, the outcomes involve perceived project successes and actual progress made in the area of patient safety and logistics. Changes in working methods and activities have to do with all the new or intensified efforts taken by the teams on behalf of their project. The literature on the implementation and dissemination of innovations in health service organisations contains many descriptions of success conditions, linked to the tasks and responsibilities of the actors involved in QIC efforts [15, 16]. An important assumption behind QICs as an improvement and spread tool  is that knowledge about best practice is made available to teams by external change agents. The teams implement this in their own hospital setting. For this reason, three categories of conditions can be recognised: the organisation of the multi-disciplinary teams that join a QIC and transform the knowledge into action (to avoid confusion, in this study team organisation and teamwork have the same meaning); the degree of support these teams receive from their hospital organisation; and the support given by the external consultants/change agents who facilitate the QIC and its meetings .
This affects the teams joining a QIC. Cohen and Bailey defined a team as 'a collection of individuals who are interdependent in their tasks, who share responsibility for outcomes, who see themselves and who are seen by others as an intact social entity embedded in one or more larger social systems (e.g., business unit or corporation), and who manage their relationships across organisational boundaries' . There is a general consensus in the literature that a team consists of at least two individuals who have specific roles, perform interdependent tasks, are adaptable, and share a common goal . To increase team effectiveness, it is important to establish timely, open, and accurate communication among team members . The notion that QIC teams are responsible and in charge of project progress  is in line with the literature suggesting that operational decision-making during implementation processes should be devolved to teams .
Other imperatives for team success are strong organisational support and integration with organisational key values . Within QICs, organisational support has to do with the leadership, support, and active involvement by top management [21, 23, 24]. Regular contact is needed between team and hospital leaders, and the innovation must match the goals of the management . Øvretveit et al. state that topics should be of strategic importance to the organisation . In addition to the presence of necessary means and instruments , many of the internal support tasks are to be executed by the strategic management. Executives have to communicate a vision or key values throughout the organisation [26, 27], and must stimulate the organisation's and employees' willingness to change . These tasks fall within the priority setting areas defined by Reeleeder et al.: namely, foster vision, create alignment, develop relationships, live values, and establish processes .
External change agent support
The involvement of external change agents, arranging group meetings for teams of different organisations, is a typical QIC feature. In Table 1, the role of the external change agents within the larger programme is described. Their efforts should contribute to the empowerment and motivation of teams to implement new working methods in order to alter a quality aspect of their care delivery. Team training is a success factor for team-based implementation , and can be more effective than individual training, especially when the learning is about a complex technology . External change agents should provide teams with an applicable model together with appealing performance expectations . This implies and requires a gap between a desirable and an actual situation, as well as outlining the potential added value of the innovation to the teams . A second prerequisite is that teams joining the QIC need to gain information and skills that are new to them, otherwise an important argument for joining the QIC is void.
In an earlier study, a questionnaire was developed and validated to measure the extent to which these conditions are met . In this article, a model will be tested based on a number of hypotheses that affect the relation between conditions, team-initiated changes due to QIC participation, and two outcome measures (Figure 1).
In the literature, a positive relation is suggested between the presence of these conditions and successful implementation of change [15, 16, 24]. Successful implementation means that teams manage to adopt new working methods or to alter existing practices. The 18 QICs within the multi-level quality collaborative were aimed at achieving specific targets in the area of patient safety and patient logistics. The implementation of the new working methods and improvement concepts was to be advocated and supported by the external change agents of the QICs. Programme hospitals were expected to provide the necessary internal support. The teams, moreover, were made responsible for the progress of the implementation in their own local hospital setting. Based on the literature and the tasks and responsibilities of actors within the programme in which the QICs are implemented, two hypotheses can be formulated:
Hypothesis A: organisational support, team organisation and external support have a positive effect on the number of applied changes by teams.
Hypothesis B: the number of applied changes has a positive effect on perceived and actual outcomes.
Both hypotheses imply a causal relation. In other instances, it is more difficult to determine the direction of an effect. This applies to hypotheses C and D. Because (A) the number of applied changes is hypothesised to be influenced by the presence of the right conditions and (B) an increase in the number of applied changes has a positive effect on the outcomes, it is logical that (C) the presence of the conditions is expected to be positively related to the outcomes of the implementation:
Hypothesis C: a positive relation exists between conditions and outcomes.
A final assumption has to do with the relation between perceived and actual project outcomes. If an outcome indicator shows that a project's main topic is improved, a project leader is more likely to be positive about the success of the project. Or conversely, if the team leader has a tendency to think more positively about the result, this may have influenced his or her behaviour in such a way that it actually contributed to a higher level of improvement.
Hypothesis D: a positive relation exists between perceived and actual outcomes.
The total study population consists of 168 teams from 24 hospitals and 18 QICs. Project teams from three hospital groups started, one group after the other, in October 2004, October 2005, and October 2006, with the implementation of the six types of QIC projects as described in Table 1.
Data sources and variables
Two data sources were accessed to gain information on six variables that were used for the purpose of statistical modelling. The QIC team leaders served as a first data source. In January 2006, 2007, and 2008, the team leaders received a questionnaire at the end of the first year of implementation and were asked to rate the overall success of their project on a scale from zero (min) to ten (max). Other questions reflected relevant conditions for successful implementation. Principal component analysis showed that several of the items measured with the questionnaire (on a seven-point scale) cluster together into three constructs, resembling the categories described in the introduction: organisational support, team organisation, and external change agent support (for information on the items see the notes under Table 2). Scale reliability, internal item consistency, and divergent validity were satisfactory . To measure the number of applied changes, eight activities, relevant for achievement of the project goal, were selected for each QIC from the QIC instruction manuals. Team leaders could mark one out of four options--this is something: we do not do, we have already done, we have intensified/improved since the start of the project, or completely new. For each team, the number of applied changes (intensified/improved or new since the project began) was counted. The applied change rate ranges from zero (no change) to eight (high number of changes).
Each QIC served a particular purpose. The external change agents translated project targets into measurable indicators, and teams had to deliver monitoring data to a central database. In this study, these monitoring data were used to model the actual success of the teams. An agreement was made with the organisation funding the programme (as well as the independent evaluation, of which the current study is a part) that the data collection burden for participating hospital staff was to be minimised. Therefore, the central database was the sole source for team performance indicators. Spreadsheet files with team monitoring data were provided three times by the change agency approximately six months after the end of the first implementation year (April to June 2006, 2007, and 2008). These data were used in the analyses that are described later. Project indicators were: prevalence of pressure ulcers (pressure ulcers), prevalence of wound infections (postoperative wound infections), access time for outpatient appointments in days (waiting lists), throughput time for diagnostics and treatment in days (process redesign), and percentage of allocated time actually used (operating theatre productivity). Three types of medication-safety projects had their own indicators: percentage of unnecessary blood transfusions, intravenous antibiotics, or patients with a pain score above four. Medication-safety scores were calculated using the first and last 20 patients treated. Pressure ulcers, operating theatre productivity, and waiting-list project results were based on the change between the scores of the first and last two months. In the case of process redesign and postoperative wound infections, the project period was compared to an identical period in the past.
The change percentages in this study were converted into a three-point scale: (1) at least 10% worse than before, (2) neutral, and (3) improved by at least 10%. Compared to goals such as 30%, 40 to 90% and 50% improvement (Table 1), 10% improvement seems modest. However, several evaluations revealed that even 10% is unrealistic for some teams, making a higher threshold too strict [32, 33]. A lower threshold is not an option either, because then the improvement is no longer substantial. It is known from research that an average improvement rate of 10% is common , particularly if the improvement strategy--e.g., breakthrough--is based on feedback .
Multi-level regression analyses were conducted to answer the research questions. The main argument behind multi-level modelling is that social processes often take place within a layered structure. The assumption that data structures are purely hierarchical, however, is often an over-simplification. Entities, such as people or teams, may belong to more than one grouping, and each grouping can be a source of variation. Each team in the current study belongs to one of the 18 QICs and to one of the 24 programme hospitals. For that reason, a cross-classified multi-level model is the most accurate model to study the hypothesised relations between conditions, applied changes and outcomes (Figure 2).
The variance can be separated into three parts: one due to differences between teams (level one), one due to differences between QICs (level two), and one due to differences between hospitals (level three). In the model, the hypotheses were tested in a three-level cross-classified structure as depicted in Figure 2. Intercept variances of all variables were estimated at all three levels. Correlations between the variables were estimated at level one to begin with (given the relatively limited sample size), and at higher levels if the variables belonging to the relations in Figure 1 differed between QICs or hospitals. Five fixed effects were included in the model to test the relation between conditions and applied changes (hypothesis A) and between applied changes and outcomes (hypothesis B).
All analyses were performed using MLwiN software version 2.02. Estimation method was iterated generalised least squares (IGLS) .
A total of 168 team leaders, belonging to 23 hospitals (one hospital refused to participate) and 18 QICs, filled out the questionnaire (71% response rate). Table 2 contains the means, medians, inter-quartile ranges and ranges of the three conditions, the number of applied changes, perceived success, and actual outcome. The number of changed activities was known of 95% of the responding teams (n = 159), overall grades (perceived success) are available with regard to 82% of the teams (n = 137), and 61% of the teams were capable and willing to deliver enough monitoring data to calculate a before and after measurement (actual outcome) (n = 103). Indicator data were available of 94% of the operating theatre productivity teams, 82% of the pressure ulcer teams, 78% of the waiting list teams, 50% of the wound infection teams, 41% of the medication safety teams, and 36% of the process redesign teams.
Team activities and actual outcomes per project type
The information presented in Table 3 serves as background material. The table shows the number of teams who changed their activities after the project had begun and the average number of applied changes per project type. Pressure ulcer teams mainly applied regular change of patient position (68%) and performed a risk assessment (64%). Medication safety interventions predominantly reflect the three sub-topics the teams dealt with: postoperative pain, blood transfusions, and intravenous antibiotics (29 to 38%). Operating theatre teams focused on starting on time (61%). Wound infection teams reduced the number of door movements and the number of individuals in the operating theatre (89%). They also paid attention to a protocol for optimal administering of antibiotic prophylaxis (61%). Process redesign teams reduced the number of planning moments, reserved slots for specific diagnosis (61%), and clarified decision lines and division of responsibilities (58%). Waiting list teams blocked agendas for six to eight weeks (72%) and anticipated fluctuations (64%). The average number of applied changes per project type ranged from 2.06 (medication safety) to 4.4 (working without waiting lists).
As well as the average changes in activities, the percentage of teams (with data available) experiencing an improvement in the performance indicator by at least 10% also differs between the six project types. This criterion is met by 70% of the pressure ulcer teams (reduction of pressure ulcers), 100% of the medication safety teams, 12% of the operating theatre teams (use of allocated time), 56% of the wound infections teams, 83% of the process redesign teams (throughput times for diagnostics and treatment), and 46% of the waiting list teams (access time).
To learn more about the process and outcomes of QIC implementation, the four hypotheses were tested using multi-level analyses. In Table 4, the estimated correlations, fixed effects, random effects, and the percentage of variance at each level are shown.
Hypothesis A concerns the relation between the three conditions and the number of changes teams applied. The association between organisational support and external change agent support and the number of applied changes is confirmed to be significant (p < 0.001). An increase in organisational support or external change agent support is accompanied by an increase in the number of applied changes. The relation between team organisation and the number of applied changes is insignificant. The multi-level model reveals that organisational support differs significantly between hospitals: 18% of the variance is situated at hospital level. Hypothesis B concerns the effect of applied changes on project outcomes. An increase in the number of applied changes is verified to have a positive effect on perceived success (p < 0.001) and indicator outcomes (p < 0.05). Hypothesis C involves the direct relation between conditions and outcomes. In the case of organisational support and perceived success, and team organisation and perceived success, a positive correlation was found of 0.29 (p < 0.001) and 0.30 (p < 0.001), respectively. The relation between external change agent support and perceived success is not significant (p > 0.05), similar to the relation between the three conditions and actual outcome (p > 0.05). In addition to these test results, a two-tailed Sobel Test was conducted to determine whether the relation between the support conditions and both outcomes is mediated by the number of applied changes . Partial mediation effects were confirmed in the case of organisational support and perceived success (test statistic: 2.77; p < 0.01) and external change agent support and perceived success (test statistic: 3.45; p < 0.001). The mediation of the relationship between conditions and actual outcome is less significant (p < 0.10). At team level, hypothesis D, the existence of a positive relation between perceived success and actual outcome could not be confirmed (p > 0.05). Perceived successes and actual outcomes differ significantly between QICs (p < 0.05). By means of an iterative process, the possibility was explored that the expected hypothesised relation exists at QIC level. After an estimation of the level-two correlation between both variables, the relation could be confirmed: there is a maximal correlation at QIC level (Pearson's r = 1.00; p < 0.05). At this higher group level, perceived successes say more about actual outcomes than at the level of individual teams.
In this article, a model was tested to gain a better understanding of the QIC black box. The study objective was to answer two questions.
Question 1: Do expected relationships exist between conditions, applied changes, and outcomes?
The analysis resulted in several findings, contributing to a better understanding of the implementation process that took place in the context of the multi-level quality collaborative.
First, when a team leader is more positive about organisational and external change agent support, this has a positive effect on the number of intensified or new working methods applied by the team. Second, a higher number of applied changes has a positive influence on the degree of perceived success and actual outcomes. Third, positive relations between perceived success and organisational support and team organisation could be confirmed. The direct connection between actual outcomes and the three conditions is insignificant. Moreover, the relation between perceived success and organisational support and external change support is partly mediated by the number of applied changes. With regard to the degree of actual success, a similar mediation effect could be verified with 90% certainty.
Finally, the association between actual outcome and perceived success is insignificant at team level but strong at QIC level. The high correlation between perceived and actual success at QIC level indicates that teams who joined a QIC, in which the perceived success ratings of team leaders are high, have also relatively high performance indicator scores.
Question 2: Are differences in conditions and outcomes due to nesting in hospitals or to QICs?
The multi-level model adds an important dimension that would have been overlooked in a single-level approach. Judgements on external change agent support and team organisation and actual outcomes do not seem to differ between hospitals, but organisational support does. Not one of the conditions differs at QIC level. In the case of external change support, this is particularly interesting because this condition represents the core of the QIC. Apparently, there are no differences in external change agent support between QICs, while at the same time QICs do differ in the level of perceived and actual success. Nevertheless, the finding that an increase in external change agent support is accompanied by an increase in the number of applied changes confirms the relevance of external change agents within QICs as a mechanism for best practice transfer.
It was mentioned in the introduction that the evidence on QIC effectiveness is mixed but positive. Mittman explained how subjective ratings provided by collaborative participants and leaders are subject to unintentional and unrecognised biases generated by common human decision and judgment heuristics. In that respect, he exemplified how a combination of expectation biases and belief perseverance produces systematic overweighting of evidence and observations. A priori expectations and beliefs are confirmed, while evidence that does not support the effectiveness of the QIC method is underweighted or discounted . This study confirms the risk addressed by Mittman. The overall judgement of an individual team leader is confirmed to say little about actual indicator outcomes and vice versa. This is not necessarily a bad thing--at least as long as the evaluation goal is not about assessing cost effectiveness or public accountability of the means invested in QIC programmes. Still, parties involved in implementing QIC projects should be cautious when it comes to rating and explaining the merits of their work, especially when monitoring data are not yet available. This also applies to QIC researchers who use perceived successes as proxy variables for actual performance. The overall success judgement apparently represents something other than monitored progress towards project goals. Like the actual outcomes, it depends on the number of applied changes. It is also likely that team leaders base their success judgement on other accomplishments: for instance, they notice how patients benefited from the project or how the team managed to change old routines and implemented new interventions that are expected to pay off in the long run.
The study confirms the association between organisational and external change agent support and the number of changes realised by QIC teams. Hospital managers, project teams, external change agents, and public stakeholders may benefit from the survey instrument, because it potentially provides tangible information, applicable for real-time adjustments or intake procedures.
Researchers are in a situation where relevant questions remain unanswered. Generally, the advice to adopt hierarchical models in future research should be taken as seriously, as are recommendations for more experimental , narrative , or action-based research studies . Further research is needed to test the effectiveness of QICs as spread strategy  and to assess how external change agent support influences team organisation, how team learning within a QIC takes place, and how QICs contribute to organisational learning. In addition to the black box of QIC implementation, there is another black box that needs to be opened: that of sustainability. In the extensive 'diffusion of innovation' review, Greenhalgh et al. found many studies addressing adoption, implementation, and diffusion, but only a limited number of studies dealing with sustainability .
Strengths and weaknesses
The multi-level approach is one of the strengths of this study. Other strengths are that the conditions were measured using a validated and reliable instrument, and perceptions were linked to outcome data. The dependence on data provided by the teams is a limitation. Despite the high response rate, the use of self-reported perceptions always involves a risk of overestimation or social desirability. Outcome indicators could be linked to questionnaire data in 61% of all teams in the study sample. It is very likely that the positive results are overrepresented, particularly because the absence of monitoring data may very well be caused by the fact that teams were incapable of implementing the project (and the required measurements) as planned. In that sense, actual outcomes presented in this article do not entirely represent the overall level of success of the programme.
While the vast majority of the projects had a planned length of one year, operation theatre, process redesign, and postoperative wound infections were in fact two-year projects. Because the team questionnaires were administered at a fixed moment by the end of the first year, second-year data on conditions, perceived success, and applied changes are unfortunately unavailable. Hence, for practical reasons, the analyses described in this article are based entirely on first-year data. A potential limitation is that the success level of two-year projects was determined without the project being finished. At first glance, it is reasonable to assume that the improvement rate of those projects is likely to be more positive after two years. A recent evaluation, however, illustrates that the level of improvement has remained the same . An additional analysis would yield similar results.
Finally, the number of applied changes was modelled without taking into account the influence of individual and key interventions or specific combinations. In reality, some interventions are more time-consuming and complex than others, and some may not even be suited for application within a collaborative .
By examining 18 QICs, part of a quality improvement programme for hospitals, several expected relationships could be verified. Organisational and external change agent support had a positive influence on the number of changes applied by QIC teams during the implementation. The number of applied changes had a positive effect on perceived success as well as on actual outcomes. By taking into account the fact that teams are nested in hospitals and in QICs, it became clear that some hospitals are better than others in providing organisational support. Project outcomes differ between QICs. One should be cautious when accepting perceived successes as a proxy for the actual success of individual teams.
Appendix 1 - Description of the three pillars of Better Faster
The purpose of the first pillar was to create awareness and provide room for new paradigms by having authoritative experts from other fields of service delivery and industry communicate appealing approaches and ideas about how to deal with issues of safety, logistics, and accountability in healthcare. Focus was added to national and local discussions on necessary changes.
Transparency is thought to guide purchasing decisions and improvement efforts. The second pillar is considered an important step in generating comparative data on healthcare quality. A national set of standardised quality indicators for hospital care has been developed and maintained by the Healthcare Inspectorate .
A national programme to stimulate transparency, efficiency, and quality of care was implemented in three groups of eight hospitals between 2004 and 2008, covering approximately a quarter of all hospitals in the Netherlands. This multi-level quality collaborative combined interventions at the bottom and the top of member organisations.
At bottom level, physicians, nursing staff, and managers were encouraged to participate in quality improvement collaboratives to continuously improve the quality of their work by trying out interventions using a breakthrough model while being supported by their institution and by external change agents (Table 1).
At top level, hospital executives participated in a special collaborative leadership programme (leadership and organisational development). An internal programme organisation was established to monitor the progress of the various programmes. The strategic management was expected to encourage active staff participation  and to support the implementation and spread of the new working methods and results. Feedback loops were established at unit and process level, part of the learning cycles during the implementation of the breakthrough model. In addition, the leadership programme strived explicitly for realisation of feedback loops at institutional level to promote the congruence between strategic hospital goals and the performance at unit level .
Øvretveit J, Bate P, Cleary P, Cretin S, Gustafson D, McInnes K, McLeod H, Molfenter T, Plsek P, Robert G: Quality collaboratives: lessons from research. Quality and Safety in Health Care. 2002, 11: 345-351. 10.1136/qhc.11.4.345.
Ferlie EB, Shortell SM: Improving the quality of health care in the United Kingdom and the United States: a framework for change. The Milbank Quarterly. 2001, 79: 281-315. 10.1111/1468-0009.00206.
Cretin S, Shortell SM, Keeler EB: An evaluation of collaborative interventions to improve chronic illness care: framework and study design. Evaluation Review. 2004, 28: 28-51. 10.1177/0193841X03256298.
Mittman BS: Creating the evidence base for quality improvement collaboratives. Annals of Internal Medicine. 2004, 140: 897-901.
Øvretveit J: What are the best strategies for ensuring quality of hospitals. Health Evidence Network. 2003, Copenhagen, World Health Organisation Europe
Schouten LMT, Hulscher MEJL, Van Everdingen JJE, Huijsman R, Grol RPTM: Evidence for the impact of quality improvement collaboratives: systematic review. British Medical Journal. 2008, 336: 1491-1494. 10.1136/bmj.39570.749884.BE.
Landon BE, Wilson IB, McInnes K, Landrum MB, Hirschhorn L, Marsden PV, Gustafson D, Cleary PD: Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: the EQHIV study. Annals of Internal Medicine. 2004, 140: 887-896.
Leatherman S: Optimizing quality collaboratives. Quality and Safety in Health Care. 2002, 11: 307-10.1136/qhc.11.4.307.
Solberg LI: If you've seen one quality improvement collaborative ... Ann Fam Med. 2005, 3: 198-199. 10.1370/afm.304.
Dückers M, Makai P, Vos L, Groenewegen P, Wagner C: Longitudinal analysis on the development of hospital quality management systems in the Netherlands. International Journal for Quality in Health Care. 2009, 21: 330-40. 10.1093/intqhc/mzp031.
Berg M, Meijerink Y, Gras M, Goossensen A, Schellekens W, Haeck J, Kallewaard M, Kingma H: Feasibility first: developing public performance indicators on patient safety and clinical effectiveness for Dutch hospitals. Health Policy. 2005, 75: 59-73. 10.1016/j.healthpol.2005.02.007.
Custers T, Arah O, Klazinga N: Is there a business case for quality in the Netherlands? A critical analysis of the recent reforms of the health care system. Health Policy. 2007, 82: 226-239. 10.1016/j.healthpol.2006.09.005.
van de Ven WPMM, Schut FT: Universal mandatory health insurance in the Netherlands: a model for the United States?. Health Affairs. 2008, 27 (3): 771-781. 10.1377/hlthaff.27.3.771.
Grinten Van der TED: System revision of Dutch health care: an analysis of policy reform (in Dutch). Tijdschrift voor Sociale Geneeskunde. 2007, 84: 227-233.
Greenhalgh T, Robert G, MacFarlane F, Bate P, Kyriakidou O: Diffusion of innovations in service organisations: systematic review and recommendations. The Milbank Quarterly. 2004, 82: 581-629. 10.1111/j.0887-378X.2004.00325.x.
Leggatt S, Dwyer J: Factors supporting high performance in health care organisations. 2003, Melbourne, National Institute of Clinical Studies
Dückers MLA, Wagner C, Groenewegen PP: Developing and testing an instrument to measure the presence of conditions for successful implementation of quality improvement collaboratives. BMC Health Services Research. 2008, 8: 172-10.1186/1472-6963-8-172.
Cohen SG, Bailey DE: What makes teams work: group effectiveness research from the shop floor to the executive suite. Journal of Management. 1997, 23: 239-290. 10.1177/014920639702300303.
Salas E, Dickinson TL, Converse SA, Tannenbaum SI: Toward an understanding of team performance and training. Teams: their training and performance. Edited by: Swezey RW, Salas E. 1992, Ablex Publishing Corporation, Westport, CT, 3-29.
Fried B, Rundall T, Topping S: Groups and teams in health service organisations. Health care management: organisation design and behaviour. Edited by: Shortell S, Kalunzny A. 2000, Albany: Delmar, 154-190.
Meyers PW, Sivakumar K, Nakata C: Implementation of industrial process innovations: factors, effects, and marketing implications. Journal of Product Innovations Management. 1999, 16: 295-311. 10.1016/S0737-6782(98)00044-7.
Mills PD, Weeks WB: Characteristics of successful quality improvement teams: lessons from five collaborative projects in the VHA. Joint Commission Journal on Quality and Safety. 2004, 30: 152-162.
Green PL: Improving clinical effectiveness in an integrated care delivery system. Journal for Healthcare Quality. 1998, 20: 4-8.
Gustafson DH, Sainfort F, Eichler M, Adams L, Bisognano M, Steudel H: Developing and testing a model to predict outcomes of organisational change. Health Services Research. 2003, 38: 751-776. 10.1111/1475-6773.00143.
Aubert BA, Hamel G: Adoption of smart cards in the medical sector: the Canadian experience. Social Science & Medicine. 2001, 53: 879-894. 10.1016/S0277-9536(00)00388-9.
Weiner BJ, Shortell SM, Alexander J: Promoting clinical involvement in hospital quality improvement efforts: the effects of top management, board and physician leadership. Health Services Research. 1997, 32: 491-510.
Young GJ: Managing organisational transformations: lessons from the Veterans Health Administration. California Management Review. 2000, 43: 66-83.
Parry KW: Enhancing adaptability: leadership strategies to accommodate change in local government settings. Journal of Organisational Change Management. 1999, 12: 134-156. 10.1108/09534819910263677.
Reeleeder D, Goel V, Singer PA, Martin DK: Leadership and priority setting: the perspective of hospital CEOs. Health Policy. 2006, 79: 24-34. 10.1016/j.healthpol.2005.11.009.
Edmondson AC, Bohmer RM, Pisano GP: Disrupted routines: team learning and new technology implementation in hospitals. Administrative Science Quarterly. 2001, 46: 685-716. 10.2307/3094828.
Sparks JR, Schenk JA: Explaining the effects of transformational leadership: an investigation of the effects of higher-order motives in multi-level marketing organisations. Journal of Organisational Behavior. 2001, 22: 849-869. 10.1002/job.116.
Vos L, Dückers M, Wagner C: Evaluation Better Faster pillar 3: results of an improvement programme for hospitals (in Dutch). 2008, Utrecht: NIVEL
Dückers MLA: Changing hospital care: evaluation of a multilayered organisational development and quality improvement programme. 2009, Utrecht: NIVEL, [http://igitur-archive.library.uu.nl/dissertations/2009-0826-200135/UUindex.html]
Grol R, Grimshaw J: From best evidence to best practice: effective implementation of change in patients' care. Lancet. 2003, 362: 1170-10.1016/S0140-6736(03)14546-1.
van der Weijden T, Grol R: Feedback and reminders. Improving patient care; the implementation of change in clinical practice. Edited by: Grol R, Wensing M, Eccles M. 2005, Edinburgh: Elsevier, 158-172.
Snijders TAB, Boskers RJ: Multi-level analysis; an introduction to basic and advanced multi-level modeling. 1999, London: Sage Publishers
Sobel ME: Asymptotic intervals for indirect effects in structural equations models. Sociological methodology. Edited by: Leinhart S. 1982, San Francisco: Jossey-Bass, 290-312. 10.2307/270723.
Øvretveit J: Action evaluation of health programmes and change: a handbook for a user focused approach. 2002, Oxford: Radcliffe Medical Press
Vos L, Dückers MLA, Wagner C, Van Merode GG: A breakthrough collaborative for process redesign: does it work?. Conference abstract, 25th International Society for Quality in Health Care (ISQua), Copenhagen, October, 19-22. 2008
Dückers MLA, Stegeman I, Spreeuwenberg P, Wagner C, Sanders K, Groenewegen PP: Consensus on the leadership of hospital CEOs and it impact on the participation of physicians in improvement projects. Health Policy. 2009, 91: 306-13. 10.1016/j.healthpol.2009.01.003.
Langley GL, Nolan KM, Nolan TW, Norman CL, Provost LP: The improvement guide: a practical approach to enhancing organisational performance. 1996, San Francisco: Jossey-Bass Publishers
Berwick DM: Developing and testing changes in delivery of care. Annals of Internal Medicine. 1998, 128: 651-656.
This study was funded by ZonMw, the Netherlands organisation for health research and development.
The authors declare that they have no competing interests.
MLAD was responsible for designing the study, acquiring, analyzing and interpreting the data, and drafting the manuscript. PS assisted with the analyses and interpretation of the data. As research manager of the independent evaluation study of the hospital improvement programme, CW was responsible for designing the study. CW and PPG assisted in interpreting the results and revising the manuscript for intellectual content. All authors have read and approved the final manuscript.
About this article
Cite this article
Dückers, M.L., Spreeuwenberg, P., Wagner, C. et al. Exploring the black box of quality improvement collaboratives: modelling relations between conditions, applied changes and outcomes. Implementation Sci 4, 74 (2009). https://doi.org/10.1186/1748-5908-4-74