Skip to main content

Development of a minimization instrument for allocation of a hospital-level performance improvement intervention to reduce waiting times in Ontario emergency departments



Rigorous evaluation of an intervention requires that its allocation be unbiased with respect to confounders; this is especially difficult in complex, system-wide healthcare interventions. We developed a short survey instrument to identify factors for a minimization algorithm for the allocation of a hospital-level intervention to reduce emergency department (ED) waiting times in Ontario, Canada.


Potential confounders influencing the intervention's success were identified by literature review, and grouped by healthcare setting specific change stages. An international multi-disciplinary (clinical, administrative, decision maker, management) panel evaluated these factors in a two-stage modified-delphi and nominal group process based on four domains: change readiness, evidence base, face validity, and clarity of definition.


An original set of 33 factors were identified from the literature. The panel reduced the list to 12 in the first round survey. In the second survey, experts scored each factor according to the four domains; summary scores and consensus discussion resulted in the final selection and measurement of four hospital-level factors to be used in the minimization algorithm: improved patient flow as a hospital's leadership priority; physicians' receptiveness to organizational change; efficiency of bed management; and physician incentives supporting the change goal.


We developed a simple tool designed to gather data from senior hospital administrators on factors likely to affect the success of a hospital patient flow improvement intervention. A minimization algorithm will ensure balanced allocation of the intervention with respect to these factors in study hospitals.

Peer Review reports


Balancing potential confounders in evaluation of hospital-level interventions

Rigorous evaluation of an intervention requires that its allocation be unbiased with respect to confounders. Randomization provides a mechanism for ensuring that intervention and control groups are balanced in terms of both measured and unmeasured confounders. However, if the sample size for the intervention is small there still may be substantial imbalance in the distribution of key confounders due to random error. One way to help circumvent this problem is to stratify or match on key characteristics before randomization. In order for this to work, a small but inclusive set of key potential confounders must be identified.

This paper describes a modified-delphi and nominal group process that resulted in the development of a short survey instrument that defines potential confounding factors likely to influence the success of a hospital-level intervention to improve patient flow in order to reduce emergency department length-of-stay. The purpose of the instrument is to guide the dynamic randomization of participating hospitals to the intervention, using the method of minimization. Dynamic randomization, enabled by the method of minimization, is a widely accepted randomization approach in clinical and multi-institutional trials [15]. The minimization method begins with the determination of a small number of factors known or believed to confound the effect of the intervention. The method assigns subjects to a balanced allocation sequence or to treatment groups with respect to marginal frequencies between these selected covariates. This is achieved by an algorithm that allocates the intervention to each subject, in our case, a hospital, that volunteers and is eligible to receive the intervention [68].

Overview of the intervention being evaluated

Every year in Canada more than 12 million emergency department (ED) visits are made,[9] and about a quarter of Canadians visit an ED for themselves or a close family member [10]. Recently, prolonged waiting times in EDs have been the subject of much debate in Canada and elsewhere, and several jurisdictions have launched interventions to reduce them. In 2008, the Ontario Ministry of Health (MOH) announced a provincial ED 'wait times strategy' designed to improve ED patient wait times, patient flow and patient satisfaction. The strategy includes an 'Emergency Department Process Improvement Program' (ED-PIP), a hospital-level intervention intended to improve hospital processes for admitted ED patients in order to improve access to in-patient beds and reduce ED waiting times [1115].

The intervention will be implemented over three years in approximately 90 acute care Ontario hospitals with high-volume EDs (those receiving >20,000 patient visits/annum). It will focus on organizational changes in three areas: more efficient processes (reforming/standardizing policies and practices); greater engagement of frontline staff in problem-solving; and supportive management systems. Modeled after three Ontario demonstration projects [16], the intervention is supported by a leadership and training program and organizational change experts in the form of coaching and training teams who facilitate the program in collaboration with local leaders and staff teams from participating hospitals. Change experts and hospital teams are tasked with improving processes from patient presentation in the ED to in-patient admission through to discharge by the integration of performance improvement pilot solutions across the ED and general medicine units.

In collaboration with senior decision makers at the Ontario MOH, a roll-out and evaluation strategy for the intervention was developed. The primary objective of the evaluation of the intervention is to determine whether the ED-PIP reduces total ED length-of-stay (ED-LOS). The secondary objectives are to determine the effects on time-to first physician contact and several measures of quality of care.


We conducted a literature review to identify a list of possible minimization factors to guide the allocation of hospitals to the ED-PIP. Subsequently, a multi-stage modified-delphi expert panel process was performed that included candidate factor review, quantitative assessment, and a nominal group process in a final teleconference discussion.

Literature review

To generate the list of candidate minimization factors, we reviewed databases from Management and Organizational Studies, PubMed/Medline and Ovid HealthSTAR using the search terms: organizational culture, healthcare/health system reform, transformation, intervention(s), context, evaluation, readiness for change, change management, implementation, process, and outcomes. We sought to identify articles and research papers specifically focused on organizational change and behaviour, change interventions, and research reports specific to healthcare and health services administration. One author (CL) examined all relevant references; candidate factors were considered regardless of any demonstrated empirical association to outcomes of the policy intervention under study.

The literature review [1726] generated a preliminary list of potential factors associated with the success of organizational change interventions in healthcare settings. These were organized according to a published four-stage framework for healthcare professionals managing organizational change [20]. This framework builds on observational studies in change management literature and provides a model of change implementation in healthcare organizations, informed by the implementation of a major patient safety initiative at a large, multi-site, academic hospital in Toronto, Canada. Candidate factors were retained if they were relevant to the first three stages in the framework, which represent the most applicable domains of organizational capacity and readiness for change relevant to the implementation success of the ED-PIP. The last stage addresses long-term sustainability of change initiatives. Given the breath of indicators relevant to change stage two, we expanded this stage into two subcategories: organizational readiness for change; and situational analysis and redesign of organizational systems.

Expert panel

We assembled an international multi-disciplinary panel of 21 experts consisting of hospital and ED administrators, physicians and nurse clinicians, health services and policy researchers, Ministry of Health senior leaders, organizational change researchers, and consultants with extensive experience in hospital change management interventions. Panelists represented health systems in Canada, the United Kingdom, and Australia. Diversity of experience from teaching and non-teaching hospitals was well represented among panelists. Consultants identified by two co-authors (RB, BG) were contacted and asked to nominate global experts who had experience facilitating organizational change management in health sectors abroad and were familiar with the proposed intervention concept.

Modified-delphi and nominal group process

In a preliminary stage, panelists reviewed the list of factors generated from the literature review and were asked to suggest additional factors based on their knowledge of the literature and experience with health system improvement initiatives. A final list of candidate factors was generated and a two-round modified-delphi survey process followed. In round one, panelists rated candidate factors with respect to their expected correlation (high, low, or unsure) with the allocation strata for the intervention (hospital volume and geographic region). Previous research in Ontario suggests that variation in ED-LOS is based on ED volume and the geographic region of a given hospital [27]. Factors that were highly correlated with stratification variables were excluded because any confounding associated with them would be assumed to be dealt with through stratification. Panelists also rated the degree to which the factor would likely confound the effect of the ED-PIP on achieving improvements in ED-LOS and in-patient flow. Those rated as 'somewhat' and 'very' were coded as 'predictive – potential confounder', those rated as 'slightly' and 'not at all' were coded as 'not predictive – not a potential confounder'. Factors rated by greater than 70% of panelists as 'predictive – potential confounder' were retained for the second survey.

In order to obtain a broader perspective on potential confounders, we expanded the number of participants for the second survey [28, 29]. In this phase, panelists rated each of the factors retained previously on a scale of one to nine, where one was 'completely disagree' and nine was 'completely agree' for the following three statements:

  1. 1.

    The factor measures a core component of a hospital's readiness to implement and facilitate an organizational change policy intervention aimed to improve ED-LOS and in-patient flow through to discharge.

  2. 2.

    The factor is highly predictive of the capacity for an organization to successfully implement the intervention and achieve improvements in patient flow.

  3. 3.

    The factor is evidence-based and linked to a hospital's ability to manage change activities related to the patient flow intervention.

A final score for each factor was derived by averaging the responses from the three questions noted above (a + b + c/3). Results were reviewed by panelists and discussed among the core group of panelists via teleconference guided by the nominal group technique. The highest ranking factor for each change stage domain was brought forward for discussion, definition, and specification of a measurement scale. The resulting minimization instrument was pilot tested using a web-based survey to Chief Executive Officers from six hospitals chosen to pilot the ED-PIP intervention. Hospitals were selected by the Ministry of Health. We categorized responses from one to nine as: lowest (one to three); moderately low (four, five); moderately high (six, seven); and highest (eight, nine). This study was approved by the Sunnybrook Health Sciences Centre Research Ethics Board (reference number 324-2007).


A total of 33 candidate minimization factors were generated from a literature review and initial consultation with panelists (See Additional file 1). Candidate factors related to the implementation of the ED-PIP and covered a broad spectrum of issues (see Appendix 1).

The first round questionnaire was circulated to the core group of panelists (n = 19); 11 (59%) panelists completed it. Twelve of the original 33 (36%) factors were retained for the second survey. The second round questionnaire was distributed to 21 panelists, (original 19, plus 2 international representatives) and 17 (80%) panelists completed it. Table 1 lists the second round questionnaire results for all 12 indicators emerging from the original 33. For each change stage, the top ranking factors across the domains were discussed; the factors with the highest average score in each domain were confirmed in the discussion as the consensus choice to include in the minimization algorithm. Panelist discussion via teleconference using the nominal group technique served to further clarify factor definition, appropriate wording, and response scale (one to nine) for the short survey instrument. The final four minimization factors are listed in Table 2.

Table 1 Factors relating to achievement of a patient flow improvement – organizational change policy intervention
Table 2 Minimization variables

A total of six CEOs from a selected sample of ED-PIP hospitals received an invitation to complete the online survey and all (100%) completed it. The CEOs who scored each factor highest, moderately high, moderately low and lowest were as follows, Factor 1: 4,0,1,1; Factor 2: 1,3,2,0; Factor three: 0,5,1,0; and Factor four: 0,2,2,2.


Using a combined approach of evidence synthesis and a modified-delphi panel and nominal group process we identified four factors to be used in a minimization algorithm to guide the allocation of hospitals to the ED-PIP intervention. This structured panel process reduced 33 initial candidate factors to four, expressed as a simple four-item quantitative survey instrument. To our knowledge, this is the first published example of a minimization algorithm being used to allocate hospitals to a major health system policy intervention.

The intervention being developed to improve patient flow is complex, and complex interventions generally demonstrate modest gains in empirical study [30]. Evaluating such interventions requires careful balance of known and unknown confounders, because the effect of confounders may exceed the effect of the intervention, in either direction, to create a benefit that is either not real or hide a benefit that is real. This is an important advantage of randomized studies (and one which policymakers are generally not aware of), and pragmatic randomized trials of complex interventions can be designed so that they are no more difficult for policy makers to implement, and evaluative rigor is ensured. This can be especially important when the number of intervention units is small, say less than a hundred hospitals, rather than several hundred or several thousand patients as is more typical in patient-level intervention studies.

The disadvantages of randomized trials in the healthcare system include their cost, complexity, and the desire for rapid changes evidenced within political mandates (randomized controlled trials take considerable time). Due to these issues, decision makers often implement non-randomized observational designs (e.g., before-after) that are vulnerable to confounding and offer relative uncertainty with regard to understanding the true impact of transformative efforts to improve system performance, accountability, and quality of care to the consumer. Methods such as matching or stratifying by factors such as geography, hospital type, or volume are appropriate means to balance some confounders, but there is a limit to the number of strata one may use; minimization offers an alternative or complementary approach to ensure allocation is balanced with respect to important confounders of the ED-PIP intervention.

The minimization algorithm aims to ensure unbiased allocation of the intervention during its phased roll-out. Each factor has been defined in the form of a question with a nine-level response scale. Responses from volunteering hospitals will be assessed for variance and grouped into two levels (zero 'low' and one 'moderate/high') accordingly for evaluation in the minimization algorithm. The algorithm allocates the first hospital in presenting sequence of eligibility to receive the intervention in the first (year one) or later phases of implementation at random. The algorithm then allocates subsequent hospitals to each respective phase of the intervention minimizing differences across factor levels, such that, in each phase of implementation the sample is balanced with respect to hospitals with both low and moderate/high levels of each factor. In our pilot testing, we observed substantial variability between the six respondents on three of the four factors, suggesting that our minimization factors do discriminate and are suitable for use in the minimization algorithm to guide the allocation of the intervention to hospitals. All respondents rated factor three (effectiveness of bed-management) as 'moderately high'. It will therefore be important to monitor the variability in this factor when the survey is completed by CEOs from additional hospitals in Ontario as the ED-PIP is rolled out. Further pilot testing in additional hospitals is likely required before this tool can be widely recommended.

The organizational change management literature contains a large number of potential factors or mechanisms likely to represent either a barrier or facilitator to achieving change [17, 19, 20, 23, 3139]. These are largely based on retrospective cross-sectional observation and evaluation of change interventions [40]. There are few longitudinal [41] studies or rigorous evaluations of these factors [42]. Gustafson and colleagues [39], however, offer a concise review of potential factors; and illustrate and test an 18-factor model devised to predict and explain the success or failure of a change process in healthcare settings. The model was derived from an expert panel process and literature review, but was neither evaluated with respect to objective outcomes nor designed to be used for intervention allocation purposes. Rather, the factors were compiled to guide managers initiating and managing a change initiative within a healthcare setting on actionable determinants of implementation success. The model is too complex for allocation using a minimization algorithm due to the number of factors and levels within each. Further, most factors are concerned with optimal intervention design and implementation rather than organizational culture or context factors likely to confound intervention success or failure. Our four factors are not designed as a comprehensive list of all potential factors affecting the success of a hospital level policy intervention, but rather as important hospital-specific factors likely to confound the success or failure of the intervention at all phases of implementation.

Some study limitations are worth noting with respect to our process to define potential determinants to implementation success of the ED-PIP. While our literature review was comprehensive, it was confined to English peer-reviewed publications and may not have identified all possible previously cited factors. Our consultation with the panel of experts, however, did yield additional factors in the preliminary exercise. The minimization factors were developed with specific reference to the ED-PIP intervention; therefore, the four factors we identified may not necessarily be relevant for other hospital-level interventions. However, many of the obstacles to organizational change in healthcare settings potentially affecting success of a patient flow improvement initiative are likely common to other interventions as well. Indeed, our factors are similar to previously cited themes of obstacles to implementation success described in organizational change research within and beyond the health sector [18, 19, 22, 26, 31, 3739, 43]. While our pilot results suggest reasonable variability across the four factors, we suggest caution to researchers who may wish to use these factors in other settings; piloting the instrument in a small number of centres prior to allocation based on these minimization factors is advisable.

Finally, the international membership of our panel made an in-person meeting prohibitively costly; however, regular electronic contact was maintained and timely feedback occurred. Biases may have resulted during the in-person/teleconference panel meeting from single panelists whose opinion may have been overly influential; however, the teleconference method may have mitigated this, and input was actively sought from all attendees.


Change in all industries is difficult, perhaps in none more so than healthcare, where multiple stakeholders, sometimes conflicting missions and goals, professional independence of key staff, and difficulty accessing high-quality performance data present particular challenges [20]. Policies and interventions to improve hospital performance frequently require significant human and financial resource inputs, and rigorous evaluation is necessary both to evaluate their effectiveness and to better understand organizational factors contributing to success [44, 45]. The evaluative strategy for the ED-PIP ensures that the intervention can be implemented in a way that is consistent with the needs of policy and health system decision makers, while at the same time offering a study design that provides for a rigorous evaluation of its effect on patient LOS in the ED.

Appendix 1: main themes of candidate minimization factors

  • Leadership/staff concern/prioritization of patient flow issues

  • Historical experience with change initiatives (such as: total number in the past year, intensity of previous initiatives upon staff, number of planned initiatives for the upcoming year).

  • Organizational infrastructure (such as: number of general internal medicine beds, effectiveness of bed management, information technology and decision support).

  • Communication culture across professional groups.

  • Capacities for participatory and collaborative engagement (such as: assessments of staff burn-out and staff capacity/resistance to lead, finance, or resource a change initiative).

  • Importance of added values embedded in the intervention (such as: training opportunities, communication development strategies).


  1. Treasure T, MacRae K: Minimisation: the platinum standard for trials?. BMJ. 1998, 317: 362-363.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Scott NW, McPherson GC, Ramsay CR, Campbell MK: The method of minimization for allocation to clinical trials. a review. Control Clin Trials. 2002, 23: 662-674. 10.1016/S0197-2456(02)00242-8.

    Article  PubMed  Google Scholar 

  3. Green H, McEntegart DJ, Byrom B, Ghani S, Shepherd S: Minimization in crossover trials with non-prognostic strata: theory and practical application. Journal of Clinical Pharmacy and Therapeutics 1 A.D. 26: 121-128. 10.1046/j.1365-2710.2001.00332.x.

  4. Hagino A, Hamada C, Yoshimura I, Ohashi Y, Sakamoto J, Nakazato H: Statistical comparison of random allocation methods in cancer clinical trials. Controlled Clinical Trials. 2004, 25: 572-584. 10.1016/j.cct.2004.08.004.

    Article  PubMed  Google Scholar 

  5. Signorini DF, Leung O, Simes RJ, Beller E, Gebski VJ, Callaghan T: Dynamic balanced randomization for clinical trials. Statistics in Medicine. 1993, 12: 2343-2350. 10.1002/sim.4780122410.

    Article  CAS  PubMed  Google Scholar 

  6. Pocock SJ, Simon R: Sequential treatment assignment with balancing for prognostic factors in the controlled clinical trial. Biometrics. 1975, 31: 103-115. 10.2307/2529712.

    Article  CAS  PubMed  Google Scholar 

  7. Zelen M: The randomization and stratification of patients to clinical trials. J Chronic Dis. 1974, 27: 365-375. 10.1016/0021-9681(74)90015-0.

    Article  CAS  PubMed  Google Scholar 

  8. Kenjo Y, Antoku Y, Akazawa K, Hanada E, Kinukawa N, Nose Y: An easily customized, random allocation system using the minimization method for multi-institutional clinical trials. Comput Methods Programs Biomed. 2000, 62: 45-49. 10.1016/S0169-2607(99)00047-4.

    Article  CAS  PubMed  Google Scholar 

  9. Chan B, Schull MJ, Schultz S: Atlas of Emergency Department Services in Ontario 1992/1993 to 1999/2000. Toronto. 2001

    Google Scholar 

  10. Satisfaction with Health Care Services: A Survey of Albertans 2004.

  11. Schull M: Benchmarking patient delays in Ontario's emergency departments: what are we waiting for?. Healthc Q. 2005, 8: 21-22.

    Article  PubMed  Google Scholar 

  12. Hospital Emergency Department and Ambulance Effectiveness Working Group: Improving access to emergency services: a system commitment. Toronto. 2005

    Google Scholar 

  13. Ontario Hospital Association, Ontario Medical Association, Ontario Ministry of Health: Improving access to emergency care: addressing system issues. Toronto. 2006

    Google Scholar 

  14. Canadian Institute for Health Information: Understanding Emergency Department Wait Times: Access to Inpatient Beds and Patient Flow. Ottawa, ON, Canada. 2007

    Google Scholar 

  15. Alberti KG: Transforming Emergency Care in England. London. 2005

    Google Scholar 

  16. Macleod H, Bell RS, Deane K, Baker C: Creating sustained improvements in patient access and flow: experiences from three Ontario healthcare institutions. Healthc Q. 2008, 11: 38-49.

    Article  PubMed  Google Scholar 

  17. Jetten J, O'Brien A, Trindall N: Changing identity: predicting adjustment to organizational restructure as a function of subgroup and superordinate identification. Br J Soc Psychol. 2002, 41: 281-297. 10.1348/014466602760060147.

    Article  PubMed  Google Scholar 

  18. Way C, Gregory D, Davis J, Baker N, LeFort S, Barrett B, Parfrey P: The impact of organizational culture on clinical managers' organizational commitment and turnover intentions. J Nurs Adm. 2007, 37: 235-242. 10.1097/01.NNA.0000269741.32513.7f.

    Article  PubMed  Google Scholar 

  19. Redfern S, Christian S: Achieving change in health care practice. J Eval Clin Pract. 2003, 9: 225-238. 10.1046/j.1365-2753.2003.00373.x.

    Article  PubMed  Google Scholar 

  20. Golden B: Transforming healthcare organizations. Healthc Q. 2006, 10 (Spec No: 10-9, 4):

  21. Bell R, Golden B, Lee L: Transforming healthcare organizations: looking back to see the future. Healthc Q. 2006, 10 Spec No: 84-87. 84-7, 6.

    Article  PubMed  Google Scholar 

  22. Hamilton S, McLaren S, Mulhall A: Assessing organisational readiness for change: use of diagnostic analysis prior to the implementation of a multidisciplinary assessment for acute stroke care. Implement Sci. 2007, 2 (21): 21-10.1186/1748-5908-2-21.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Kane-Urrabazo C: Management's role in shaping organizational culture. J Nurs Manag. 2006, 14: 188-194. 10.1111/j.1365-2934.2006.00590.x.

    Article  PubMed  Google Scholar 

  24. Heartfield M: Regulating hospital use: length of stay, beds and whiteboards. Nurs Inq. 2005, 12: 21-26. 10.1111/j.1440-1800.2005.00251.x.

    Article  PubMed  Google Scholar 

  25. Davies M: Challenges of Inpatient flow in VA. Forum. 2007, 1-8.

    Google Scholar 

  26. Devereaux MW, Drynan AK, Lowry S, MacLennan D, Figdor M, Fancott C, Sinclair L: Evaluating organizational readiness for change: a preliminary mixed-model assessment of an interprofessional rehabilitation hospital. Healthc Q. 2006, 9: 66-74.

    Article  PubMed  Google Scholar 

  27. Canadian Institute for Health Information: Understanding Emergency Department wait times: How long do people spend in Emergency Departments in Ontario? . Ottawa. 2007

    Google Scholar 

  28. Cetron MJ: Technological forecasting. 1969, New York: Gordon and Breach

    Google Scholar 

  29. Alahlafi A, Burge S: What should undergraduate medical students know about psoriasis? Involving patients in curriculum development: modified-delphi technique. BMJ. 2005, 19 (330): 633-636. 10.1136/bmj.330.7492.633.

    Article  Google Scholar 

  30. Campbell NC, Murray E, Darbyshire J, Emery J, Farmer A, Griffiths F, Guthrie B, Lester H, Wilson P, Kinmonth AL: Designing and evaluating complex interventions to improve health care. BMJ. 2007, 334: 455-459. 10.1136/bmj.39108.379965.BE.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Helfrich CD, Weiner BJ, McKinney MM, Minasian L: Determinants of implementation effectiveness: adapting a framework for complex innovations. Med Care Res Rev. 2007, 64: 279-303. 10.1177/1077558707299887.

    Article  PubMed  Google Scholar 

  32. Jansen KJ: From Persistence to Pursuit: A Longitudional Examination of Momentum During the Early Stages of Strategic Change. Organization Science. 2004, 276-294. 10.1287/orsc.1040.0064.

    Google Scholar 

  33. Narayan A, Steele-Johnson D, Delgado KM, Cole PA: Differential effects of pretraining influences on readiness to change. J Psychol. 2007, 141: 47-60.

    Article  PubMed  Google Scholar 

  34. Walston SL, Burns LR, Kimberly JR: Does reengineering really work? An examination of the context and outcomes of hospital reengineering initiatives. Health Serv Res. 2000, 34: 1363-1388.

    CAS  PubMed  PubMed Central  Google Scholar 

  35. Suter E, Hyman M, Oelke N: Measuring key integration outcomes: a case study of a large urban health center. Health Care Manage Rev. 2007, 32: 226-235.

    Article  PubMed  Google Scholar 

  36. Macleod H, Bell RS, Deane K, Baker C: Creating sustained improvements in patient access and flow: experiences from three Ontario healthcare institutions. Healthc Q. 2008, 11: 38-49.

    Article  PubMed  Google Scholar 

  37. Bradley EH, Holmboe ES, Mattera JA, Roumanis SA, Radford MJ, Krumholz HM: A qualitative study of increasing beta-blocker use after myocardial infarction: Why do some hospitals succeed?. JAMA. 2001, 285: 2604-2611. 10.1001/jama.285.20.2604.

    Article  CAS  PubMed  Google Scholar 

  38. Bradley EH, Curry LA, Webster TR, Mattera JA, Roumanis SA, Radford MJ, McNamara RL, Barton BA, Berg DN, Krumholz HM: Achieving rapid door-to-balloon times: how top hospitals improve complex clinical systems. Circulation. 2006, 113: 1079-1085. 10.1161/CIRCULATIONAHA.105.590133.

    Article  PubMed  Google Scholar 

  39. Gustafson DH, Sainfort F, Eichler M, Adams L, Bisognano M, Steudel H: Developing and testing a model to predict outcomes of organizational change. Health Serv Res. 2003, 38: 751-776. 10.1111/1475-6773.00143.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Ployhart RE, Holtz BC, Bliese PD: Longitudinal data analysis: Applications of random coefficient modelling to leadership research. Leadership Quarterly. 2002, 13: 455-486. 10.1016/S1048-9843(02)00122-4.

    Article  Google Scholar 

  41. Ven Van De AH, Huber GP: Longitudinal Field Research Methods for Studying Processes of Organziational Change. Organization Science. 1990, 1: 213-219. 10.1287/orsc.1.3.213.

    Article  Google Scholar 

  42. Pettigrew AM, Woodman RW, Cameron KS: Studying Organizational Change and Development: Challenges for Future Research. The Academy of Management Journal. 2001, 44: 697-713. 10.2307/3069411.

    Article  Google Scholar 

  43. Bradley EH, Holmboe ES, Mattera JA, Roumanis SA, Radford MJ, Krumholz HM: A Qualitative Study of Increasing B-Blocker Use After Myocardial Infarction: Why Do Some Hospitals Succeed?. JAMA. 2008, 285: 2064-2611.

    Google Scholar 

  44. Crump B: Should we use large scale healthcare interventions without clear evidence that benefits outweigh costs and harms? Yes. BMJ. 2008, 336: 1276-10.1136/bmj.a145.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Landefeld CS, Shojania KG, Auerbach AD: Should we use large scale healthcare interventions without clear evidence that benefits outweigh costs and harms? No. BMJ. 2008, 336: 1277-10.1136/bmj.a144.

    Article  PubMed  PubMed Central  Google Scholar 

Download references


The following individuals provided invaluable expertise, guidance and contribution to the selection of measures: Bonnie Adamson; Mark Afilalo, MD; Carolyn Baker; Christopher Baggogley, PhD; Debra Carew; Michael Carter, PhD; Matthew Cooke, MD, PhD; Ken Deane; Ken Gardener, MD; Bob Kocher; Paul Mango; Amit Nigam, PhD; Anne Sales, BScN, PhD; and Heather Sharard.

The Ontario Ministry of Health and Long-term Care (MOHLTC), Canadian Health Services Research Foundation (CHSRF); and The Canadian Institutes for Health Research (CIHR) provided support for this study and preparation of this manuscript. The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred. Partners at the MOHLTC collaborated with the research team on the study design, and participated in the expert panel review process to select minimization factors.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Michael J Schull.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

MS, AG, MZ, GA, TS, BG, BR, RB, DM and HA conceived of the study and design to systematically identify minimization factors, participated in the expert panel review process; and helped to draft the manuscript. CL carried out the literature review, coordinated and synthesized results from the panelist surveys; and drafted the manuscript. MS facilitated the teleconference. All authors read and approved the final manuscript.

Electronic supplementary material


Additional file 1: Candidate factors by change stages. Table lists 33 candidate factors by organizational change stages that the expert panel assessed across specified domains. (PDF 23 KB)

Rights and permissions

Open Access This article is published under license to BioMed Central Ltd. This is an Open Access article is distributed under the terms of the Creative Commons Attribution License ( ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Leaver, C.A., Guttmann, A., Zwarenstein, M. et al. Development of a minimization instrument for allocation of a hospital-level performance improvement intervention to reduce waiting times in Ontario emergency departments. Implementation Sci 4, 32 (2009).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: