Very few studies in the literature meet the five inclusion requirements for this review, but those that did represented diverse applications of transformation strategies. While as individual studies none were particularly generalizable, the diverse settings and interventions of Six Sigma and Lean suggest, at least, these strategies are frequently employed in healthcare. The broad applicability of Six Sigma is similar to the wide applications of other statistical process controls , and the ability of each to be adapted to new settings should facilitate their rapid adoption . As already noted, the study with the least concerns over external validity was the evaluation of the StuderGroup intervention. In addition, each of the reviewed studies concluded the respective interventions were effective, and more than one provided estimates of cost savings. For Lean and Six Sigma the effectiveness conclusion agrees with prior research in the manufacturing area. While a handful of the studies were methodologically stronger than others, all of the studies reviewed had significant threats to validity and were unable to rule out all alternative hypotheses. One might take some satisfaction from the fact that all of these studies attributed successes to the implementation of the various strategies. Unfortunately, the universally reported effectiveness of each strategy may also reflect a positive result publication bias .
Two immediate recommendations for research in transformation strategies suggested by this review are improvements in research methodologies and expansion of timeframes. Nearly all of the reviewed studies could be improved dramatically through more sophisticated statistical analysis or the addition of a comparison group. Large healthcare systems with multiple hospitals could execute stronger study designs with minimal additional effort, e.g., a phase-in of interventions would allow later implementer sites to serve as controls for early implementer sites. Alternatively, if a comparison group is not readily feasible, the very nature of these interventions facilitates interrupted time series designs, as was reported in two of the studies. A well-executed time series design not only has stronger validity claims, but also allows for the examination of a sustained effect . This latter design by nature encourages a longer time period for examination of effects. Kotter suggested organizational transformation as a process requires five to ten years to be fully realized . If this long view of evaluation research is taken, necessarily intermediate measures of process increase in importance and relevancy. Also, the longer time period can offer additional evidence of sustainability.
Creative evaluation models are possible, too, in large systems where multiple transformational strategies and units of analysis are in play. Scalability of evaluation may increase, i.e., be scaled up, division-wide and organization-wide, to aggregate impacts and interactions of multiple interventions. Alternatively, the evaluation may be scaled down to identify changes attributable to a specific intervention at smaller units. These methodological improvements could be facilitated with academic partnerships or through research trained administrators because industrial engineering departments are no longer widespread in hospitals .
While suggesting this avenue to improvement, we are aware that funding for evaluation and management research is not a priority for many health organizations. Again, however, this re-emphasizes the point for improved research studies in order to demonstrate the value of these strategies. The obvious potential for cost-savings or reductions were implied by the improvements in almost all of the reported studies, however, only a couple specifically indicated how much money was either saved  or how revenues were increased . Justification for evaluation and research is made easier when expected savings are available to offset those costs and those savings are expected to be ongoing. Still other opportunities exist for improved partnering between health services researchers and practicing organizations. Academic medical centers represent innovative institutions with a history and expectation of research, thereby appearing to be natural settings for these types of investigations. Evaluation of these and other transformative strategies may be slightly different than historical interest in clinical applications, but through academic contacts industrial/system engineers are more accessible and the culture is still one of research. Additionally, those seeking executive health management degrees, student interns, or even professionals returning to school for advanced degrees while still employed all provide opportunities and interested individuals for collaboration.
Our interest is in gaining the maximum impact from the various strategies, a situation which is most likely to occur if some degree of fidelity is maintained in implementation. We are not suggesting that there is no value from less rigorous evaluation models, or even that useful insights cannot be derived from heuristically impressive results reported in other formats. But real understanding of 'what, how, and why' of what worked (or didn't), is unlikely to occur without more exacting research and evaluation standards. That is, evaluation strategies may benefit from a realistic perspective that seeks to better inform practitioners of the applied value of these efforts . Given the substantial costs associated with these transformation strategies, healthcare managers seeking to adopt any strategy would be better served by demanding more exacting evaluation of the projects from their staff or consultants, or even better, include outside evaluators within the project budget. Organizational learning, like all learning, is based upon both action and reflection. Minimally evaluated innovations may still be successfully replicated in the same setting because of unspoken shared understandings; but chances of it working again at another site within the system or elsewhere may be very limited.
Returning to the conceptualization presented in Table 1, we suggested that transformation requires both changes in practice and culture. While all of three of the examined transformations advocate a cultural change, few of the reviewed studies examined indicators resembling organizational culture. The Lean patient alert system intervention provided limited data on culture in the form of patient safety culture, and the Six Sigma programs on surgery turnaround time and hand hygiene compliance reported staff satisfaction. However none of these studies, or the anecdotal evidence reported in other studies fully captures the multidimensional construct of organizational culture, leaving valid questions on these interventions' interaction with and affect on organizational culture unanswered.
The role of an organization's culture is not only important to safe healthcare delivery; it serves also as a precursor to other innovations . A review of TQM applications to hospitals revealed the innovation frequently faces an adverse culture, and managers incorrectly assumed employees would automatically adhere to the new philosophy . Specifically speaking about healthcare, Kovner and Rundall noted, '...efforts to introduce evidence-based decision making quickly wither and fade away because the organizational culture does not support evidence-based management' .
Lacking in the articles reviewed here, and maybe in their larger respective evaluations, is the extent to which such transformations are sustainable, and the extent to which the knowledge, attitudes, and skills developed from the transformation are retained and transferred to other problems and parts of the healthcare organization. The two exceptions to the question of sustainability are Furman and Caplan's report on the safety alert system at Virginia Mason Medical Center , which included more than four years of post-implementation observations, and Shannon and colleagues' nearly three-year study . Some of the other reviewed studies reported measurements at one to two years post-implementation [23, 27, 28, 33–36], but the rest were on much shorter timelines of a few months, reflecting the narrowly focused application of these strategies. Based upon the anticipated timeframe for transformation, noted above, it would be difficult to see or even expect widespread organizational transformation within these windows.
In addition, multiple transformation strategies can be implemented in concert. The integration of strategies was evident in this review. For example, Napoles and Quintana record consultant's Lean training program included Six Sigma instruction , and others noted how more than one transformative strategies was already in place within their organizations [22, 26, 29, 30]. Likewise, while a predominately a cultural change strategy, StuderGroup emphasizes measurement and therefore efficiency change. The potential for interactions, synergies, appropriate sequencing, or even conflicts between different strategies raises practical questions amenable both to theoretical examination and empirical testing.
As stated above, this review was not exhaustive of all transformational interventions available to healthcare leaders. We did not examine TQM or CQI, as those have been the subject of previous reviews , or the additional healthcare specific strategies like application of the Malcolm Baldrige National Quality Award framework, LeapFrog Group initiatives or Institute for Healthcare Improvement programs. A similar critical review of these later strategies, particularly compared to the finding presented in this article, might prove to be particularly informative. Similarly, while our review was broad, it did not include the grey literature; as we stated we would not dismiss the potential for valuable insights from impressive results reported in other formats, but that area of reporting was not our main interest. Nor did our search strategy allow for the inclusion of studies involving individual components or particular methods of the above strategies conducted without their Six Sigma, Lean, or StuderGroup nameplates. As noted above, these strategies and approaches have roots in other disciplines and draw on other approaches and concepts, particularly the statistical control aspects, which have certainly been examined independently. However, our interest is in these proposed transformation strategies as complete packages, as that is how they are currently proposed to healthcare organizations.
Health systems are continually innovating. Required are transformational changes that fundamentally alter practices and culture for immediate improvements in care and ever increasing capacity for continuing improvement. Improving evaluation and understanding of the implementation and outcomes of such changes are essential to sustaining ongoing transformation and restricting any legacy of failure. The healthcare literature needs more reports of rigorous examinations of these transformation efforts and ongoing dialogue between the research and practice community addressing this critical topic.