This article has Open Peer Review reports available.
The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change
© Chambers et al.; licensee BioMed Central Ltd. 2013
Received: 2 April 2013
Accepted: 20 September 2013
Published: 2 October 2013
Despite growth in implementation research, limited scientific attention has focused on understanding and improving sustainability of health interventions. Models of sustainability have been evolving to reflect challenges in the fit between intervention and context.
We examine the development of concepts of sustainability, and respond to two frequent assumptions —'voltage drop,’ whereby interventions are expected to yield lower benefits as they move from efficacy to effectiveness to implementation and sustainability, and 'program drift,’ whereby deviation from manualized protocols is assumed to decrease benefit. We posit that these assumptions limit opportunities to improve care, and instead argue for understanding the changing context of healthcare to continuously refine and improve interventions as they are sustained. Sustainability has evolved from being considered as the endgame of a translational research process to a suggested 'adaptation phase’ that integrates and institutionalizes interventions within local organizational and cultural contexts. These recent approaches locate sustainability in the implementation phase of knowledge transfer, but still do not address intervention improvement as a central theme. We propose a Dynamic Sustainability Framework that involves: continued learning and problem solving, ongoing adaptation of interventions with a primary focus on fit between interventions and multi-level contexts, and expectations for ongoing improvement as opposed to diminishing outcomes over time.
A Dynamic Sustainability Framework provides a foundation for research, policy and practice that supports development and testing of falsifiable hypotheses and continued learning to advance the implementation, transportability and impact of health services research.
As implementation science has grown [1, 2], researchers have advanced from study of facilitators and barriers that influence uptake of effective programs and policies to investigations of strategies to improve uptake. However, often studies evaluate only initial intervention adoption and implementation. Sustained practice change and broader scale-up of interventions  rarely are investigated, often due to the constrained timeframes for research that are set by grant mechanisms, and the budgetary and political necessity of many decision-makers to take on a short-term lens.
Recently, there has been interest in understanding and influencing the sustainability of implemented interventions. While this is progress, frequently used conceptualizations of sustainability implicitly replicate assumptions and limitations inherent in the traditional research-to-practice pathway [4, 5], or in its more recent conceptualization as translational research . These conceptualizations of knowledge translation often assume that interventions are optimized prior to implementation, and that they are largely independent of the context in which they are delivered .
The presumed linear model of intervention development, efficacy testing and implementation has resulted in the development of an armamentarium of efficacious healthcare treatments, preventive strategies, and public health interventions. While these discoveries have made advances in a number of health domains, they are often difficult to implement in a myriad of practice settings and even harder to sustain over time and in many real world and low-resource settings . In addition, interventions are traditionally expected to perform worse in real-world practice than in the laboratory or the rarified clinical trial setting. We argue for a new approach to sustainability that instead integrates the themes of adaptive, contextually sensitive continuous quality improvement (CQI) and a learning healthcare system with the challenge of intervention sustainment.
Evolving understandings of sustainability and of related concepts of CQI and the learning healthcare system;
An iterative, dynamic approach to sustainability, termed the 'Dynamic Sustainability Framework’ (DSF) that integrates and extends these concepts; and
Implications of this framework for research, policy, and practice.
Definitions of key terms used in this paper
The process of putting to use or integrating evidence-based interventions within a setting .
To what extent an evidence-based intervention can deliver its intended benefits over an extended period of time after external support from the donor agency is terminated .
The continued use of an intervention within practice .
The phenomenon in which interventions are expected to yield lower benefits as they move from efficacy to effectiveness and into real world use (adapted from ).
The phenomenon whereby deviation from manualized protocols in real-world delivery of interventions is expected to yield decreasing benefit for patients (adapted from ).
Moving beyond 'voltage drop’ and 'program drift
While the traditional linear process of intervention development, derived from pharmaceutical medication development models , has often resulted in the creation of initially successful interventions, it may be less helpful in enabling these innovations to maximally benefit health. A linear approach may be particularly challenging to apply to complex, multi-component interventions, psychosocial treatments, treatment of the growing number of people with multimorbid conditions [14, 15], and systemic approaches to care . Linear approaches place a premium on creating and 'freezing’ an intervention, developing manuals to ensure its consistent delivery with fidelity , and then minimizing deviations from the intervention. For example, the COMMIT stop smoking national project had expert researchers design the complex, lengthy intervention protocol based on research evidence, and then gave a several hundred page manual of operations to local staff to implement in their communities .
The importance of internal validity to the scientific process should not be ignored, but its overemphasis relative to generalizability and adaptation runs the risk of creating interventions that will not fit within different, complex or changing settings and of failing to benefit settings, clinicians, and patient populations who are underrepresented in the intervention testing process [7, 16]. Two key implicit assumptions within the traditional intervention development approach may limit ultimate progress toward intervention sustainability and population impact:
Second, the assumption that interventions can be optimally constructed in the early stages of the development and testing process, independent of context, suggests that, even at the stages of implementation and sustainability, change to the intervention is expected to have negative consequences, and that the further a practitioner deviates from the manual, the lower the benefit. This is the concept of 'program drift’ (Figure 1). Delivering the intervention within an efficacy trial may require adherence to protocols that are challenging to deliver within real-world practice. Fidelity ratings then assume that 100% fidelity to original protocols will yield optimal outcomes, and effort is expended to ensure that practitioners do not deviate from the manual. Where clinicians do deviate from the protocol, the field expects that the resulting 'program drift’ will compromise outcomes . We see that this over-reliance on quality assurance to prevent 'program drift’ leads to extensive pressure on real-world practices to adhere to the intervention protocols without evidence that this adherence will lead to optimal outcomes. Quality assurance may inadvertently hamper sustainability and ongoing improvement, customization and optimization of interventions to the detriment of population health.
In contrast, we reject the notion that an intervention can be optimized prior to implementation, and explicitly reject the validity of 'program drift’ and 'voltage drop.’ Rather, we suggest that the most compelling evidence on the maximal benefit of any intervention can only be realized through ongoing development, evaluation and refinement in diverse populations and systems . Instead of viewing contextual factors as interfering with the delivery of an effective intervention and needing to be controlled, we see the opportunity to learn about the optimal fit of an intervention to different care settings . For example, strategies have been developed to adjust organizational characteristics (e.g., culture, climate, structure) to enable improved fit between the intervention and the setting ; harnessing the understanding of context can enable beneficial adaptation of the intervention and improve sustainability. Without rejecting these assumptions, we reify early phase interventions tested in the most artificial settings, set quality assurance of interventions as an optimal outcome, and miss opportunities for continued learning and development.
Understanding and advancing sustainability research
As the field of implementation science has matured [20, 21], more emphasis has been placed on understanding sustainability. Researchers have recognized that implementation of interventions, which can often require substantial resources, is meaningless without successful long-term use. Following Rabin et al.’s glossary for Dissemination and Implementation Research in Health , we draw distinctions between 'implementation,’ which relates to the initial process of embedding interventions within settings and 'sustainability,’ which relates to the extent that these interventions can continue to be delivered over time, institutionalized within settings, and have necessary capacity built to support their delivery.
Recent articles [22–24] have advanced the idea of an adaptation phase that bridges from the initial implementation effort to a longer-term sustainability phase. They argue the need to examine the fit between the practice setting and the intervention and make changes necessary to improve the integration of the intervention into ongoing care processes. This is consistent with the institutional theory of organizations, which argues that the final stage of innovation requires the 'institutionalizing’ of the new practice so that it becomes a working part of the organization .
As a consequence, assessment of organizational characteristics (e.g., structure, climate, culture, resources) is seen as an essential component of sustainability, and indeed, the fit between context and the intervention is at the center of a sustainability phase . There has also been an emphasis on planning for sustainability much earlier in the intervention process . Recent approaches to sustainability locate key efforts squarely in the implementation phase, arguing that once a practice has been implemented within a care system, those who manage the delivery of that practice should turn their attention to ensuring that the practice can be maintained over time [6, 24]. Authors typically suggest that this entails attention to issues of long-term financing, training of the workforce, supervision, and organizational support for the practice [26, 27]. A characteristic of this approach is to postpone emphasis on sustainability until after implementation is well underway, assuming that implementation and sustainability are sufficiently independent.
Authors have also highlighted the utility of assessing outcomes of those who have received the practice, something infrequently collected in routine practice [24, 28]. Seldom is it demonstrated that continued delivery of an intervention confers benefit on the patient population that receives the intervention or the system that delivers it (e.g., cost containment, efficiency of care, quality metrics). Measurement of outcomes over time to determine continued benefit has been shown to support sustainability of the practice [29, 30].
Recent implementation projects have created new tools and scales to study sustainability, including needs assessments, long-term action plans, tracking of program adaptation, financial planning, mapping of community networks, and measurement of the degree to which practices are integrated and institutionalized into service systems [31–33]. While this emerging focus on sustainability is an advance, many studies still assume a largely static service delivery system that needs to be assessed only at key time points, but not in an ongoing manner. To better reflect complexity and change within the system and in context, a more dynamic approach to sustainability is needed.
The dynamic sustainability framework (DSF)
The DSF, like many implementation models, centers on a few major elements: the intervention, the context in which the intervention is delivered, and the broader ecological system within which the practice settings exist and operate. Distinct from those models, however, is the consideration of these elements over time. The intervention, as shown in the figure, often includes a set of individual components chosen for their ability to effect behavior or biochemical change, an assumed set of characteristics defining who should deliver the intervention, targeted, patient-centered outcomes that the intervention should generate as a result of its use, and a delivery platform (e.g., face-to-face, telephonic, web-based, mobile health app, etc.). Other constructs may also define the intervention.
The DSF anchors the ultimate benefit of the intervention in terms of its ability to fit within a practice setting, typically a clinical or community setting. This context carries its own set of characteristics, including human and capital resources, information systems, organizational culture, climate and structure, and processes for training and supervision of staff. The DSF, consistent with other models, argues that these practice characteristics will directly influence the ability of the intervention to reach the patient population that could benefit, and thus measurement of these contextual constructs is paramount to resolving fit.
At a third level, the DSF identifies the ecological system as an additional driver of the successful implementation and sustainability of an intervention. The ecological system consists of many practice settings that influence those working to incorporate a particular intervention, as well as the legislative and regulatory environment, characteristics of local, regional, state and national markets, and characteristics of the broad population. The ecological system is influenced by changes to available interventions and practice settings, and in turn, influences them.
Specific to the DSF, as emphasized by the dotted lines in Figure 2, is the expectation that change is constant at each of these levels (and ripples across multiple levels), and thus the success of an intervention to be sustained over time lies in the measured, negotiated, and reciprocal fit of an intervention within a practice setting and the practice setting within the larger ecological system. The DSF suggests that optimal fit requires that characteristics of the intervention, practice setting, and ecological system be consistently tracked, using valid, reliable and relevant measures, and expects that interventions, settings and the ecological system should change over time, particularly where data can suggest improvements for each to better meet the needs of patients, the skills and resources within the practice setting, and the larger ecology.
The DSF is intended to suggest a new paradigm to consider the long-term use and ongoing improvement of interventions, recognizing the limitations of the evidence base made available through efficacy and effectiveness trials, and allowing that continuous exposure of the intervention to new populations, new contexts, and new innovations can result in continued improvement of resulting outcomes, thus minimizing the perils of 'program drift’ and 'voltage drop.’ Indeed, the DSF posits that ongoing quality improvement of interventions is the ultimate aim, not quality assurance of them. To be clear, we see value in ensuring appropriate quality assurance within healthcare systems where clear assessments of an appropriate standard of care are made through knowledge of core intervention components. However, the DSF recognizes the limitations of intervention evidence solely from clinical trials and argues that quality improvement processes focused on intervention optimization are ultimately more relevant to achieve sustainment.
The DSF, which has benefitted from the authors’ ongoing dialogue with the Implementation Science community about the challenge of sustainability, follows the spirit of a number of existing models that emphasize three things—importance of context, the need for ongoing evaluation and decision-making, and the goal of continuous improvement. These include Wandersman’s Getting to Outcomes model , Continuous Quality Improvement (CQI) , system dynamics , complexity theory , adaptive management , and the Evidence Integration Triangle . In addition, the DSF is consistent with alternative views of organizational development  and the principles of system science . Distinct in the DSF from many of these other models is the emphasis on omnipresent change, and the central goal of continuously optimizing the fit between the intervention and a dynamic delivery context to achieve maximal benefit. The DSF is anchored around the following seven tenets, for which we think there is evidence, but recommend explicit testing in this context:
An intervention should not be optimized prior to implementation, or even prior to 'sustainability phase’ onset
Interventions benefit from ongoing optimization as they are applied in different contexts . The evidence that supports the benefits of health interventions arises from trials that represent a very small slice of the diversity of demographics, preferences, and health status of the population at large [7, 40], but we should not expect evidence collected in one set of narrow, relatively optimal circumstances to apply perfectly in other, vastly different contexts . A 'corollary’ of this recommendation is that, other things being equal, quality improvement approaches that involve adjusting and refining program should be more effective than 'quality assurance’ procedures that emphasize fidelity to an initial protocol.
Interventions can be continually improved, boosting sustainment in practice, and can enable ongoing learning among developers, interventionists, researchers and patients
There is tremendous opportunity to aggregate evidence on the real-world impact of interventions when used in practice. We can apply models of continual refinement that have been the cornerstone of software development (e.g., Firefox, Reaper, iTunes), as well as Web 2.0 sites (e.g., Wikipedia, Facebook, etc.) . By augmenting trial data with practice-based evidence, we can understand much more about what works for whom, the question underlying personalized medicine [43, 44]. This articulation of the DSF suggests the need for a long-term plan to commit resources for training and ongoing improvement. One implication of the DSF is that intervention impact can also be enhanced through increases in efficiency. The field has developed a plethora of multi-component interventions, often without studies that determine what the minimal set of components are needed to ensure benefit . The DSF, congruent with the Consolidated Framework for Implementation Research , emphasizes the importance of streamlining interventions to peel away components that may not be central to improving outcomes or to adapt intervention components to a particular context.
Ongoing feedback on interventions should use practical, relevant measures of progress and relevance
Too often, intervention trials focus on markers that are psychometrically valid but of less relevance to patients and clinicians [3, 47]. For example, very specific, intervention-related symptomatic scales may be most sensitive to change, but there is little guarantee that the measured change translates into a tangible, functional benefit for the patient . The DSF thus suggests the use of measures such as checklists that are relevant to desired outcomes of patients, as well as sensitive to the 'fit’ between interventions and context, and can be feasibly implemented . Across each of the changes in Figure 2, we see available streams of data that can offer leverage points for improving interventions. Environmental changes, for example, can be tracked via population surveys, and market and claims data. Practice changes can be captured through electronic health records, claims data and practice surveys. Evidence reviews can provide key information on knowledge changes, and policy changes can be tracked via available Federal and Non-Profit sources (e.g., CMS, Kaiser Family Foundation, etc.). This is consistent with the CQI model, but with specific emphasis on the ongoing refinement of the intervention to counteract the assumptions of 'Program Drift.’
Voltage drop is NOT inevitable
We reject the assumption that the more diverse and complex a patient population is, the smaller the benefit of intervention, referred to above as 'voltage drop’ . This stems from an expectation that intervention studies require control of the environment to isolate a treatment effect . If we embrace CQI of the specific health intervention, we expect that with more experience, we will better be able to adapt interventions to contexts and patients. As we learn more about what works and what doesn’t and adjust protocols accordingly, the 'voltage’ could maintain or possibly increase over time. This echoes the computer industry, where each new release of a hardware or software line is expected to be better than the prior version. It also finds consonance with the evolution of the flu vaccine, which is constantly refined in response to the changing nature of the influenza virus each season. A culture of improvement is central to ongoing intervention use and treats improvement of the intervention as central to the sustainability process .
Programs should be more likely to be maintained when there is strong 'fit’ between the program and the implementation setting
The concept of 'fit’ has been discussed by other authors (e.g., Estabrooks, Glasgow, Dzewaltowski),  and goes back at least as far as Rogers’ Diffusion of Innovations, where the concept of reinvention evoked the notion of departing from the original intervention concept to 'create’ a new version suited to the preferences and constraints of the local context. Fit is a multi-level construct and involves alignment along multiple dimensions . The DSF posits that fit will likely change over time, due to changes in the way in which an intervention is delivered, the characteristics of patients, providers and settings, and the broader ecological system within which healthcare settings reside. Attention to this fit, through ongoing assessment and quality improvement efforts, should improve sustainment and ultimately identify opportunities for intervention improvement.
Organizational learning should be a core value of the implementation setting
While training and ongoing analyses exist in many organizations, the demands imposed by multiple levels of change require learning to be central to organizational activity for interventions to become sustainable. The context both within an organization and in the broader ecological system is constantly changing, and requires a 'learning organization’  to engage in problem-solving capacity at multiple levels . Organizational learning should also target appropriate adaptation of evidence-based interventions, possibly in rapid learning cycles [44, 54], followed by ongoing assessment and feedback loops. The DSF is congruent with the concepts of the learning healthcare system, again with the emphasis on learning how to better develop, deliver and sustain interventions.
Ongoing stakeholder involvement throughout should lead to better sustainability
Continuously engaging stakeholders throughout the planning, implementation and adaption processes should help increase the fit between the intervention and the local context, and help address evolving issues that might interfere with sustainability. Just as researchers have proposed intervention development processes  that focus on the ultimate site where interventions will be delivered, we argue that partnership among all relevant stakeholders is essential to maintaining and improving interventions within care settings.
Contrasting static and dynamic views of sustainability
Contrasting static views of sustainability with the dynamic sustainability framework
Dynamic sustainability view
Inevitable; encouraged, monitored and guided by evidence
Initial or during implementation
During study by researchers
Incorporated as part of organization
Review of evidence
Initial- from efficacy studies
Ongoing; from convergent sources including replications
Staffing issues (e.g., turnover) and variations
Planned for; investigated
Generates new knowledge
Yes, feedback to other areas of science and to earlier stages
Implications of the DSF
This initial formulation of the DSF has implications for future practice, research and policy. For practice, the DSF highlights the need for continuous assessment of the local context, not just prior to implementation. This enables care settings to better manage the fit between their resources, needs and the interventions, including generating consistent feedback on how interventions are delivered to diverse patients and how patients do as a result. The collection and analysis of this information allows practitioners to make informed decisions about how best to utilize existing interventions, allows for potential enhancements to the interventions to be made and shared, and offers better information on which to make decisions to cease delivering interventions that do not have benefit. The intention is to recognize and support rapid learning, real-time problem-solving organizations  that are full partners in the generation of knowledge, not just its application. Thus, the DSF promotes the use of multiple methods of planning for sustainability, including simulation modeling of the impact of different decisions, pilot testing of adaptations within local contexts, and continued experimentation. Perhaps an even greater benefit to practice would come through pooling of data across a larger set of sites, practitioners and patients, something done with success for chronic disease [55, 56].
For research, the DSF dispels the notion that intervention development, refinement and improvement are completed prior to real world implementation. In contrast, we suggest that development and refinement is never complete. Rather, sustainability is the process of managing and supporting the evolution of an intervention within a changing context. We recommend (and welcome testing of the idea) that programs that monitor context and adjust accordingly do better long-term. In addition, we see research studies testing whether settings and programs using ongoing CQI or other means of feedback improvement perform better over time. More broadly, we see the DSF as changing the notion of a linear transition from research to practice into a shared process of continual experimentation and analysis through the use of both practice settings and ecological systems to track changes and assess evolving fit between interventions and practice settings. Principles of 'crowd sourcing’ made popular within the IT industry and resulting in open source products like Firefox, Wikipedia and Reaper, blur the lines between research and practice. We see this in the evolution of the electronic health record, and the rapidly evolving patient health record, which is dramatically changing the scenario from one where the medical expert makes all the decisions to one of collaborative care and shared decision-making [57, 58]. Instead of a small team of researchers developing a priori an 'optimal,’ static product, a large and often virtual community including users and consumers continuously upgrades dynamic products.
This initial formulation of the DSF also has implications for policy. Incentives are needed to support ongoing adaptation of interventions, particularly where evidence is limited, specifically including monitoring of progress and documentation of adaptations, using quality measures relevant to stakeholders and patients. In addition, research funders must determine how to support longer-term projects related to sustainability with flexible research designs, since the ultimate benefit of integrating and modifying interventions may not be evident for many years. In addition, infrastructure to support pooling of 'practice-based evidence’ will be needed, in order to ensure sufficient information is available about long-term use and adaptation of interventions. The DSF aligns directly with a number of existing policy initiatives, at national, state and local levels, including the advance of Patient-Centered Medical Homes, Accountable Care Organizations, Pay for Performance initiatives, and support for local demonstration projects.
Illustrative examples of the use of DSF for different types of interventions
Applying DSF principles
Clinical guidelines for pharmacotherapy
• Clinical guidelines for pharmacotherapy to treat a range of chronic diseases have been developed, implemented and refined as new compounds have reached the market and new evidence has been gathered about the relative benefit of different medications.
• The influences on prescribing practices exist at the patient level (preferences and predictors of response), clinician level (practice patterns, level of training), system level (formulary design, insurance coverage, adherence monitoring).
• Each influence will impact guideline implementation and overall benefit of care for patients served within the health system.
• Assessing appropriate fit between guidelines and the care setting will require analysis of multiple streams of data, including administrative, clinical, organizational and epidemiologic.
• The DSF suggests that collecting benchmarks over time on patient outcomes, adaptations in algorithms used, available evidence (from the literature, healthcare systems and patient populations), and contextual factors could result in improvements to the guidelines, to the capacity of the health system to more seamlessly integrate the guidelines, to the ecological system that could improve access, quality and health outcomes.
Psychotherapy for mood disorders
• Manualized evidence-based psychotherapies for mood disorders have been tested in numerous studies.
• Many of these therapies are designed to be delivered by specific providers, over a set number of sessions, with a clear step-by-step approach.
• Given variation in access to therapy (e.g., number of sessions covered, availability of therapists, time), limited predictive ability of response (how many sessions are needed, what are active ingredients, who should deliver therapy?), emergent options for mode of delivery (web-based, face-to-face, self-guided, asynchronous), optimizing psychotherapy for individuals and systems is still beyond our current knowledge base.
• The DSF suggests manualized psychotherapy could be improved by tracking variation in use and therapeutic response of patients, contextual characteristics that influence delivery, and additional interventions that affect clinical and functional outcomes.
• Systems could track how patients respond to varying doses of therapy, modes of delivery, and clinician characteristics. Over time, decision-makers could align available care to the needs of patient populations, and clinicians could adapt practice patterns to data on patient preferences and outcomes, and general needs of the patient population.
• By assessing the fit of psychotherapy delivery with patients, the service setting and the broader ecological system, the DSF hypothesizes that new insights about psychotherapy optimization could drive improvements in patient care.
Care management for chronic diseases
• Studies have shown the effectiveness of care management strategies to assess, intervene and monitor for a range of chronic conditions.
• Typical strategies involve initial screening, assessment, treatment planning, care and self-management strategies, and follow-up.
• While general care management approaches seem to be durable, specific approaches can have difficulty being implemented across many clinical and community settings, because of limitations in resources (both monetary and staffing), information systems, financing processes, and other barriers.
• With new technologies, additional evidence about treatment and preventive interventions, and reconfiguration of care systems, care management for chronic diseases can be sustained and improved in a large variety of care settings.
• Care management is influenced by drivers at patient, provider, organization and system levels.
• Care management requires coordination among multiple people, organizational supports and capital resources, all of which will likely shift over time.
• Therefore, care management cannot be sustained without continual assessment of fit within the local setting and the support of the components of the model.
The DSF hypothesizes that attention to local adaptations made by healthcare and community settings to fit the model; and feedback on staffing levels, intensity of care management, emerging interventions and patient outcomes could enhance long-term sustainability and model improvement.
• Evidence about who benefits most from different variants of care management, who is an ideal care manager, and what are the best ways of coordinating across primary care and specialty practice could lead to better uptake and improve patient health.
It is time to embrace the culture of a learning healthcare system [44, 59] to promote sustainability of interventions that are optimized and customized to the myriad of clinical and community settings. The enormous changes in health systems in the past few years give particular salience to a conceptualization of sustainability as a dynamic process, and provide unparalleled opportunity to test and refine the principles offered in this paper. Without this emphasis, we reify the past by asserting the primacy of evidence-based guidelines based on evidence from rarified clinical trial settings, and give second-class status to ongoing learning in real world settings. Allowing researchers to be the only ones allowed to generate new knowledge limits the opportunities to consistently improve the care we provide. In the past, these limitations were technological; we lacked the tools and the sources of information to drive rapid improvements. In this era of 'crowd sourcing,’ of exponentially-expanding processing power and global connectedness, we no longer need to adhere to a view that once created, interventions and healthcare settings must be 'frozen’ to optimize effectiveness. Instead, we propose the DSF as helping to reconfigure the research-practice-policy interface, in which the best possible information is gathered and used in real-time to inform policy, improve practice, and answer the highest priority research questions. Only then will the promise of a learning healthcare system be reached.
- Greenhalgh T, Robert G, Macfarlane F: Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004, 82 (4): 581-629. 10.1111/j.0887-378X.2004.00325.x.View ArticlePubMedPubMed CentralGoogle Scholar
- Glasgow RE, Chambers D, Developing Robust, Sustainable: Implementation systems using rigorous, rapid, and relevant science. Clin Transl Sci. 2012, 5 (1): 48-55. 10.1111/j.1752-8062.2011.00383.x.View ArticlePubMedGoogle Scholar
- Haines A, Kuruvilla S, Borchert M: Bridging the implementation gap between knowledge and action for health. Bull World Health Organ. 2004, 82 (10): 724-732.PubMedPubMed CentralGoogle Scholar
- Flay BR: Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Prev Med. 1986, 15: 451-474. 10.1016/0091-7435(86)90024-1.View ArticlePubMedGoogle Scholar
- Greenwald P, Cullen JW: The new emphasis in cancer control. J Natl Cancer Inst. 1985, 74: 543-551.PubMedGoogle Scholar
- Stirman SW, Kimberly J, Cook N: The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012, 7: 17-10.1186/1748-5908-7-17.View ArticleGoogle Scholar
- Rounsaville BJ, Carroll KM, Onken LS: A stage model of behavioral therapies research: getting started and moving on from stage 1. Clin Psychol. 2001, 8 (2): 133-142.Google Scholar
- Glasgow RE, Vinson C, Chambers D, Khoury MJ, Kaplan RM, Hunter C: National institutes of health approaches to dissemination and implementation science: current and future directions. Am J Public Health. 2012, 102 (7): 1274-1281. 10.2105/AJPH.2012.300755.View ArticlePubMedPubMed CentralGoogle Scholar
- Rabin BA, Brownson RC, Haire J, Kreuter MW, Weaver NL: A glossary for dissemination and implementation research in health. Public Health Manag Pract. 2008, 14 (2): 117-123. 10.1097/01.PHH.0000311888.06252.bb.View ArticleGoogle Scholar
- Aarons GA, Hurlburt M, Horwitz SM: Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011, 38: 4-23. 10.1007/s10488-010-0327-7.View ArticlePubMedGoogle Scholar
- Kilbourne AM, Neumann MS, Pincus HA: Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implement Sci. 2007, 2: 42-10.1186/1748-5908-2-42.View ArticlePubMedPubMed CentralGoogle Scholar
- Weisz JR, Kazdin AE: Evidence-based psychotherapies for children and adolescents. 2010, New York: Guildford PressGoogle Scholar
- Kessler R, Glasgow RE: A proposal to speed translation of healthcare research into practice: dramatic change is needed. Am J Prev Med. 2011, 40 (6): 637-644. 10.1016/j.amepre.2011.02.023.View ArticlePubMedGoogle Scholar
- Kaitin KI: Deconstructing the drug development process: the new face of innovation. Clin Pharmacol Ther. 2010, 87 (3): 356-361. 10.1038/clpt.2009.293.View ArticlePubMedPubMed CentralGoogle Scholar
- Craig P, Dieppe P, Macintyre S: Developing and evaluating complex interventions: the new medical research council guidance. BMJ. 2008, 337: a1655-10.1136/bmj.a1655.View ArticlePubMedPubMed CentralGoogle Scholar
- Prochaska JO, Evers KE, Prochaska JM: Efficacy and effectiveness trials: examples from smoking cessation and bullying prevention. J Health Psychol. 2007, 112 (1): 170-178.View ArticleGoogle Scholar
- COMMIT Research Group: Community intervention trial for smoking cessation (COMMIT): summary of design and intervention. J Natl Cancer Inst. 1991, 83: 1620-1628.View ArticleGoogle Scholar
- Cohen DJ, Crabtree BF, Etz RS: Fidelity versus flexibility: translating evidence-base research into practice. Am J Prev Med. 2008, 35: S381-S389. 10.1016/j.amepre.2008.08.005.View ArticlePubMedGoogle Scholar
- Glisson C, Schoenwald SK, Hemmelgarn A: Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. J Consult Clin Psychol. 2010, 78 (4): 537-550.View ArticlePubMedPubMed CentralGoogle Scholar
- Brownson RC, Colditz GA, Proctor EK: Dissemination and Implementation Research in Health. 2012, New York: OUPGoogle Scholar
- Chambers DA: Advancing sustainability research: challenging existing paradigms. Public Health Dent. 2011, 1 (71): S99-100.View ArticleGoogle Scholar
- Shediac-Rizkallah MC, Bone LR: Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. 1998, 13 (1): 87-108. 10.1093/her/13.1.87.View ArticlePubMedGoogle Scholar
- Scheirer MA: Is sustainability possible? A review and commentary on empirical studies of program sustainability. Am J Eval. 2005, 26 (3): 320-347. 10.1177/1098214005278752.View ArticleGoogle Scholar
- Scheirer MA, Dearing JW: An agenda for research on the sustainability of public health programs. Am J Public Health. 2011, 101 (11): 2059-2067. 10.2105/AJPH.2011.300193.View ArticlePubMedPubMed CentralGoogle Scholar
- DiMaggio PJ, Powell WW: The iron cage revisited: institutional isomorphism and collective rationality in organizational fields. Am Sociol Rev. 1983, 48 (2): 147-160. 10.2307/2095101.View ArticleGoogle Scholar
- Ory MG, Lee SM, Mier N: The science of sustaining health behavior change: the health maintenance consortium. Am J Health Behav. 2010, 34 (6): 647-659.View ArticlePubMedPubMed CentralGoogle Scholar
- Swerissen H, Crisp BC: The sustainability of health promotion interventions for different levels of social organization. Health Promot Interv. 2004, 19 (1): 123-130. 10.1093/heapro/dah113.View ArticleGoogle Scholar
- Greenhalgh J, Meadows K: The effectiveness of the use of patient-based measures of health in routine practice in improving the process and outcomes of patient care: a literature review. J Eval Clin Pract. 1999, 5 (4): 401-416. 10.1046/j.1365-2753.1999.00209.x.View ArticlePubMedGoogle Scholar
- Duckers ML, Wagner C, Vos L: Understanding organisational development, sustainability, and diffusion of innovations within hospitals participating in a multilevel quality collaborative. Implementation science. 2011, 6 (1): 18-10.1186/1748-5908-6-18.View ArticlePubMedPubMed CentralGoogle Scholar
- Glasgow RE, Green LW, Taylor M: An Evidence integration triangle for aligning science with policy and practice. Am J Prev Med. 2012, 42 (6): 646-654. 10.1016/j.amepre.2012.02.016.View ArticlePubMedPubMed CentralGoogle Scholar
- Wandersman A, Imm P, Chinman M: Getting to Outcomes: Methods and tools for planning, evaluation and accountability. 1999, Rockville, MD: Center for Substance Abuse PreventionGoogle Scholar
- Estabrooks PA, Smith-Ray RL, Dzewaltoski DA: Sustainability of evidence-based community-based physical activity programs for older adults: lessons from Active for Life. Transl Behav Med. 2011, 1 (2): 1-8.View ArticleGoogle Scholar
- Wilson KD, Kurz RS: Bridging implementation and institutionalization within organizations: proposed employment of continuous quality improvement to further dissemination. J Public Health Manag Pract. 2008, 14 (2): 109-116. 10.1097/01.PHH.0000311887.06252.5f.View ArticlePubMedGoogle Scholar
- McLaughlin CP, Kaluzny AD: Continuous Quality Improvement in Health Care. 2004, Sudbury, MA: Jones and BartlettGoogle Scholar
- Homer JB, Hirsch GB: Systems dynamics modeling for public health: background and opportunities. Am J Public Health. 2006, 96 (3): 452-458. 10.2105/AJPH.2005.062059.View ArticlePubMedPubMed CentralGoogle Scholar
- Plsek PE, Greenhalgh T: The challenge of complexity in health care. BMJ. 2001, 323: 625-628. 10.1136/bmj.323.7313.625.View ArticlePubMedPubMed CentralGoogle Scholar
- Ebi K: Climate change and health risks: assessing and responding to them through 'adaptive management. Health Aff. 2011, 30 (5): 924-30. 10.1377/hlthaff.2011.0071.View ArticleGoogle Scholar
- Bushe GR, Marshak RJ: Revisioning organizational development: diagnostic and dialogic premises and patterns of practice. J Appl Behav Sci. 2009, 45 (3): 348-368. 10.1177/0021886309335070.View ArticleGoogle Scholar
- Checkland P: Soft systems methodology: a thirty year retrospective. Syst Res. 2000, 17: S11-S58. 10.1002/1099-1743(200011)17:1+<::AID-SRES374>3.0.CO;2-O.View ArticleGoogle Scholar
- Stange KC, Breslau ES, Dietrich AJ: State-of-the-art and future directions in multilevel interventions across the cancer control continuum. J Natl Cancer Inst Monogr. 2012, 44: 20-31.View ArticleGoogle Scholar
- Green LW: Making research relevant: if it is an evidence-based practice, where’s the practice-based evidence?. Fam Pract. 2008, 25 (Supp 1): i20-i24.View ArticlePubMedGoogle Scholar
- Bughlin J, Chui M, Johnson B: The McKinsey Quarterly. The next step in open innovation. 2012, http://www.mckinseyquarterly.com/next_step_in_open_innovation_2155. AccessedGoogle Scholar
- Khoury MJ, Gwinn M, Glasgow RE: Am J Prev Med. 2012, 42 (6): 639-45. 10.1016/j.amepre.2012.02.012.View ArticlePubMedPubMed CentralGoogle Scholar
- Etheredge LM: A rapid-learning health system: what would a rapid-learning health system look like, and how might we get there. Health Aff. 2007, Web Exclusive Collection(26):w107-w118 (Published ahead of print January 26, 2007, doi:10.1377/hlthaff.26.2.w107Google Scholar
- Meissner HI, Vernon SW, Rimer BK, Wilson KM, Rakowski W, Briss PA: The future of research that promotes cancer screening. Cancer. 2004, 101 (5 Suppl): 1251-9.View ArticlePubMedGoogle Scholar
- Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science. 2009, 4: 50-10.1186/1748-5908-4-50.View ArticlePubMedPubMed CentralGoogle Scholar
- Glasgow RE, Kaplan R, Ockene J, Society of Behavioral Medicine Health Policy Committee: Patient-reported measures of psychosocial issues and health behavior should be added to electronic health records. Health Affairs. 2012, 31 (3): 497-504. 10.1377/hlthaff.2010.1295.View ArticlePubMedGoogle Scholar
- Tanenbaum S: Evidence-based practice as mental health policy: three controversies and a caveat. Health Aff. 2005, 24 (1): 163-173. 10.1377/hlthaff.24.1.163.View ArticleGoogle Scholar
- Rabin BA, Purcell P, Naveed S: Advancing the application, quality and harmonization of implementation science measures. Implement Sci. 2012, 7: 119-10.1186/1748-5908-7-119.View ArticlePubMedPubMed CentralGoogle Scholar
- Estabrooks P, Glasgow RE, Dzewaltowski DA: Physical activity promotion through primary care. JAMA. 2003, 289 (22): 2913-2916. 10.1001/jama.289.22.2913.View ArticlePubMedGoogle Scholar
- Rogers EM: Diffusion of Innovations (4th Edition). 1995, New York: Free PressGoogle Scholar
- Senge PM, Kleiner A, Roberts C: The dance of change: The challenges of sustaining momentum in learning organizations. 1999, New York: Currency/Doubleday, 1Google Scholar
- Lorig KR, Holman HR: Self-management education: history, definition, outcomes, and mechanisms. Ann Behav Med. 2003, 26: 1-7.View ArticlePubMedGoogle Scholar
- Institute of Medicine: A foundation for evidence-driven practice: A rapid learning system for cancer care. 2010, Workshop Summary. Washington, D.C: NAS PressGoogle Scholar
- Wagner EH, Austin BT, Von Korff M: Organizing care for patients with chronic illness. Milbank Q. 1996, 74: 511-544. 10.2307/3350391.View ArticlePubMedGoogle Scholar
- Unutzer J, Powers D, Katon W: From establishing an evidence-based practice to implementation in real-world settings: IMPACT as a case study. Psychiatr Clin North Am. 2005, 28 (4): 1079-1092. 10.1016/j.psc.2005.09.001.View ArticlePubMedGoogle Scholar
- Detmer D, Bloomrosen M, Raymond B: Integrated personal health records: Transformative tools for consumer-centric care. BMC Med Inform Decis Mak. 2008, 8 (45): 1-14.Google Scholar
- PatientsLikeMe: PatientsLikeMe website. 2012, http://patientslikeme.com. Accessed March 15, 2012Google Scholar
- Institute of Medicine: Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. 2012, Consensus Report. Washington, D.C: NAS PressGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.