Skip to main content

Characterizing evolving frameworks: issues from Esmail et al. (2020) review

Abstract

There are complex issues in understanding and categorizing implementation science theories, models, and frameworks. Systematic reviews of these models are important undertakings for synthesizing current knowledge. The issues involved are even more challenging when reviewing a large number of frameworks and when some of the frameworks have evolved significantly over time. This paper addresses how the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework was described in the recent Esmail (2020) review and identifies four mischaracterizations. This is followed by a more general discussion of how advances or extensions of frameworks after an original source publication or influential review tend to be overlooked. We discuss why inadvertent mischaracterization of what a framework is and is not, and what it can and cannot be used for, can have deleterious consequences. Finally, we suggest initial ideas about what could be done to prevent or alleviate some of these problems by reviewers, framework developers, and scholars at large.

We are writing concerning the recent review article by Esmail et al. “A scoping review of full-spectrum knowledge translation theories, models, and frameworks” in Implementation Science [1]. While we see this article as making an important contribution to the literature, it also, likely inadvertently, mischaracterized the RE-AIM framework in a few salient areas. Our intent is to provide some clarity on these areas, highlight the implications of mischaracterizations of theories, models, and frameworks (TMF), and offer ideas for positive solutions that can advance implementation science.

It is important to start with the potential source of mischaracterizations related to the RE-AIM framework. The only RE-AIM reference cited in the review article was the original 1999 citation [2]. The RE-AIM framework like many others has evolved considerably over the past 20 years. Since 1999, we have published several reviews, updates, modifications, newer applications, and guidance for use of RE-AIM [3, 4]. In brief, four of the statements made about RE-AIM are incorrect, some were partially correct up until 2013, but others have not been accurate for the past decade or more. Below, we summarize the mischaracterizations, followed by a more accurate statement about the RE-AIM framework:

  1. 1.

    The RE-AIM framework is solely quantitative in nature. There have been reports of qualitative uses of RE-AIM for at least a decade. Qualitative assessments of reasons why RE-AIM results were found have been recommended in published reviews since at least 2013 and explicitly called out as a strong recommendation in Kessler et al. [5].

  2. 2.

    RE-AIM assumes all dimensions are equally important. All RE-AIM dimensions are important and contribute to overall public health impact. However, since 2006 [6], we have discussed pros and cons of different weightings of RE-AIM elements. We advise users to consider all elements, but to prioritize the dimensions most important to stakeholders in a given project [7].

  3. 3.

    The time interval for assessing implementation and maintenance are arbitrary. The time interval for implementation and maintenance are not arbitrarily mandated to be 6 months and 2 years, respectively. In our first publication on RE-AIM in 1999 [2], this was the case, but in many articles since then, the framework and these temporal perspectives have evolved—e.g., from sustained implementation directly after grant funding ceases to as long as 5 years after the initial implementation was completed [8].

  4. 4.

    The RE-AIM framework is only an evaluation framework. RE-AIM is not just an evaluation framework. It has been widely and successfully used to plan interventions for almost 15 years [9] and, more recently, to help guide adaptations during implementation. In addition, the PRISM framework which includes RE-AIM was added in 2008 [10] to address contextual factors that impact RE-AIM outcomes.

What promoted this letter is that similar mis-categorization and statements that RE-AIM is only an evaluation framework, cannot be used for any other purpose, is not an implementation model, or cannot be used as a qualitative approach are often heard from grant and journal article reviewers despite abundant evidence to the contrary. These inaccurate assertions have had negative impacts on advancing science and on the evaluation of proposals of several promising emerging researchers.

We take partial responsibility for this situation, since our group has not consistently provided clear guidance or unambiguous statements on these issues. Although we try to keep our website [3] current, there have been inconsistencies on the website, and as discussed above, the model has and continues to evolve. We have recently published the Glasgow et al. [4] paper to more clearly describe what is now Expanded RE-AIM/PRISM, the meaning and distinctions among various RE-AIM dimensions, and provide guidance on related issues.

Our larger concern is that similar mischaracterization issues apply not only to RE-AIM, but also to other TMFs. From discussions with other TMF developers, we speculate that many reviewers and scientists rely only on either (a) the original TMF article without considering refinements and extensions over time or (b) an influential review, or even increasingly reviews of reviews. We understand that when reviewing the broad range of TMFs, it is not possible to review the entire literature on each TMF, but conclusions in major publications have consequences.

Beyond specific issues concerning RE-AIM, we would like to begin discussion of how the field can collectively prevent or address unintended consequences of earlier categorizations or initial TMF papers being cited as conclusive without considering additional literature. We recommend that discussion sections of reviews include a highlighted statement that not all studies on each TMF were evaluated, that there may have been changes since an original publication on a given TMF, and that readers should not rely solely on that review and/or the original reference. We welcome other suggestions on what reviewers, TMF developers, and readers can do to alleviate this troubling and relatively common issue.

Availability of data and materials

N/A

Abbreviations

RE-AIM:

Reach, Effectiveness, Adoption, Implementation, and Maintenance

TMF:

Theories, models, and frameworks

PRISM:

Practical, Robust, Implementation, and Sustainability Model

References

  1. 1.

    Esmail R, Hanson HM, Holroyd-Leduc J, Brown S, Strifler L, Straus SE, et al. A scoping review of full-spectrum knowledge translation theories, models, and frameworks. Implement Sci. 2020;15(1):11.

    Article  Google Scholar 

  2. 2.

    Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.

    CAS  Article  Google Scholar 

  3. 3.

    Available from: www.re-aim.org.

  4. 4.

    Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. 2019;7:64.

    Article  Google Scholar 

  5. 5.

    Kessler RS, Purcell EP, Glasgow RE, Klesges LM, Benkeser RM, Peek CJ. What does it mean to “employ” the RE-AIM model? Eval Health Prof. 2013;36(1):44–66.

    Article  Google Scholar 

  6. 6.

    Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res. 2006;21(5):688–94.

    Article  Google Scholar 

  7. 7.

    Glasgow RE, Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis. 2018;15:E02.

    Article  Google Scholar 

  8. 8.

    Estabrooks PA, Bradshaw M, Dzewaltowski DA, Smith-Ray RL. Determining the impact of Walk Kansas: applying a team-building approach to community physical activity promotion. Ann Behav Med. 2008;36(1):1–12.

    Article  Google Scholar 

  9. 9.

    Klesges LM, Estabrooks PA, Dzewaltowski DA, Bull SS, Glasgow RE. Beginning with the application in mind: designing and planning health behavior change interventions to enhance dissemination. Annals of Behavioral Medicine. 2005;29(2):66–75.

    Article  Google Scholar 

  10. 10.

    Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008;34(4):228–43.

    PubMed  Google Scholar 

Download references

Acknowledgements

None

Funding

NCI grant 1P50CA244688-01

Author information

Affiliations

Authors

Contributions

RG, PE, and MO worked in collaboration to draft and finalize this letter to the editor. The authors read and approved the final manuscript.

Authors’ information

Russell Glasgow, PhD, is the Director of the Dissemination and Implementation Program of ACCORDS and research professor in the Department of Family Medicine at the University of Colorado School of Medicine. Prior to Fall 2013, he was Deputy Director for Implementation Science in the Division of Cancer Control and Population Science at the US National Cancer Institute. Dr. Glasgow is one of the original developers of the RE-AIM and PRISM frameworks.

He is an implementation scientist whose recent work has focused on public health issues of enhancing the reach and adoption of evidence-based programs, adaptation and context, and pragmatic research methods and measures to enhance health equity and sustainment. He currently focuses much of his time on mentoring, training, and building capacity in implementation science

Paul Estabrooks, PhD, is the Harold M. Maurer Distinguished Chair of Health Promotion in the College of Public Health at the University of Nebraska Medical Center. He is a founding member of the RE-AIM Workgroup who focuses on community and clinical adoption, implementation, and maintenance of evidence-based nutrition, physical activity, and weight control interventions.

Marcia G. Ory, PhD, MPH, is Regents and Distinguished Professor at Texas A&M University School of Public Health -College Station Texas, and Founding Director of the university-wide Center for Population Health and Aging. She was first engaged in promoting RE-AIM in the early 2000s as a program director at the National Institute on Aging and currently instrumental as a founding member of the national workgroup on RE-AIM Planning and Evaluation Framework in disseminating RE-AIM as a planning and evaluation tool throughout the aging services network.

Corresponding author

Correspondence to Russell E. Glasgow.

Ethics declarations

Ethics approval and consent to participate

N/A

Consent for publication

N/A

Competing interests

This letter to the editor was written by members of the RE-AIM workgroup.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Glasgow, R.E., Estabrooks, P.A. & Ory, M.G. Characterizing evolving frameworks: issues from Esmail et al. (2020) review. Implementation Sci 15, 53 (2020). https://doi.org/10.1186/s13012-020-01009-8

Download citation

Keywords

  • Review
  • RE-AIM
  • Implementation science framework
  • Classification