Prioritizing evidence-based practices for acute respiratory distress syndrome using digital data: an iterative multi-stakeholder process
Implementation Science volume 17, Article number: 82 (2022)
Evidence-based practices (EBPs) for patients receiving invasive mechanical ventilation vary in the quality of their underlying evidence and ease of implementation.
How do researchers and clinicians prioritize EBPs to help guide clinical decision-making and focus implementation efforts to improve patient care using existing, validated measures?
Study design and methods
We developed a 4-step rapid method using existing criteria to prioritize EBPs associated with lower mortality and/or shorter duration of invasive mechanical ventilation for patients suffering from acute respiratory failure or acute respiratory distress syndrome. Using different types of data including surveys, we (1) identified relevant EBPs, (2) rated EBPs using the Guideline Implementability Appraisal (GLIA) tool, (3) surveyed practicing ICU clinicians from different hospital systems using a subset of GLIA criteria, and (4) developed metrics to assess EBP performance. In this paper, we describe steps 2 and 3.
In step 2, we prioritized 11 EBPs from an initial list of 30, using surveys and ratings among a small group of clinician researchers. In step 3, 42 clinicians from 8 different hospital systems provided assessments of these 11 EBPs which inform the final step of metric development.
Our prioritization process allowed us to identify 11 EBPs out of a larger group that clinicians perceive is most likely to help optimize invasive mechanical ventilation and improve the outcomes of this vulnerable patient population. While this method was developed in critical care related to adults receiving invasive mechanical ventilation, it is adaptable to other health contexts.
The translational science movement has helped enhance patient care by incorporating clinical research into daily practice. As clinical research evolves, the number of evidence-based practices (EBPs) for a given health problem may also grow. Consequently, deciding which EBPs are most clinically important and feasible to be implemented in care can become burdensome for providers. In addition, it is essential to prioritize EBPs for implementation, given limited resources and time. However, little guidance is provided for this work in implementation process models, in part because the specifics depend on the characteristics of the processes and context in which implementation is being done. To address this need, we used existing criteria to develop a rapid 4-step method for prioritizing EBPs that can be replicated across settings and EBPs. In this report, we outline this method and provide an example of the prioritization process in the context of invasive mechanical ventilation in intensive care unit (ICU) treatment of acute respiratory failure and acute respiratory distress syndrome.
Prior to the COVID pandemic, approximately 200,000 critically ill adults received invasive mechanical ventilation in an ICU for acute respiratory failure and acute respiratory distress syndrome in the USA each year [1,2,3]. These are vulnerable patient populations—mortality rates remain high at 30–40%, and survivors are at risk for a number of poor outcomes [1,2,3,4]. EBPs that improve the outcomes of patients who receive invasive mechanical ventilation are described in multiple guidelines, yet recommendations regarding the care of these patients have not been fully implemented into routine practice [5,6,7,8]. Clinicians may find it difficult to choose among the many EBPs supported by reasonable evidence. The team environment of critical care makes this a complex task, as the opinion of a single provider is rarely sufficient. While categorizing the EBPs across complex processes of care is helpful in knowing when an EBP might be implemented, yet we lack systematic and replicable processes for helping clinicians decide which EBPs to prioritize and implement in these complex care scenarios, where processes overlap and patient care progresses at different rates through the phases of an idealized care continuum. To address this gap, we developed a method using existing and reproducible tools to prioritize EBPs while also recognizing and accounting for the interrelations among EBPs across the care continuum. As made evident by the ongoing COVID epidemic, optimizing invasive mechanical ventilation is of utmost importance, when it is required, but the methods we describe can be used in many similar clinical contexts.
Our objective in this paper is to describe our methods for prioritizing EBPs for the provision of invasive mechanical ventilation for acute respiratory failure/acute respiratory distress syndrome. To illustrate the approach, we report data from a network of 8 hospital systems across the USA in which we developed and used the methods. Overall, our goal is to improve the delivery of care to critically ill adults while describing a scalable method of EBP prioritization that can be used in other settings.
Study design and methods
Our research team included clinicians and researchers from 4 health systems specializing in pulmonary and critical care medicine, implementation science, learning health systems, and organizational behavior. The larger project was a planning grant from the US National Heart, Lung and Blood Institute of the National Institutes of Health (U01HL143453, Sales and Gong co-PIs), called Digital Implementation Trials in Acute Lung Care (DIGITAL-C), with the ultimate goal to plan a multi-site hybrid type 2 implementation-effectiveness trial of digital implementation strategies. We report abridged methods here; more detailed background and methods are available in Additional file 1.
We engaged in a 4-step multi-method process to evaluate and prioritize EBPs to assist in clinical decision-making and concentrate future implementation efforts. Throughout this process, we focused on EBPs most relevant to and strongly associated with improved clinical outcomes (i.e., shorter duration of mechanical ventilation and/or lower mortality), as identified in previously established guidelines, among patients receiving invasive mechanical ventilation for acute respiratory failure or acute respiratory distress syndrome. In steps 2–4, we also considered the feasibility of using digital data, extracted from electronic health records, to assess EBP performance, rather than processes of human data abstraction.
An overview of our 4-step prioritization process is shown in Table 1. In step 1 (see Ervin et al. ), clinician experts from our research team identified key guidelines that included several EBPs, and we searched the literature for related reviews.
Step 1 is reported in our previous paper, in which we describe the 20 EBPs that we collated from the literature review . Step 1 was conducted from July to December 2018. The focus of the current report is on steps 2 and 3.
In step 2, clinician researchers on our team (MWS, MNG, TJI, CLH) rated a list of 26 EBPs generated from step 1 using two Qualtrics surveys. The first survey contained the full set of criteria from the Guideline Implementability Appraisal 2.0 (GLIA) tool, which we show in Table 2 .
In this and subsequent steps, we used radar graphs (Figs. 1, 2 and 3) to assess responses across all of the GLIA dimensions concurrently. To construct these graphs, we averaged the responses and plotted them on the 11 axes of the GLIA dimensions, using Microsoft Excel. At this point, following discussion focused on the radar graphs, we removed a total of 15 EBPs from the list, leaving 11. Our primary criteria focused on measurability, resource intensiveness, and source credibility. We conducted step 2 in February-May 2019.
In step 3, frontline clinicians from 8 participating hospital systems evaluated the distilled list of 11 EBPs produced from step 2 using a much-reduced survey instrument. We reduced the survey to 3 GLIA elements (measurability, resource intensiveness, and source credibility) based on the experience of the team clinicians who engaged in the first 2 rounds of surveys, who felt that the full GLIA was very burdensome. We conducted these surveys in June-September 2019.
We surveyed clinicians directly involved in caring for patients receiving invasive mechanical ventilation, including attending physicians, house staff, nurse managers, registered nurses, and respiratory therapists. We used Qualtrics as the platform for the survey. Clinicians were asked whether we should include the EBPs in the final list for implementation (yes; no; maybe) and then rated the EBPs on 3 GLIA criteria: measurability, resource intensiveness, and source credibility. These three criteria were selected based on the specific needs of this project, which included the ability to use electronic medical record data to extract data for measurement, feasibility which we operationalized as “resource intensiveness,” and whether the recommendation came from a credible source. These are modified from the original GLIA terms and were selected based on input from the clinician members of the research team. We note that we used the term “source credibility” rather than “evidence validity” to reduce the burden to clinicians of fully assessing the validity of the evidence for each recommendation. In future work, we recommend that all the GLIA questions be considered for prioritization, depending on the groups involved in assessing recommendations. Descriptive statistics were calculated in Microsoft Excel; missing data were omitted using pairwise deletion.
The 4th and final step, not addressed in this paper, was to use all of the available data to generate the final list of EBPs to focus future implementation efforts. This step required developing metrics for each of the included EPBs using digitally extracted data from electronic health records, which we will report in subsequent papers. We placed heavy emphasis on measurability and variability in practice within and across ICUs and health systems.
This study was deemed exempt from human subject’s oversight by IRBMED at the University of Michigan. Data were gathered throughout 2019.
Step 1: Identifying EBPs
Our process of initially identifying relevant EBPs is described in Ervin et al. , and involved a review of reviews to identify EBPs along the full continuum (three phases) of invasive mechanical ventilation care, corresponding to specific processes of care.
Step 2: Rating EBPs
We present findings from the first survey visually as radar graphs (see Figs. 1, 2 and 3) and in Additional file 2—tabular results of the clinician researcher review. The radar graphs show the degree of variation across the ratings for the EBPs. The graphs showing a fully rounded circle were rated most highly across all dimensions of the GLIA, while those with jagged circles, with some dimensions scoring a 1 or 2 rather than a 4 or 5, demonstrate that some GLIA elements were rated highly while others were not. For example, in Fig. 1, lung-protective ventilation was rated high on all domains with little variability, whereas the use of recruitment maneuvers was rated mid to low on most domains with greater variability.
The second survey completed by clinician researchers contained the 26 EBPs from step 1, as well as 4 additional EBPs identified by clinician researchers after the first survey: protocol-based pain assessment and management, conservative fluid management, daily awakening trials, extubation to high-flow nasal cannula. Findings from the second survey supported the prioritization of 11 EBPs (Additional file 2).
The prioritized EBPs from step 2 represent processes across the continuum of invasive mechanical ventilation care; the 2 EBPs associated with escalation of care and initiation of invasive mechanical ventilation were lung-protective ventilation  and prone position ; 6 EBPs reducing complications included an analgesia-first approach to sedation , protocol-based pain assessment and management , conservative fluid management , daily awakening and breathing trials , early mobilization , and sedation protocols ; and the 3 EBPs associated with de-escalation of care and post-extubation recovery included use of a ventilator liberation protocol , extubation to noninvasive ventilation , and extubation to high-flow nasal cannula .
Step 3: Engage stakeholders beyond the research team
Forty-two ICU clinicians from 8 integrated health systems across the USA responded to a survey, rating the 11 EBPs identified in step 2 using a shortened survey instrument (Additional file 3 contains all survey instruments). As reported in Table 3, lung-protective ventilation, prone positioning, and sedation protocols were rated the highest in measurability. Early mobilization, prone positioning, and extubation to noninvasive ventilation were rated as the most resource intensive. Lung-protective ventilation, paired spontaneous awakening and breathing trials, and prone positioning were rated the highest in source credibility.
Step 4: Final selection through metric creation
We will report the final synthesis methods and results in future reports. In this step, we extracted electronic health record data from 6 of the 8 sites participating in this study to develop performance metrics for the prioritized EPBs.
In this paper, we present a rapid systematic method for prioritizing EBPs using previously established criteria. We developed this process and present invasive mechanical ventilation as an example. A similar multistep process could be used for any discrete clinical processes, especially complex clinical care processes. Prioritization is an essential step for implementation in complex interventions, described in many implementation process models [17,18,19] although typically with little detail. One approach uses conjoint analysis, which is valuable but quite burdensome and time-consuming to conduct . Other approaches, such as modified Delphi techniques [21, 22], use relatively unstructured brainstorming and selection by varying groups involved in the implementation processes. Steps 2 and 3 of the prioritization process that we describe in this paper, constituting the most direct components of prioritization, took about 3 months to complete. Use of surveys and the GLIA criteria, as well as assessment of clinical importance, facilitated this work.
The literature on implementation in healthcare emphasizes the importance of engaging stakeholders throughout the process of implementation. The process we describe in this paper is feasible and can be used in many settings and provides a method of engaging and obtaining input from a wide range of clinicians (step 3), which is often very difficult in most implementation projects across multiple settings. While interviews and/or observation yield richer data, particularly about thinking underlying responses to criteria, obtaining a broad assessment of criteria is important. Even in high acuity settings, where clinicians often have little time and energy for engaging with researchers and others, a simple survey like the one we used in step 3 can get their essential input. In future work, implementing the final set of EBPs, we will inform clinicians about how EBPs were selected and ultimately prioritized.
We note that despite the importance of a criterion-based approach (the GLIA 2.0 instrument), clinicians on our team found our initial survey to be excessively burdensome, despite their high motivation to respond. Based on their input, we reduced the survey from eleven dimensions to the three that clinician members of the team deemed most important. Even when we administered the survey to the clinicians on the research team, we only used the main dimensions rather than the full GLIA instrument, which is intended to address implementability of guideline recommendations, a more extensive application than ours in this work. This may be important to revisit for other applications of this method, where different GLIA criteria may be relevant.
We detail time and effort required in using this approach in Additional file 1. In steps 2 and 3, we found radar graphs to be particularly helpful when comparing a large number of EBPs on all 11 GLIA criteria. They provide a visual representation that is easier to assimilate than tabular representation of the results, in part by providing easy visuals of variability across criteria and across EBPs. Using these, as well as surveys and established criteria, out of a list of 30, we were able to identify 11 EBPs that met the greatest number of criteria and represented the continuum of care, from intubation to liberation, for patients receiving invasive mechanical ventilation. We note that in future work, we will include clinical importance as a criterion throughout the prioritization process.
Finally, site liaisons and champions have been critical to the success of this work to date. In externally driven work, these champions have helped with buy-in, have coordinated and participated in interviews identifying barriers and facilitators to the prioritized EBPs, and helped identify digital metrics. This work is not possible without engagement and support of these key stakeholders. Engaging them in the discussions and in the review of findings from the surveys has been instrumental in ensuring that our prioritization decisions are based in sites that represent different contexts.
The evidence base in critical care medicine is constantly evolving. Large randomized controlled studies are published each year that change current guidelines and recommendations. Information about how COVID affects patients has already led to changes in the strength of some evidence related to invasive mechanical ventilation care. Therefore, it will be necessary to review and re-prioritize EBPs periodically, to ensure that the most up-to-date evidence is guiding medical decision-making and implementation efforts.
With regard to the prioritization process itself, we found that throughout steps 2 and 3, study participants, as well as members of our research team, were anchoring judgments of measurability based on their own experience and awareness of data. This is not a limitation per se; however, it is important to be aware of individual- versus unit- and health system-level variability.
Our primary purpose in this study is to describe a rapid method of eliciting priorities from clinicians who will be affected by implementation processes. We used pragmatic approaches, such as distributing surveys through site champions, not conducting inference testing to assess statistical significance of agreement among raters, or using the more understandable mean rather than median in creating the radar graphs, for example. Adding levels of rigor to what is often necessarily highly pragmatic work , preparing for rigorous implementation, adds cost and complexity to already complex preparatory work.
Our research team prioritized 11 EBPs that are supported by quality evidence and are feasible to implement but not yet fully implemented in routine practice. Next steps include the development and validation of performance metrics for these EPBs from digital data and the analysis of interview data to identify barriers and facilitators to specific EBPs in order to guide development of implementation interventions to promote evidence-driven care for this vulnerable patient population.
Availability of data and materials
Data are available by contacting the senior author, Anne Sales.
Intensive care unit
Wunsch H, Linde-Zwirble WT, Angus DC, Hartman ME, Milbrandt EB, Kahn JM. The epidemiology of mechanical ventilation use in the United States. Crit Care Med. 2010;38:1947–53 Ovid Technologies (Wolters Kluwer Health).
Rubenfeld GD, Caldwell E, Peabody E, Weaver J, Martin DP, Neff M, et al. Incidence and outcomes of acute lung injury. N Engl J Med. 2005;353:1685–93 Massachusetts Medical Society.
Stefan MS, Shieh M-S, Pekow PS, Rothberg MB, Steingrub JS, Lagu T, et al. Epidemiology and outcomes of acute respiratory failure in the United States, 2001 to 2009: a national survey. J Hosp Med. 2013;8:76–82 Wiley.
Fan E, Del Sorbo L, Goligher EC, Hodgson CL, Munshi L, Walkey AJ, et al. An official American Thoracic Society/European society of intensive care medicine/society of critical care medicine clinical practice guideline: mechanical ventilation in adult patients with acute respiratory distress syndrome. Am J Respir Crit Care Med. 2017;195:1253–63 American Thoracic Society.
Phua J, Weng L, Ling L, Egi M, Lim C-M, Divatia JV, et al. Intensive care management of coronavirus disease 2019 (COVID-19): challenges and recommendations. Lancet Respir Med. 2020;8:506–17 Elsevier BV.
Weiss CH. Why do we fail to deliver evidence-based practice in critical care medicine? Curr Opin Crit Care. 2017;23:400–5.
Kahn JM. Bringing implementation science to the intensive care unit. Curr Opin Crit Care. 2017;23:398–9.
Ervin J, Rentes VC, Dibble E, Sjoding MW, Hough CTL, Iwashyna TJ, et al. Evidence based practices in the continuum of care for mechanically ventilated patients: a review of reviews. C46 critical care: acute respiratory failure and mechanical ventilation: American Thoracic Society; 2020. p. A5235.
Shiffman RN, Dixon J, Brandt C, Essaihi A, Hsiao A, Michel G, et al. The GuideLine Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation. BMC Med Inform Decis Mak. 2005;5:23 Springer Science and Business Media LLC.
Strøm T, Martinussen T, Toft P. A protocol of no sedation for critically ill patients receiving mechanical ventilation: a randomised trial. Lancet. 2010;375:475–80 Elsevier BV.
Devlin JW, Skrobik Y, Gélinas C, Needham DM, Slooter AJC, Pandharipande PP, et al. Clinical practice guidelines for the prevention and management of pain, agitation/sedation, delirium, immobility, and sleep disruption in adult patients in the ICU. Crit Care Med. 2018;46:e825–73 Ovid Technologies (Wolters Kluwer Health).
Semler MW, Wheeler AP, Thompson BT, Bernard GR, Wiedemann HP, Rice TW, et al. Impact of initial central venous pressure on outcomes of conservative versus liberal fluid management in acute respiratory distress syndrome. Crit Care Med. 2016;44:782–9 Ovid Technologies (Wolters Kluwer Health).
Girard TD, Kress JP, Fuchs BD, Thomason JWW, Schweickert WD, Pun BT, et al. Efficacy and safety of a paired sedation and ventilator weaning protocol for mechanically ventilated patients in intensive care (awakening and breathing controlled trial): a randomised controlled trial. Lancet. 2008;371:126–34 Elsevier BV.
Schmidt GA, Girard TD, Kress JP, Morris PE, Ouellette DR, Alhazzani W, et al. Official executive summary of an American Thoracic Society/American College of Chest Physicians clinical practice guideline: liberation from mechanical ventilation in critically ill adults. Am J Respir Crit Care Med. 2017;195:115–9 American Thoracic Society.
Ouellette DR, Patel S, Girard TD, Morris PE, Schmidt GA, Truwit JD, et al. Liberation from mechanical ventilation in critically ill adults: an official American College of Chest Physicians/American Thoracic Society clinical practice guideline: inspiratory pressure augmentation during spontaneous breathing trials, protocols minimizing sedation, and noninvasive ventilation immediately after extubation. Chest. 2017;151:166–80.
Hernández G, Vaquero C, González P, Subira C, Frutos-Vivar F, Rialp G, et al. Effect of postextubation high-flow nasal cannula vs conventional oxygen therapy on reintubation in low-risk patients: a randomized clinical trial. JAMA. 2016;315:1354–61.
Mitton C, Donaldson C. Health care priority setting: principles, practice and challenges. Cost Eff Resour Alloc. 2004;2:3 Springer Nature.
Fernandez ME, ten Hoor GA, van Lieshout S, Rodriguez SA, Beidas RS, Parcel G, et al. Implementation mapping: using intervention mapping to develop implementation strategies. Front Public Health. 2019;7:158.
Kilbourne AM, Goodrich DE, Miake-Lye I, Braganza MZ, Bowersox NW. Quality enhancement research initiative implementation roadmap: toward sustainability of evidence-based practices in a learning health system. Med Care. 2019;57:S286 Wolters Kluwer Health.
Farley K, Thompson C, Hanbury A, Chambers D. Exploring the feasibility of conjoint analysis as a tool for prioritizing innovations for implementation. Implement Sci. 2013;8:56 Springer Science and Business Media LLC.
Katcher ML, Meister AN, Sorkness CA, Staresinic AG, Pierce SE, Goodman BM, et al. Use of the modified Delphi technique to identify and rate home injury hazard risks and prevention methods for young children. Inj Prev. 2006;12:189–94 injuryprevention.bmj.com.
Uphoff EPMM, Wennekes L, Punt CJA, Grol RPTM, Wollersheim HCH, Hermens RPMG, et al. Development of generic quality indicators for patient-centered cancer care by using a RAND modified Delphi method. Cancer Nurs. 2012;35:29–37 journals.lww.com.
Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. BMJ. 2015;350:h2147.
This work was accepted for presentation at the American Thoracic Society 2020 International Conference in Philadelphia.
This project was supported by U01HL143453 and K12HL138039 from the National Heart, Lung, and Blood Institute. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Heart, Lung, and Blood Institute.
This work was reviewed by IRBMED at the University of Michigan and deemed exempt from human subject’s oversight.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Ervin, J.N., Dibble, M.R., Rentes, V.C. et al. Prioritizing evidence-based practices for acute respiratory distress syndrome using digital data: an iterative multi-stakeholder process. Implementation Sci 17, 82 (2022). https://doi.org/10.1186/s13012-022-01255-y