- Study protocol
- Open Access
A cluster randomized controlled trial comparing Virtual Learning Collaborative and Technical Assistance strategies to implement an early palliative care program for patients with advanced cancer and their caregivers: a study protocol
Implementation Science volume 16, Article number: 25 (2021)
Virtual Learning Collaboratives (VLC), learning communities focused on a common purpose, are used frequently in healthcare settings to implement best practices. Yet, there is limited research testing the effectiveness of this approach compared to other implementation strategies. This study evaluates the effectiveness of a VLC compared to Technical Assistance (TA) among community oncology practices implementing ENABLE (Educate, Nurture, Advise, Before Life Ends), an evidence-based, early palliative care telehealth, psycho-educational intervention for patients with newly diagnosed advanced cancer and their caregivers.
Using Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) and Proctor’s Implementation Outcomes Frameworks, this two-arm hybrid type-III cluster-randomized controlled trial (RCT) will compare two implementation strategies, VLC versus TA, among the 48 National Cancer Institute Community Oncology Research Program (NCORP) practice clusters that have not historically provided palliative care to all patients with advanced cancer. Three cohorts of practice clusters will be randomized to the study arms. Each practice cluster will recruit 15–27 patients and a family caregiver to participate in ENABLE. The primary study outcome is ENABLE uptake (patient level), i.e., the proportion of eligible patients who complete the ENABLE program (receive a palliative care assessment and complete the six ENABLE sessions over 12 weeks). The secondary outcome is overall program implementation (practice cluster level), as measured by the General Organizational Index at baseline, 6, and 12 months. Exploratory aims assess patient and caregiver mood and quality of life outcomes at baseline, 12, and 24 weeks. Practice cluster randomization will seek to keep the proportion of rural practices, practice sizes, and minority patients seen within each practice balanced across the two study arms.
This study will advance the field of implementation science by evaluating VLC effectiveness, a commonly used but understudied, implementation strategy. The study will advance the field of palliative care by building the capacity and infrastructure to implement an early palliative care program in community oncology practices.
Clinicaltrials.gov. NCT04062552; Pre-results. Registered: August 20, 2019. https://clinicaltrials.gov/ct2/show/NCT04062552?term=NCT04062552&draw=2&rank=1
Background and rationale
Due to the benefits of early palliative care (EPC) in randomized controlled trials [1,2,3], the American Society of Clinical Oncology recommends “ … combined standard oncology care and palliative care … early in the course of illness for any patient with metastatic cancer and/or high symptom burden”  (p.881). Yet, the National Cancer Institute (NCI)’s “Landscape Study” revealed that only 30% of community oncology practices have access to palliative care multidisciplinary teams in outpatient settings [5, 6]. Even practices offering EPC are often unable to meet the needs of all advanced cancer patients due to a limited palliative care workforce and the large number of patients with advanced cancer and caregivers [7, 8]. Training existing oncology staff to provide evidence-based palliative care interventions  is a potentially scalable and sustainable approach to address the national shortage of palliative care specialists. The ENABLE (Educate, Nurture, Advise, Before Life Ends) program is an example of such an intervention that can be implemented by healthcare organizations seeking to develop or enhance palliative care services.
ENABLE is an evidence-based, scalable model of EPC promoted by the National Cancer Institute Evidence-Based Cancer Control Programs (EBCCP), formerly known as Research-Tested Intervention Program (RTIP) [10,11,12]. ENABLE is led by a Nurse Coach and includes a comprehensive palliative care assessment, weekly telehealth coaching sessions (6 for patient; 3 for caregiver) using the Charting Your Course© guidebook. The coaching sessions review essential palliative care topics, such as problem solving, coping, symptom management, self-care, communication, decision-making, and advance care planning and monthly follow-up calls (Fig. 1). The parallel ENABLE program topics for caregivers mirror the patient sessions. ENABLE has not been widely used in community oncology practices despite being publicly available through the EBCCP. An effective implementation strategy to address implementation barriers (e.g., lack of knowledge about evidence-based EPC, insufficient infrastructure) is needed to equip oncology practices to develop the skills to implement and sustain ENABLE. However, it is unclear which strategy would be most successful in assisting implementation.
Two common strategies to enhance implementation of evidence-based practices include VLC and TA. VLC is an innovative version of the traditional (face-to-face) Learning Collaborative, developed by the Institute for Healthcare Improvement (IHI) to guide clinical teams in implementing evidence-based practices . Learning collaboratives use group interaction among teams to guide clinical improvement efforts. VLC, which uses a virtual (web-based) platform instead of face-to-face meetings, was designed to decrease the cost of participation and increase the speed of incorporating lessons into practice [14, 15]. The evidence on VLCs appears to improve processes, but is inconclusive on outcomes [16, 17].
The second strategy, TA, is defined as “interactive support that is individualized to the specific needs of individuals or teams.” (p.3) . TA is typically provided one-on-one to a single organization and does not use a community structure to encourage shared learning or collaboration . The evidence on TA effectiveness is minimal [19, 20]. To our knowledge, there are few rigorous trials comparing the effectiveness of VLC or TA and more testing with rigorous study design is needed [14, 15, 21, 22]. Which strategy would be most effective for the widespread implementation of ENABLE in community oncology practices is unclear.
To address these gaps in implementation science and palliative care, we developed a cluster randomized controlled trial (RCT) that compares VLC to TA for implementing ENABLE in community oncology practices. Our primary hypothesis is that practices randomized to VLC will outperform those randomized to TA resulting in a higher proportion of patients completing the ENABLE program. The more effective implementation of ENABLE will be due to the benefits of group dynamics (e.g., shared experiences, accountability, learning, discussion, and interactions), and quality improvement learning/skills that occur in the VLC but not in TA. Ultimately, a greater uptake of ENABLE will lead to better outcomes (mood and quality of life) for the patients and caregivers at the practices exposed to VLC compared to those exposed to TA.
Study aims and hypotheses
Based on Proctor’s implementation framework, the purpose of this two-arm hybrid type-III cluster RCT in community oncology practices is to compare the effectiveness of two strategies, VLC and TA for implementing the evidence-based ENABLE program (Fig. 2).
Primary aim (implementation outcome, patient level)
Compare the effectiveness of VLC vs. TA on ENABLE program uptake, as measured by the proportion of patients who complete the ENABLE program (i.e., have a palliative care assessment and complete the six ENABLE sessions). Hypothesis: Practice clusters randomized to VLC will have greater ENABLE program uptake than practice clusters randomized to TA.
Secondary aim (service outcome, practice level)
Compare the effectiveness of VLC vs. TA on practice clusters’ overall ENABLE program implementation. This will be assessed using the ENABLE General Organizational Index (GOI), which measures factors that influence institutional implementation of evidence-based practices. Hypothesis: Practice clusters randomized to VLC will have better overall ENABLE program implementation than practice clusters randomized to TA.
Exploratory aim 1 (patient & caregiver outcomes)
Compare practice clusters randomized to VLC or TA on patient and caregiver quality of life (QOL) and mood outcomes. Hypothesis: Patients and caregivers at practice clusters randomized to VLC will have better QOL and mood compared to those at practice clusters randomized to TA.
Exploratory aim 2
Determine the relationship among ENABLE program uptake, overall ENABLE program implementation, and patients’ QOL and mood across the two strategies. Hypothesis: Practice clusters with better ENABLE program uptake and overall ENABLE implementation (regardless of implementation group) will demonstrate a higher degree of improved patient outcomes (i.e., QOL and mood) compared to practice clusters that have low/poor ENABLE program uptake and overall ENABLE implementation.
This study is guided by the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework, which is designed to enhance the quality, speed, and public health impact of efforts to translate research into “real world” settings [23, 24]. Further, this study applies Proctor’s Implementation Outcomes framework  to assess implementation success via outcomes on three levels: implementation (effects of actions to implement new treatment), service (reflect the practice process changes), and client (evidence-based program’s effect on the target population).
This two-arm hybrid type-III cluster RCT compares implementation strategy effectiveness, while gathering additional information on the ENABLE program outcomes [26,27,28]. VLC and TA have distinct communication channels, time requirements, social systems, and structures [29, 30]. A target sample of 48 NCI’s Community Oncology Research Program (NCORP) practices will be randomized to one of two conditions: 1) VLC group which will receive ENABLE training and participate in a 15-month VLC (n = 24) or 2) TA group which will receive ENABLE training and participate in a 15-month TA process (n = 24). Each NCORP practice in this study will be formed into “practice clusters.” Practice cluster is defined as an NCORP practice location where oncology physicians and study staff work independently and do not crossover to another practice. Practice clusters will be recruited in 3 cohorts of approximately 16 practice clusters per cohort. In each cohort half of the practice clusters will be randomized to VLC and half will be randomized to TA (Fig. 3).
URCC NCORP practice cluster participation
This study is being carried out from the University of Rochester Cancer Center (URCC) NCORP Research Base. NCORP is a national network that brings cancer treatment and care delivery trials to people in community settings. NCORP is comprised of 7 Research Bases, and 46 Community Affiliates, each with affiliated practices . The URCC NCORP Research Base is associated with 22 NCORPs and 358 sub-affiliates across the country. URCC NCORP practices vary in size and geographic location; however, to be eligible to participate in this study, they must meet the following inclusion criteria:
Identify two (or more) Nurse Coaches to deliver the ENABLE program. For practices that only have 1 nurse available, one Nurse Coach is allowed.
Commitment of Nurse Coaches to be trained to conduct the palliative care assessment and deliver the ENABLE sessions
Desire to implement ENABLE, including presence of an investigator (e.g., Primary Affiliate Principal Investigator, oncology physician, Cancer Care Delivery Research (CCDR) Lead) and/or program administrator/supervisor who are willing to be key contacts
Demonstrated support/buy-in from oncology physicians who are willing to enroll patients
Agreement of practice leadership and staff to support/participate in study activities
If necessary, willingness to participate in a phone interview to determine capacity to implement the ENABLE program.
Practice clusters that currently offer fully functioning inter-professional early palliative care to all cancer patients will be excluded. If practices offer early palliative care services to some, but not to all cancer patients, practices will only enroll patients to this study who have not received early palliative care services.
Within practice cluster patients and caregivers participants
Patients will be encouraged (but not required) to select a caregiver, defined below, to participate in the study. Participants can receive cancer treatment while participating in the study but will not have received palliative care services at the time of enrollment.
Patient inclusion criteria
Patients must be diagnosed with advanced cancer (stage III/IV, recurrence, or progressive solid tumor cancer) within the last 90 days and have an expected survival of at least 6 months. Patients must be English-speaking, at least 18 years old, willing to complete the palliative care assessment and ENABLE sessions, have access to a telephone that can receive calls, and able to provide informed consent to participate in the study.
Patient exclusion criteria
Patients who received previous palliative care services will be excluded. However, after enrollment, patients may receive palliative care services outside of the ENABLE program, if appropriate.
Each patient can identify a single caregiver to participate in the study. The caregiver is any person who provides support to the patient. Caregivers may remain in the study even if the patient withdraws.
Caregiver inclusion criteria
Caregivers are defined as an “unpaid relative or friend who knows them well and who provides regular support to their cancer.” Caregivers must be English-speaking, at least 18 years old, willing to complete the ENABLE Caregiver program sessions, have access to a telephone that can receive calls, and able to provide informed consent to participate in the study.
Practice cluster recruitment
The URCC NCORP Research Base will announce the study via email to all URCC NCORP community affiliates and post study activation on the Cancer Trials Support Unit (CTSU) website. In addition, we will hold information webinars. Interested practice clusters will complete a form to describe the practice (e.g., current palliative care services, demographics of patient population, staff members to be involved in the study) and submit it to URCC Research Base. Each practice cluster will be expected to enroll and deliver ENABLE to 15–27 patients.
Patient and caregiver recruitment
Staff in each practice cluster will identify eligible patients from clinical schedules and inform oncology physicians. If the physician determines the patient to be eligible and the patient agrees to be informed about the study, study personnel at the practice (e.g., Nurse Coach, research coordinator) will meet the patient either in person or via HIPAA compliant phone or videoconference to review the study. Informed consent will be obtained either in person or using a HIPAA compliant phone or videoconference and documented.
Evidence-based practice to be implemented: ENABLE
The ENABLE program for patients consists of  a standardized in-person or telehealth Palliative Care Assessment  and  a series of six, 20–60 min weekly telephone coaching sessions provided by a Nurse Coach using the Charting Your Course© (CYC) guidebook [33, 34]. ENABLE fosters patient empowerment [35, 36] through coaching on problem-solving coping, symptom management, self-care, communication, decision-making, and life review and reflection. A Nurse Coach, who is a registered nurse or advanced practice provider (e.g., nurse practitioner or physician assistant) who has at least 2 years’ experience working in oncology or palliative care, delivers the ENABLE program. Caregivers receive a separate series of similar telephone sessions using a CYC© guidebook with caregiver-relevant content. After completing the weekly sessions, both the patient and caregiver receive monthly check-in phone calls. Figure 1 presents the CYC© session topics. Trials testing ENABLE have demonstrated improved patient quality of life, mood, and survival outcomes and reduced caregiver burden and improved mood [10, 11, 37].
Implementation interventions (practice cluster level)
Table 1 summarizes the essential elements of the VLC and TA implementation strategies and outlines the distinctive features between the two. Each practice cluster will assemble an implementation team who will participate in the assigned implementation strategy (VLC or TA). At minimum, this includes one or two Nurse Coaches who will deliver the intervention and at least one coordinator who will facilitate recruitment and patient and caregiver outcome assessment. Other members of the practice cluster (e.g., oncologists, data managers) are also encouraged to participate in implementation team and VLC or TA activities.
Virtual Learning Collaborative
Practice clusters randomized to the 15-month VLC will participate in monthly virtual group-based learning sessions to discuss key components of the ENABLE program, implementation progress (e.g., nurse coach training, number referrals, number patients/caregivers enrolled, sessions completed), data collection activities, and current successes and challenges. In addition, the VLC will include instruction on quality improvement tools and techniques to assist with the implementation process, coaching, and the opportunity to share stories and experiences and learn from other practice clusters using an “all teach, all learn” approach . The VLC will be co-led by ENABLE and quality improvement experts (RT and KDL).
Practice clusters randomized to the 15-month TA study arm will receive individual, interactive monthly consultation calls with an ENABLE expert (JND-O). The TA leader will ask similar questions about the practice cluster’s implementation progress as are covered in the VLC study arm (e.g., nurse training, referral process, enrollment, sessions completed, data collection activities, and current successes and challenges). The relationship between the TA provider (JND-O) and practice clusters will be friendly, trusting, strengths-based, collaborative, and will support practice cluster autonomy. We anticipate that practice clusters will solicit input or guidance from the TA leader on challenges or areas of concern. The TA leader will offer advice and suggestions to the degree desired by the practice cluster.
Nurse Coaches who deliver the ENABLE program will complete the approximately 19-h self-paced 10-module ENABLE training on a web-based platform (Canvas). The training, developed by ENABLE experts, consists of  an overview of the ENABLE program;  instruction on how to conduct the palliative care assessment and deliver each ENABLE session;  role play of the assessment and phone sessions that will be reviewed by an ENABLE expert who will provide feedback; and  tools and resources for Nurse Coaches. Upon completion of the ENABLE training, Nurse Coaches will complete an evaluation of the training to receive Continuing Education Unit (CEU) credits.
Additional trainings will cover study procedures and how to use the web-based study platforms: Canvas, Zoom, and Research Electronic Data Capture (REDCap), an electronic data capture tool to enter study data.
Measures and outcomes
Aim 1: ENABLE uptake (implementation outcome, patient level)
The primary outcome will be ENABLE program uptake (program completion). To capture ENABLE program uptake, each ENABLE Nurse Coach will use a standardized ENABLE Nurse Coach Contact Log to document patient and caregiver referrals, enrollment, and completion of the essential ENABLE elements (i.e., palliative care assessment, ENABLE sessions). The Nurse Coach Contact Log, which is completed in REDCap, will document each contact and ENABLE session using an assigned study identification for each patient or caregiver.
Aim 2: Overall ENABLE Program implementation (service outcome, practice cluster level)
This outcome will be measured for each practice cluster via phone interviews with Nurse Coaches and coordinators using the ENABLE General Organizational Index (GOI). This instrument was developed to assess factors that influence institutional implementation of the ENABLE program . We previously adapted  the General Organizational Index (GOI) which measures fidelity to an evidence-based practice by assessing two key elements of implementation: individualization and quality improvement . The GOI has been shown to have acceptable psychometric properties . The ENABLE GOI structured interview guide covers 12 domains (program philosophy, eligibility/client identification, penetration, assessment, individualized initial palliative care treatment plan, individualized subsequent ENABLE contacts, supervision, process monitoring, outcome monitoring, quality assurance, and client choices regarding service provision) associated with successful implementation. ENABLE GOI assessments will occur at baseline, 6 months (± 8 weeks), and 1 year (± 8 weeks). After the phone interview, the interviewer  will score each domain on a Likert scale for a total range of 12–60. Moreover, interviews will be recorded and transcribed for qualitative analysis to augment quantitative assessment of this outcome.
Table 2 presents additional implementation measures, including RE-AIM, process outcomes for the study arms (VLC and TA), and ENABLE process measures.
Exploratory aim: patient measures
Table 2 presents patient measures. Patient assessments will be completed at baseline, 12, and 24 weeks. There will be several options to collect the data:  study coordinator reads the questions to the patient (either in person or on the phone) and enters the patient responses,  the patient completes the assessments in person with a study coordinator, or  the patient will take a paper copy home, complete it, and return it at either the next visit, via mail, or scan and email.
Exploratory aim: Caregiver measures
Caregiver assessments (Table 2) will be completed at baseline, 12 and 24 weeks using the same methods as for patients described above.
Aim 1: Implementation outcome (patient level)
The RCT is powered for the aim 1 outcome, ENABLE program uptake, defined as the proportion of patients who complete the ENABLE program. In determining the targeted sample size, we made the following assumptions:  an intra-class correlation coefficient (ICC) value of 0.05,  alpha of 1% (α = 0.01) and 80% power,  20% enrolled patients for whom completion of ENABLE cannot be assessed due to death or becoming too ill to continue with participation, and  a proportion of patients completing ENABLE of 0.52 in prior studies. With 24 practice clusters per group and a minimum of 12 patients per practice cluster (down from 15 after 20% unable to assess completion), for a total of 288 patients per group, the detectable difference in ENABLE program uptake is 0.18 (OR = 2.06; RR = 1.33), a medium effect size.
This cluster-randomized trial will be carried out in 3 sequential cohorts of 8–20 practice clusters (4–10 per study arm) to reach the desired sample size of 48 practice clusters. Within each cohort, RCT arm assignment will occur after completion of all required study trainings, including the ENABLE training (Fig. 3). A minimization algorithm for group assignment with random initial start will be used to attempt to simultaneously balance the proportion of rural practices, practice sizes, and minority patients seen within each practice across the two RCT arms. The minimization algorithm will be generated using R software. Study team members and practice cluster members will not be blinded to group assignment or study hypotheses.
All randomized practice clusters will be included in primary comparisons, according to their group assignment, and regardless of their degree of participation in the study. Primary data analysis will include descriptive statistics for baseline practice cluster characteristics, patient characteristics, and outcomes in each RCT arm. We will examine the balance between RCT arms with respect to baseline characteristics (e.g., age, sex, race, cancer type, stage) using effect sizes, such Cohen’s d and Cramer’s. Factors showing non-trivial imbalances between comparators will be used as adjusting covariates in the between-group comparisons. Analysis will be conducted using the latest versions of SAS and R.
Analysis for aim 1: Implementation outcome (patient level)
We will use a logit GEE model with exchangeable correlation structure (to account for clustering within practice) for the binary patient uptake indictor (“yes” if the patient completed a palliative care assessment and all ENABLE program sessions prior to the 12 week assessments and “no” if the patient is still alive but has not completed a palliative care assessment and all six ENABLE program sessions prior to the 12-week assessments), with the group assignment as the main predictor. Model-predicted outcome proportions, odds ratios, relative risks, and confidence intervals for these measures will be used for interpretation. Supplemental analyses using appropriate methods for clustered data will consist of between-group comparisons of the proportion of recruited patients who could not complete the ENABLE sessions due to illness or death, and the number of sessions completed among all patients recruited.
Analysis for aim 2: Service outcome (practice cluster level)
Analysis will compare VLC vs. TA on NCORP practices’ overall ENABLE program implementation measured by the ENABLE GOI. Due to the small sample size for this practice cluster-level outcome (24 VLC vs. 24 TA), rather than conducting formal inference testing, comparisons will be descriptive statistics and effect size measures.
The audio-recorded interviews of the ENABLE GOI will be coded by study team members (EC-S)  in Atlas.ti using a qualitative content analysis approach using a priori domains from the GOI measure and additional categories that emerge from the inductive review of transcripts . Coded data will be aggregated and reviewed within and across codes to construct an in-depth understanding of facilitators, barriers, and contextual factors that may influence implementation of the ENABLE program.
Findings from the qualitative data will be integrated with the main findings of the RCT at the interpretation and reporting stage . We will use a convergent mixed methods analysis to collect both qualitative and quantitative ENABLE GOI data at roughly the same time, assess information using parallel constructs for both types of data, separately analyze both types of data, and then compare results through discussion and jointly displaying both forms of data. The joint display will map each ENABLE GOI domain score to corresponding concepts or themes from these qualitative findings to more fully understand the context of the GOI domain scores and identify patterns across practice clusters . This will provide validation for each data type and create a foundation for drawing conclusions about the implementation of the ENABLE program .
Analysis for exploratory aims (patient and caregiver outcomes)
Exploratory aim 1
To compare NCORP practices randomized to VLC or TA on patient and caregiver quality of life and mood, linear mixed models will be used to estimate differences in patient and caregiver outcome trajectories over the 24-week follow-up period. If necessary, a false discovery rate adjustment will be used to adjust for multiple exploratory inferences .
Exploratory aim 2
The association between ENABLE sessions and change in patient outcomes will be examined with linear mixed models. Adjusting covariates will include patient characteristics that may influence participation in sessions (e.g., sex, illness acuity, age). Linear mixed models will also be used to examine the association between GOI scores and patient outcomes.
The relative strengths of learning collaboratives compared to other implementation strategies are poorly understood [16, 17]. This study will provide a direct comparison between the VLC and TA strategies to determine if VLC is a superior method to facilitate implementation of the evidence-based ENABLE program in community-based oncology practices. Finding successful strategies to implement models of early palliative care in cancer is imperative given national guidelines that have highlighted this priority .
Most published studies on VLCs are descriptive and have not definitively tested its effectiveness to enhance adoption and implementation [16, 71,72,73,74,75,76,77,78,79,80,81,82]. There are also few randomized-clinical trials that evaluate traditional in-person learning collaboratives [14, 15, 21, 22] or compare their effectiveness to other implementation approaches [16, 17, 71, 83]. For example, learning collaboratives have been shown to outperform financial incentives or toolkits, but only on some process or secondary outcomes [16, 17]. This study will contribute to the fields of implementation science and palliative care as we will learn about the implementation effectiveness of VLC and TA while delivering an evidence-based practice. While we hypothesize the VLC will outperform TA due to the shared wisdom generated by the group, the VLC strategy has greater demands for sites and facilitators (and possibly costs) and has less opportinuity for individual attention than TA. As such, it is important to assess the added value of each strategy in order to know how to better spread and sustain EPC. Therefore, this study will inform efforts to more widely disseminate ENABLE, while simultaneously advancing implementation science methods and approaches that can be used for other evidence-based practices.
Like most research, we have encountered challenges during the study start-up period. First, the COVID-19 pandemic significantly impacted the ability for NCORP practices to conduct research and resulted in a need to rely on telehealth. Although most ENABLE activities were already offered via telephone, we modified the protocol so that all study procedures can now be conducted remotely, including the consent process and the comprehensive palliative care assessment. During this time, NCORP practices also expressed concern about the time required to complete the Nurse Coach training. To reduce training burden, the length of the web-based ENABLE Nurse Coach training was reduced from 25 h to approximately 19 h. We anticipate the reduced number of training hours may be beneficial beyond the study given our goal to make the ENABLE training more broadly scalable to nurses and advanced practice providers interested in implementing this EPC program. We also amended the originally required cohort size (n = 16) and will permit cohorts of 8–20 practice clusters (4–10 per study arm). To account for this in analyses, we will assess for confounding by cohort. Lastly, although each practice cluster will be expected to deliver ENABLE to 15–27 patients, we recognize that may not happen. To account for this, we will include an option to recruit additional practice clusters (up to 20% of 48 which is 58 practices) and additional cohorts. Despite these challenges, we anticipate the first cohort will launch in the first half of 2021.
There are several anticipated lessons learned from this study. First, to the best of our knowledge, this is the first large-scale study measuring the implementation of early palliative care in community-based oncology practices. To measure implementation, we developed and refined palliative care implementation measures and anticipate that lessons learned will be used to improve these measures. Second, this study not only seeks to identify the superior implementation strategy (VLC or TA), but also to examine the key components of each strategy. To do so, we have developed measures of participation, engagement, and fidelity that have not been applied in other published literature. Lessons learned will inform refinement of the measures and advance the field.
Strengths and limitations
There are several strengths to this study. First, this study will evaluate both the uptake of an evidence-based EPC intervention and organizational factors associated with improving delivery of EPC in community based care. Second is the collaboration with the URCC NCORP. Conducting this work outside a consortium would be difficult, particularly with respect to recruiting practice clusters. Third, this study provides a solution to the limited palliative care workforce by implementing this program as a generalist palliative care model . As part of study activities, Nurse Coaches will be trained on how to deliver the ENABLE EPC program, thereby increasing practice capacity to offer palliative care services [7, 8, 85]. A final, yet unanticipated strength to this study is the opportunity to implement the effective ENABLE EPC telehealth model in community oncology clinics during COVID-19 when telehealth services are particularly important.
While this study will contribute to implementation science methods and approaches, there are limitations. Patients and caregivers will not know which study arm their practice cluster is assigned. However, practice clusters will not be blinded to study aims and hypotheses as sharing those are essential to study recruitment and consent procedures. Similarly, blinding the TA and VLC facilitators and the assessors (e.g., ENABLE GOI interviewer) to the group assignment is not feasible. However, it is feasible to limit their knowledge of specific activities in each study arm to minimize bias. Contamination between practice cluster cohorts is another potential limitation. Since all practice clusters will be affiliated with URCC NCORP, enrolled practice clusters may discuss their study involvement with practices that register for a future cohort. To minimize potential contamination between cohorts, we will instruct second and third cohorts to reach out to the TA or VLC members (depending on randomization assignment) as opposed to outside practices for advice.
The results from this study will be twofold:  we will learn about the implementation of an early palliative care intervention among community oncology practices, and  we will learn about the effectiveness of the VLC as compared to TA to facilitate such implementation. In addition, lessons learned will advance the field of implementation science through further refinement of palliative care implementation measures and of the VLC implementation strategy.
Availability of data and materials
The investigators will provide access to data according to NIH and University of Rochester NCI Community Oncology Research Program policies.
American Cancer Society
Cancer Care Delivery Research
Chronic Care Model
Chronic Illness Care
Central Institutional Review Board
Consolidated standards of reporting trials
Charting Your Course
Data Safety Monitoring Committee
Educate, Nurture, Advise, Before Life Ends
Early Palliative Care
Expert Recommendations for Implementing Change
Functional Assessment of Cancer Therapy–General
Functional Assessment of Chronic Illness-Palliative Care
General Organizational Index
Hospital Anxiety and Depression Scale
Intra-class Correlation Coefficient
Montgomery Borgatta Caregiver Burden Scale
MD Anderson Symptom Inventory
National Cancer Institute
NCI Community Oncology Research program
Patient Reported Outcome Measurement Information System
Quality of life
Randomized controlled trial
Reach, Effectiveness, Adoption, Implementation, Maintenance Framework
Research Electronic Data Capture
Rural-Urban Commuting Area
University of Alabama at Birmingham
University of Rochester
University of Rochester Cancer Center
Virtual Learning Collaborative
Kavalieratos D, Corbelli J, Zhang D, Dionne-Odom JN, Ernecoff NC, Hanmer J, et al. Association between palliative care and patient and caregiver outcomes: a systematic review and meta-analysis. JAMA. 2016;316(20):2104–14.
Meneses K, Landier W, Dionne-Odom JN. Vulnerable population challenges in the transformation of cancer care. Semin Oncol Nurs. 2016;32(2):144–53.
Hoerger M, Wayser GR, Schwing G, Suzuki A, Perry LM. Impact of interdisciplinary outpatient specialty palliative care on survial and quality of life in adults with advanced cancer: a meta-analysis of randomized controlled trials. Ann Behav Med. 2018;53(7):674–85.
Smith TJ, Temin S, Alesi ER, Abernethy AP, Balboni TA, Basch EM, et al. American Society of Clinical Oncology provisional clinical opinion: the integration of palliative care into standard oncology care. J Clin Oncol. 2012;30(8):880–7.
Carlos RC, Sicks JD, Chang GJ, Lyss AP, Stewart TL, Sung L, et al. Capacity for cancer care delivery research in National Cancer Institute Community Oncology Research Program Community Practices: availability of radiology and primary care research partners. J Am Coll Radiol. 2017;14(12):1530–7.
Kent EE, Mitchell SA, Castro KM, DeWalt DA, Kaluzny AD, Hautala JA, et al. Cancer care delivery research: building the evidence base to support practice change in community oncology. J Clin Oncol. 2015;33(24):2705–11.
Kamal AH, Wolf SP, Troy J, Leff V, Dahlin C, Rotella JD, et al. Policy changes key to promoting sustainability and growth of the specialty palliative care workforce. Health Aff (Millwood). 2019;38(6):910–8.
Lupu D, Quigley L, Mehfoud N, Salsberg ES. The growing demand for hospice and palliative medicine physicians: will the supply keep up? J Pain Symptom Manage. 2018;55(4):1216–23.
Hui D, Bruera E. Models of palliative care delivery for patients with cancer. J Clin Oncol. 2020;38(9):852–65.
Bakitas M, Lyons KD, Hegel MT, Balan S, Brokaw FC, Seville J, et al. Effects of a palliative care intervention on clinical outcomes in patients with advanced cancer: the Project ENABLE II randomized controlled trial. JAMA. 2009;302(7):741–9.
Bakitas MA, Tosteson TD, Li Z, Lyons KD, Hull JG, Li Z, et al. Early versus delayed initiation of concurrent palliative oncology care: patient outcomes in the ENABLE III randomized controlled trial. J Clin Oncol. 2015;33(13):1438–45.
Dionne-Odom JN, Azuero A, Lyons KD, Hull JG, Tosteson T, Li Z, et al. Benefits of early versus delayed palliative care to informal family caregivers of patients with advanced cancer: outcomes from the ENABLE III randomized controlled trial. J Clin Oncol. 2015;33(13):1446–52.
The breakthrough series: IHI’s collaborative model for achieving breakthrough improvement. Boston, MA; 2003.
Nadeem E, Olin SS, Hill LC, Hoagwood KE, Horwitz SM. Understanding the components of quality improvement collaboratives: a systematic literature review. Milbank Q. 2013;91(2):354–94.
Schouten LM, Hulscher ME, van Everdingen JJ, Huijsman R, Grol RP. Evidence for the impact of quality improvement collaboratives: systematic review. BMJ. 2008;336(7659):1491–4.
Fu LY, Zook K, Gingold JA, Gillespie CW, Briccetti C, Cora-Bramble D, et al. Strategies for improving vaccine delivery: a cluster-randomized trial. Pediatrics. 2016;137(6).
Speroff T, Ely EW, Greevy R, Weinger MB, Talbot TR, Wall RJ, et al. Quality improvement projects targeting health care-associated infections: comparing virtual collaborative and toolkit approaches. J Hosp Med. 2011;6(5):271–8.
Leeman J, Calancie L, Hartman MA, Escoffery CT, Herrmann AK, Tague LE, et al. What strategies are used to build practitioners' capacity to implement community-based interventions and are they effective?: a systematic review. Implement Sci. 2015;10:80.
Chilenski SM, Perkins DF, Olson J, Hoffman L, Feinberg ME, Greenberg M, et al. The power of a collaborative relationship between technical assistance providers and community prevention teams: a correlational and longitudinal study. Eval Program Plann. 2016;54:19–29.
Katz J, A W. Technical assistance to enhance prevention capacity: a research synthesis of the evidence base. Prev Sci. 2016;17(4):417–28.
Hulscher ME, Schouten LM, Grol RP, Buchan H. Determinants of success of quality improvement collaboratives: what does the literature show? BMJ Qual Saf. 2013;22(1):19–31.
Nadeem E, Olin S, Hill L, Hoagwood K, Horwitz S. A literature review of learning collaboratives in mental health care: used but untested. Psychiatr Serv. 2014;65(9):1088–99.
What is RE-AIM? 2016 [Available from: http://www.re-aim.org.
Glasgow RE, McKay HG, Piette JD, Reynolds KD. The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Educ Couns. 2001;44(2):119–27.
Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76.
Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26.
Damschroder LJ, Moin T, Datta SK, Reardon CM, Steinle N, Weinreb J, et al. Implementation and evaluation of the VA DPP clinical demonstration: protocol for a multi-site non-randomized hybrid effectiveness-implementation type III trial. Implement Sci. 2015;10:68.
Smelson DA, Chinman M, McCarthy S, Hannah G, Sawh L, Glickman M. A cluster randomized Hybrid Type III trial testing an implementation support strategy to facilitate the use of an evidence-based practice in VA homeless programs. Implement Sci. 2015;10:79.
Dearing JW, Kreuter MW. Designing for diffusion: how can we increase uptake of cancer communication innovations? Patient Educ Couns. 2010;81(Suppl):S100–10.
Rogers EM. Diffusion of innovations. Fifth Edition ed. New York: Free Press; 2003.
National Cancer Institute. Community Oncology Research Program. US: Department of Health and Human Services; 2020. [Available from: https://ncorp.cancer.gov/about/.
Manca DP, Aubrey-Bassler K, Kandola K, Aguilar C, Campbell-Scherer D, Sopcak N, et al. Implementing and evaluating a program to facilitate chronic disease prevention and screening in primary care: a mixed methods program evaluation. Implement Sci. 2014;9:135.
Wagner EH, Austin BT, Davis C, Hindmarsh M, Schaefer J, Bonomi A. Improving chronic illness care: translating evidence into action. Health Aff (Millwood). 2001;20(6):64–78.
Wasson JH, Splaine ME, Bazos D, Fisher ES. Overview: working inside, outside, and side by side to improve the quality of health care. Jt Comm J Qual Improv. 1998;24(10):513–7.
McCorkle R, Ercolano E, Lazenby M, Schulman-Green D, Schilling LS, Lorig K, et al. Self-management: enabling and empowering patients living with cancer as a chronic illness. CA Cancer J Clin. 2011;61(1):50–62.
Dionne-Odom JN, Kono A, Frost J, Jackson L, Ellis D, Ahmed A, et al. Translating and testing the ENABLE: CHF-PC concurrent palliative care model for older adults with heart failure and their family caregivers. J Palliat Med. 2014;17(9):995–1004.
Bakitas M, Lyons KD, Hegel MT, Balan S, Barnett KN, Brokaw FC, et al. The project ENABLE II randomized controlled trial to improve palliative care for rural patients with advanced cancer: baseline findings, methodological challenges, and solutions. Palliat Support Care. 2009;7(1):75–86.
Zubkoff L, Dionne-Odom JN, Pisu M, Babu D, Akyar I, Smith T, et al. Developing a “toolkit” to measure implementation of concurrent palliative care in rural community cancer centers. Palliat Support Care. 2017:1–13.
Heiervang KS, Egeland KM, Landers M, Ruud T, Joa I, Drake RE, et al. Psychometric properties of the general organizational index (GOI): a measure of individualization and quality improvement to complement program fidelity. Adm Policy Ment Health. 2020.
Freund T, Wensing M, Mahler C, Gensichen J, Erler A, Beyer M, et al. Development of a primary care-based complex care management intervention for chronically ill patients at high risk for hospitalization: a study protocol. Implementation Science. 2010;5(1):70.
Babu D, Dionne-Odom JN, Smith T, Zubkoff L, Bakitas M. Oncology providers’ perceptions of early/concurrent palliative care. In: Annual Assembly of the Americal Academy of Hospice and Palliative Medicine and the Hospice and Palliative Nurses Association. Chicago: IL; 2016.
Bjelland I, Dahl AA, Haug TT, Neckelmann D. The validity of the hospital anxiety and depression scale. An updated literature review. J Psychosom Res. 2002;52(2):69–77.
Clark A, Fallowfield LJ. Quality of life measurements in patients with malignant disease: a review. J R Soc Med. 1986;79(3):165–9.
Herrmann C. International experiences with the hospital anxiety and depression scale--a review of validation data and clinical results. J Psychosom Res. 1997;42(1):17–41.
McMillan SC, Small BJ, Weitzner M, Schonwetter R, Tittle M, Moody L, et al. Impact of coping skills intervention with family caregivers of hospice patients with cancer: a randomized clinical trial. Cancer. 2006;106(1):214–22.
Zigmond AS, Snaith RP. The hospital anxiety and depression scale. Acta Psychiatr Scand. 1983;67(6):361–70.
Brady MJ, Cella D. Assessing quality of life in palliative care. Cancer Treat Res. 1999;100:203–16.
Brady MJ, Cella DF, Mo F, Bonomi AE, Tulsky DS, Lloyd SR, et al. Reliability and validity of the functional assessment of cancer therapy-breast quality-of-life instrument. J Clin Oncol. 1997;15(3):974–86.
Lyons KD, Bakitas M, Hegel MT, Hanscom B, Hull J, Ahles TA. Reliability and validity of the functional assessment of chronic illness therapy-palliative care (FACIT-Pal) scale. J Pain Symptom Manage. 2009;37(1):23–32.
Facit.org. Functional Assessment of Chronic Illness Therapy- Palliative Care. 2020. [Available from https://www.facit.org/measures/FACIT-Pal]. Accessed 22 Feb 2020.
Cella D, Riley W, Stone A, Rothrock N, Reeve B, Yount S, et al. The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. J Clin Epidemiol. 2010;63(11):1179–94.
Hays RD, Bjorner JB, Revicki DA, Spritzer KL, Cella D. Development of physical and mental health summary scores from the patient-reported outcomes measurement information system (PROMIS) global items. Qual Life Res. 2009;18(7):873–80.
HealthMeasures. Obtain & Administer Measures: Northwestern University; 2018 [Available from: http://www.healthmeasures.net/explore-measurement-systems/promis/obtain-administer-measures.
HealthMeasures. Global Health: Northwestern University; 2018 [Available from: http://www.healthmeasures.net/administrator/components/com_instruments/uploads/PROMIS Global Scoring Manual.pdf.
Center MAC. The MD Anderson Symptom Inventory: University of Texas; 2018 [Available from: https://www.mdanderson.org/research/departments-labs-institutes/departments-divisions/symptom-research/symptom-assessment-tools/md-anderson-symptom-inventory.html.
Cleeland CS, Mendoza TR, Wang XS, Chou C, Harle MT, Morrissey M, et al. Assessing symptom distress in cancer patients: the M.D. Anderson Symptom Inventory. Cancer. 2000;89(7):1634–46.
Montgomery R. Using and interpreting the montgomery borgatta caregiver burden scale 2016 [Available from: https://www.researchgate.net/profile/Rhonda_Montgomery2/publication/265679222_Using_and_Interpreting_the_Montgomery_Borgatta_Caregiving_Burden_Scale/links/56aac20508aed5a01358c5bd.pdf?origin=publication_list.
Montgomery R, Borgatta E, Borgatta M. Societal and family change in the burden of care. In: Liu W, Kendig H, editors. Who Should Care for the Elderly? Singapore: Singapore University Press; 2000. p. 27–54.
O'Hara RE, Hull JG, Lyons KD, Bakitas M, Hegel MT, Li Z, et al. Impact on caregiver burden of a patient-focused palliative care intervention for patients with advanced cancer. Palliat Support Care. 2010;8(4):395–404.
Tarlow BJ, Wisniewski SR, Belle SH, Rupert M, Ory MG, Gallagher-Thompson D. Positive aspects of caregiving: contributions of the REACH project to the development of new measures for alzheimer's caregiving. Research on Aging. 2004;26(4):429–53.
Archbold PG, Stewart BJ, Greenlick MR, Harvath T. Mutuality and preparedness as predictors of caregiver role strain. Res Nurs Health. 1990;13(6):375–84.
Carter JH, Stewart BJ, Archbold PG, Inoue I, Jaglin J, Lannon M, et al. Living with a person who has Parkinson’s disease: the spouse’s perspective by stage of disease. Parkinson’s Study Group. Mov Disord. 1998;13(1):20–8.
Zwicker D. Preparedness for Caregiving Scale. New York: New York University, College of Nursing; 2010.
McHugo GJ, Drake RE, Whitley R, Bond GR, Campbell K, Rapp CA, et al. Fidelity outcomes in the National Implementing Evidence-Based Practices Project. Psyciatr Serv. 2007;58(10):1279–84.
Brooks J, McCluskey S, Turley E, King N. The utility of template analysis in qualitative psychology research. Qual Res Psychol. 2015;12(2):202–22.
Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs-principles and practices. Health Serv Res. 2013;48(6 Pt 2):2134–56.
Guetterman TC, Fetters MD, Creswell JW. Integrating quantitative and qualitative results in health science mixed methods research through joint displays. Ann Fam Med. 2015;13(6):554–61.
Wisdom J, Creswell JW. Mixed methods: integrating quantitative and qualitative data collection and analysis while studying patient-centered medical home models. Rockville, MD; 2013 February 2013. Report No.: 13-0028-EF.
Glickman ME, Rao SR, Schultz MR. False discovery rate control is a recommended alternative to Bonferroni-type adjustments in health studies. J Clin Epidemiol. 2014;67(8):850–7.
Ferrell BR, Temel JS, Temin S, Alesi ER, Balboni TA, Basch EM, et al. Integration of Palliative Care Into Standard Oncology Care: American Society of Clinical Oncology Clinical Practice Guideline Update. J Clin Oncol. 2017;35(1):96–112.
Boushon B, Provost L, Gagnon J, Carver P. Using a virtual breakthrough series collaborative to improve access in primary care. Jt Comm J Qual Patient Saf. 2006;32(10):573–84.
Butler A, Canamucio A, Macpherson D, Skoko J, True G. Primary care staff perspectives on a virtual learning collaborative to support medical home implementation. J Gen Intern Med. 2014;29(Suppl 2):S579–88.
Cranley LA, Norton PG, Cummings GG, Barnard D, Estabrooks CA. SCOPE: safer care for older persons (in residential) environments: a study protocol. Implement Sci. 2011;6:71.
Gance-Cleveland B, Aldrich H, Dandreaux D, Oetzel KB, Schmiege S. A virtual childhood obesity collaborative: satisfaction with online continuing education. J Pediatr Health Care. 2015;29(5):413–23.
Gance-Cleveland B, Aldrich H, Schmiege S, Tyler K. Virtual obesity collaborative with and without decision-support technology. Int J Qual Health Care. 2016;28(3):316–23.
Gingold JA, Briccetti C, Zook K, Gillespie CW, Gubernick RS, Moon RY, et al. Context matters: practitioner perspectives on immunization delivery quality improvement efforts. Clin Pediatr (Phila). 2016;55(9):825–37.
John T, Morton M, Weissman M, O'Brien E, Hamburger E, Hancock Y, et al. Feasibility of a virtual learning collaborative to implement an obesity QI project in 29 pediatric practices. Int J Qual Health Care. 2014;26(2):205–13.
McHugh MC, Margolis LH, Rosenberg A, Humphreys E. Advancing MCH interdisciplinary/interprofessional leadership training and practice through a learning collaborative. Matern Child Health J. 2016;20(11):2247–53.
Zubkoff L, Neily J, Mills PD, Borzecki A, Shin M, Lynn MM, Gunnar W, Rosen A. Using a virtual breakthrough series collaborative to reduce postoperative respiratory failure in 16 Veterans Health Administration hospitals. Jt Comm J Qual Patient Saf. 2014;40(1):11–20.
Zubkoff L, Neily J, King B, Dellefield ME, Krein S, Young-Xu Y, Boar S, Mills PD. Virtual breakthrough series, part 1: preventing catheter-associated urinary tract infection and hospital-acquired pressure ulcers in the veterans health administration. Jt Comm J Qual Patient Saf. 2016;42(11):485–98.
Zubkoff L, Neily J, Quigley P, Soncrant C, Young-Xu Y, Boar S, Mills PD. Virtual breakthrough series, part 2: improving fall prevention practices in the veterans health administration. Jt Comm J Qual Patient Saf. 2016;42(11):497–517.
Zubkoff L, Neily J, King B, Morgan S, Young-Xu Y, Boar S, Mills PD. Preventing pressure ulcers in the VHA using a virtual breakthrough series. J Nurs Care Qual. 2016;32(4):301–8.
Aschbrenner K.A., Pratt S.I., Bond GR., Zubkoff L, Naslund J.A., Jue K, et al. A virtual learning collaborative to implement health promotion in routine mental health settings: protocol for a cluster randomized trial. Contemp Clin Trials. 2019;84(105816).
Quill TE, Abernethy AP. Generalist plus specialist palliative care--creating a more sustainable model. N Eng J Med. 2013;368(23465068):1173–5.
Lupu D. American Academy of Hospice and Palliative Medicine Workforce Task Force. Estimate of current hospice and palliative medicine physician workforce shortage. J Pain Symptom Manage. 2010;40(6):899–911.
The research reported in this protocol uses resources of the Dartmouth Hitchcock Medical Center, Dartmouth College, The Norris Cotton Cancer Center, University of Alabama, at Birmingham, University of Rochester, University of Rochester Cancer Center, and University of Rochester Cancer Center NCORP Research Base. Some work for this manuscript was conducted while Dr. Zubkoff was affiliated with the Geisel School of Medicine at Dartmouth, Dartmouth Hitchcock Medical Center, and the White River Junction VA Medical Center.
The study is actively recruiting practices for cohort 1.
Protocol amendments are reviewed and approved by NCI CIRB.
Current, state, federal, and institutional regulations concerning informed consent will be followed. Patients must have capacity to sign an informed consent. Consent documents will be signed by patients and caregivers; copies of the signed documents will be provided to the participants. A waiver of documentation of consent will be used for oncology physicians, ENABLE Nurse Coaches, Coordinators, and Other NCORP staff (e.g., Primary Affiliate PIs, CCDR lead, administrators/supervisors).
All written materials will be kept confidential, locked in the private offices, and limited-access file room of the URCC NCORP Research Base and identified by ID numbers. All electronic information will be kept confidential with password-protected, limited access.
Access to data
Per protocol, all study data will be retained by the URCC NCORP Research Base for 5 years after completion of the study. Designated research personnel, including those at Dartmouth and UAB, is granted access to the data stored in the REDCap and Access databases. After completion of the study and publication of the study results, the de-identified study data will be available on the NCTN/NCORP Data Archive.
Data and safety monitoring plan
The James P Wilmot Cancer Center Data Safety Monitoring Committee (DSMC) will serve as the DSMC of record for this study. The DSMC provides oversight of study progress and safety by review of accrual and events at regularly scheduled meetings. The DSMC will monitor all adverse event rates. DSMC reports will be available for download on the Research Base website.
Provenance and peer review
Not commissioned; peer-reviewed for ethical and funding approval prior to submission.
The funder did not have a role in the design or conduct of the study, and the authors have ultimate authority over study activities. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Cancer Institute.
Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA229197. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of health.
Ethics approval and consent to participate
This study, URCC18110CD, was reviewed and approved by the NCI Central Institutional Review Board (CIRB). Original approval was received on July 2, 2019. Subsequent amendments were approved on October 11, 2019, September 3, 2020, and October 21, 2020. NCI CIRB acts as the single IRB of record for all study sites. The protocol is registered at ClinicalTrials.gov (NCT04062552).
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Zubkoff, L., Lyons, K.D., Dionne-Odom, J.N. et al. A cluster randomized controlled trial comparing Virtual Learning Collaborative and Technical Assistance strategies to implement an early palliative care program for patients with advanced cancer and their caregivers: a study protocol. Implementation Sci 16, 25 (2021). https://doi.org/10.1186/s13012-021-01086-3
- Implementation strategies
- Implementation science
- Early palliative care
- Advanced cancer
- Cluster-randomized controlled trial