Scaling and sustaining COVID-19 vaccination through meaningful community engagement and care coordination for underserved communities: hybrid type 3 effectiveness-implementation sequential multiple assignment randomized trial
Implementation Science volume 18, Article number: 28 (2023)
COVID-19 inequities are abundant in low-income communities of color. Addressing COVID-19 vaccine hesitancy to promote equitable and sustained vaccination for underserved communities requires a multi-level, scalable, and sustainable approach. It is also essential that efforts acknowledge the broader healthcare needs of these communities including engagement in preventive services.
This is a hybrid type 3 effectiveness-implementation study that will include a multi-level, longitudinal, mixed-methods data collection approach designed to assess the sustained impact of a co-created multicomponent strategy relying on bidirectional learning, shared decision-making, and expertise by all team members. The study capitalizes on a combination of implementation strategies including mHealth outreach with culturally appropriate messaging, care coordination to increase engagement in high priority preventive services, and the co-design of these strategies using community advisory boards led by Community Weavers. Community Weavers are individuals with lived experience as members of an underserved community serving as cultural brokers between communities, public health systems, and researchers to co-create community-driven, culturally sensitive public health solutions. The study will use an adaptive implementation approach operationalized in a sequential multiple assignment randomized trial design of 300 participants from three sites in a Federally Qualified Health Center in Southern California. This design will allow examining the impact of various implementation strategy components and deliver more intensive support to those who benefit from it most. The primary effectiveness outcomes are COVID-19 vaccine completion, engagement in preventive services, and vaccine confidence. The primary implementation outcomes are reach, adoption, implementation, and maintenance of the multicomponent strategy over a 12-month follow-up period. Mixed-effects logistic regression models will be used to examine program impacts and will be triangulated with qualitative data from participants and implementers.
This study capitalizes on community engagement, implementation science, health equity and communication, infectious disease, and public health perspectives to co-create a multicomponent strategy to promote the uptake of COVID-19 vaccination and preventive services for underserved communities in San Diego. The study design emphasizes broad engagement of our community and clinic partners leading to culturally sensitive and acceptable strategies to produce lasting and sustainable increases in vaccine equity and preventive services engagement.
ClinicalTrials.gov Identifier: NCT05841810 May 3, 2023
COVID-19 disparities experienced by underserved communities in California
The COVID-19 pandemic has punctuated pervasive health disparities among immigrant, refugee, Latino, Asian, and Black, Indigenous, People of Color (BIPOC) communities [1, 2]. These communities are significantly more likely to experience mortality and morbidity from COVID-19, along with lagging testing and vaccination rates compared to predominantly white communities . Despite the availability of multiple safe and effective vaccines that have been recommended by the Centers for Disease Control and Prevention (CDC), vaccination rates have been slower and variable in BIPOC communities [4, 5].
Although current data on vaccination rates in the United States of America (USA) offer hope towards ending the COVID-19 pandemic (as of this writing), 85% of adults completed the primary vaccine series; 65% of eligible adults have received a booster dose in addition to completing the primary series; and vaccine uptake has lagged across BIPOC communities in San Diego County with only 51% of Black, 69% of Hispanic, and 71% of Asian residents having completed the primary vaccines series . To address persistent inequities, the CDC National Forum on COVID-19 vaccine emphasizes that “barriers to vaccine access must be removed and evidence-based approaches to improving vaccine confidence and acceptance are essential” [7, 8].
Multi-level factors driving vaccine hesitancy and vaccine uptake: individual, structural, systemic, and technological
“Vaccine uptake” is defined as the completion of a vaccination series per administration schedule as recommended by medical experts and appropriate institutions. Vaccine uptake is complex and driven by individual-level factors, structural and systemic factors, and technological factors; these impacts are amplified for underserved communities. Although vaccine uptake has increased, underserved communities have been historically overlooked by public health strategies tailored to resource-rich communities with predominantly English-speaking residents .
At the individual level, the research and popular literature have increasingly focused on the concept of “vaccine hesitancy.” Vaccine hesitancy refers to delay in acceptance or refusal of vaccination despite availability of vaccination services . Vaccine hesitancy is the midpoint in decision-making between vaccine refusal and acceptance and can be influenced by societal/community norms, individual attitudes (e.g., complacency, confidence), and structural factors (e.g., convenience, access) . A related term, “vaccine confidence,” is defined as the belief that vaccines work, are safe, and are part of a trustworthy medical system . Factors driving vaccine hesitancy and confidence include medical distrust shaped by historical trauma rooted in biomedical research and continues to be reinforced by current experiences of discrimination within medical encounters. The COVID-19 pandemic has also prompted a deluge of information, opinion, and conspiracies known as the “infodemic” which has played a significant role in vaccine hesitancy . Rapid and widespread sharing of both evidence-based and inaccurate information about COVID-19 and vaccines leads some people to inaction. This has been particularly salient in Spanish-language social media [14,15,16,17]. Vaccine hesitancy or confidence at the individual level is one, but often not the most, important explanatory variable of vaccine uptake . Additional factors driving uptake are largely related to access, including structural and systemic barriers.
The pandemic has created an overall reduction in timeliness of immunizations and preventive service completion [18, 19] and underscored an urgent need to overcome structural barriers preventing certain populations from accessing high quality, convenient, and affordable healthcare. “Preventive services” are defined as routine, guideline-concordant health care that includes chronic health screenings, check-ups, and patient counseling to prevent illnesses, disease, or other health problems . Vaccine distribution is traditionally embedded within existing healthcare infrastructure that is inequitably designed and inadequately funded. Even among those with a healthcare provider, studies show that BIPOC tend to receive lower-quality healthcare than white people after controlling for insurance status, income, age, and condition severity . As a vaccine comparison, influenza vaccination rates are consistently lower among Black, Latino, and Asian American persons, as well as individuals of lower socioeconomic status (SES), compared with their white and higher SES counterparts . San Diego residents that are particularly impacted by barriers to healthcare access include undocumented residents, isolated community members, individuals whose preferred language is not English, or those who need transportation and access accommodations.
Systemic and logistical factors also impact access to the COVID-19 vaccine. The digital divide (i.e., differential access to technology needed to schedule appointments and obtain updated health information) precludes a sizeable portion of the population from accessing web-based scheduling portals to set up vaccination appointments. Individuals also need to access information in their preferred languages to make informed decisions about vaccination and to schedule vaccination appointments. Compounding these structural factors, concerns about the need to take sick or unpaid leave, or secure childcare, due to vaccine side effects might negatively impact vaccine uptake.
Promise of implementation science in addressing vaccine hesitancy and uptake for underserved communities
Addressing COVID-19 vaccine hesitancy to promote equitable and sustained vaccination for underserved communities requires a multi-level, scalable, and sustainable approach. It is also essential that efforts acknowledge the broader healthcare needs of underserved communities including engagement in preventive services [23,24,25]. The current study builds on the project team’s community-engaged infrastructure to conduct a hybrid type 3 effectiveness-implementation trial [26, 27] to examine the impact of a co-created multicomponent implementation strategy on increased and sustained COVID-19 vaccination and engagement in preventive services for underserved communities within a FQHC network. This work will combine methods from implementation science, health equity research, and community engagement.
This study is a hybrid type 3 effectiveness-implementation trial in which the evidence-based interventions are COVID-19 vaccines and age- and gender-specific recommended preventive care services. The three implementation strategies that support the uptake of these evidence-based interventions are as follows: (1) co-creation with community advisory boards (CABs) led by Community Weavers, (2) the use of mobile health (mHealth) technologies for health communication, and (3) care coordination. The next section provides a brief overview of each implementation strategy.
Co-creation with CABs led by Community Weavers
Meaningful community engagement is an important factor in facilitating the uptake, implementation, and sustained use of interventions designed to address health inequities [28, 29]. Community engagement, however, is not a trivial task, especially when working with communities representing diverse cultural heritages and languages and whose voices have historically been silenced [30, 31]. In the creation of community-vetted solutions to inequitable vaccine distribution and uptake, as well as access to preventive services, it is critical to utilize approaches that acknowledge and address these historical patterns . Community Weavers are individuals with lived experience as a member of an underserved community who serve as cultural brokers between communities, public health systems, and researchers to co-create community-driven, culturally sensitive public health solutions [33, 34]. Community Weavers can bridge the gap between the underserved community and the mainstream public health systems and support research and clinical partners in learning about the priorities, preferences, needs, and strengths of the community, which can lead to solutions that are culturally sensitive and responsive . They can also support the engagement of community members in health care, including COVID-19 vaccination and other preventive services. The Global Action Research Center (ARC), a nonprofit social change organization with expertise conducting participatory action research to address public health needs and a trusted resource for many of the grassroots organizations, will be the key lead in identifying, training, and engaging our Community Weavers in this project. A key component in this process is to engage in co-creation of solutions with members of the community. Co-creationbuilds on the concepts of bidirectional learning and shared decision-making and relies on the expertise brought by all members of the team [35, 36]. Co-creation has been successfully used in diverse contexts, and programs that incorporate co-creation tend to have better uptake, more comprehensive implementation, and lead to more sustained positive outcomes [37,38,39]. Co-creation has been at the heart of our prior community-engaged COVID-19 research with our community and clinical partners to meaningfully engage diverse community members in participatory methods.
Use of mobile health (mHealth) technologies for health communication
Lessons addressing challenges with vaccine distribution and vaccine hesitancy can be drawn from decades of vaccine campaign research from low- or middle-income countries (LMICs) through reverse innovation. Reverse innovation involves replicating an innovation that was originally developed for use in LMIC settings and translating to a high-income country . Traditionally in clinical research, innovation is scaled from high-resource settings to low-resource settings, resulting in predictable challenges with resource limitations impacting feasibility. Reverse innovation allows researchers to derive insights from LMIC implementation research and innovatively adapt for higher-income settings. Given the importance of trust in promoting vaccine uptake, the current study will amplify the voices of trusted local leaders within historically disadvantaged communities. Emerging research illustrates the popularity of mHealth interventions to extend service capacity and reach, promote healthcare engagement, reduce stigma, and promote vaccine acceptance . The use of digital health products has significantly increased during the COVID-19 pandemic era . Important considerations for the implementation of mHealth in real-world settings include technology literacy, personalization of interventions, and perceptions of health risk and status . We will address these considerations to inform the appropriate selection of implementation strategies tailored for context and implementation phase .
In addition to information on vaccine effectiveness and access through mHealth strategies, additional support with more intense coordination might be needed for some community members. Care coordination is an evidence-based practice that is widely recognized as a core component of high-quality care . It is also a pillar of the patient-centered medical home (PCMH), which is a model of primary care increasingly adopted by healthcare systems, including FQHCs, to meet the needs of diverse patients and to improve experiences, outcomes, safety, and system efficiency for patients, providers, and staff members [46, 47]. We define care coordination as “the deliberate organization of patient care activities between two or more participants involved in a patient’s care to facilitate the appropriate delivery of health care services” [48,49,50]. Through personalized health education and care management and collaborative problem-solving, care coordinators will work with patients to develop and achieve a health action plan in collaboration with their primary care physician. The National Committee for Quality Assurance (NCQA) leads the most widely adopted evaluation program for PCMH accreditation, called the NCQA Recognition program . Our FQHC healthcare partner has successfully obtained and maintained NCQA recognition, due, in part, to their care coordination program. This study will expand eligibility for care coordination to target reduction of outstanding preventive services, including but not limited to diabetes and cancer screenings, and influenza vaccination, and offer co-created enhancements to maximize cultural relevance to priority patient communities served by the FQHC.
Taken together, these three implementation strategies — co-creation led by Community Weavers, mHealth, and care coordination — have strong potential to accelerate efforts to sustain COVID-19 vaccination uptake and recommended use of preventive services. Research indicates that social determinants of health, such as low SES, educational level, and access to transportation, along with identifying as members of immigrant, refugee, Latino, and BIPOC communities, reduce one’s likelihood to engage in evidence-based preventive services [52, 53]. As a response to the COVID-19 pandemic, many preventive services were delayed, resulting in more members of underserved communities having overdue preventive service needs . Through a multicomponent health program, COVID-19 vaccination will be coupled with engagement of members of underserved communities in broader preventive services. Engagement in preventive services will be addressed through all components of the program including co-creation of strategies, mHealth outreach, and care coordination. Combining care coordination with culturally relevant, linguistically appropriate, and co-created mHealth health promotion messages offer a powerful approach to both facilitating vaccine equity and preventive services engagement.
Theory of change and implementation science framework
Our study is guided by a theory of change developed by our team through our National Institutes of Health (NIH)-funded research studies and the practical, robust implementation, and sustainability model (PRISM).
Theory of change (ToC)
The multicomponent health program will be informed by an overarching ToC that was developed based on the team’s prior work with two CABs. The goal of the CO-CREATE CAB was to directly inform co-created implementation strategies for a tailored COVID-19 testing program and included 22 members representing community residents, public health researchers, and clinical partners. The goal of the STOP COVID-19 CA CAB was to inform materials and resources needed to support vaccine clinical trial participation and equity initiatives in underserved communities and included 11 community leaders representing ten local grassroots community organizations and two policy partners [23, 24]. ToCs have been broadly used to guide the planning, implementation, and evaluation of public health programs in low-resource settings and provide a comprehensive description of how and why a desired change is expected to happen in a particular context [55, 56]. In these two COVID-19 projects, a rigorous, multistep process was used to create a separate ToC for each CAB  which were overlaid to create an overarching, combined ToC. This combined ToC will be confirmed and refined and used to inform our strategy selection and implementation and evaluation planning.
Practical, Robust Implementation, and Sustainability Model
The Practical, Robust Implementation, and Sustainability Model (PRISM) will guide the optimization, implementation, and evaluation of our multicomponent health program [57, 58]. PRISM allows for multi-level conceptualization of implementation efforts (context domains) and also provides guidance on how to measure relevant proximal implementation outcomes through integration of the RE-AIM measures (reach, effectiveness, adoption, implementation, maintenance) — one of the most widely used sets of implementation measures . PRISM is ideal for guiding our real-world community-based research project because it accounts for multi-level effects; it builds on several implementation science frameworks and can guide development, implementation, and evaluation. Multiple levels of context will be considered: recipient characteristics (community members and delivery agents — Community Weavers and care coordinators), intervention characteristics (perceived by diverse partners), implementation and sustainability infrastructure within communities and clinics, and the external environment (national guidelines and regulations). These factors will be considered during the co-creation and optimization of the multicomponent health program and assessed dynamically across the lifetime of the study. Several studies show that the reach, adoption, implementation, and sustainment of interventions are related to how well the intervention is integrated into the local context [59,60,61]. These factors will be assessed in addition to effectiveness, using the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework.
The specific aims of the study are as follows:
Aim 1: Optimize a multicomponent health program to promote COVID-19 vaccine uptake and engagement in preventive healthcare using our established co-creation approach to address multi-level (individual, community, systemic) barriers to vaccine uptake and preventive services engagement.
Aim 2: Evaluate the implementation, effectiveness, and sustainment of the multicomponent COVID-19 vaccine and preventive services engagement program using a hybrid type 3 effectiveness-implementation sequential multiple assignment randomized trial (SMART) design across immigrant, refugee, Latino, and BIPOC communities in Central, North, and East San Diego.
We will optimize, implement, and test the impact of a multicomponent health program that includes three implementation strategies: co-creation led by Community Weavers, mHealth outreach, and care coordination (described earlier). A participatory approach will be used to engage community members in co-creating and optimizing mHealth outreach and care coordination program components (Aim 1). A hybrid type 3 effectiveness-implementation SMART design will assess the impact of the multicomponent health program on implementation and clinical outcome measures (Aim 2) [62, 63]. See Fig. 1 for the study’s conceptual model.
Through the team’s ongoing community-engaged COVID-19 studies and guided by PRISM, determinants that underlie COVID-19 testing and vaccine disparities in underserved communities (immigrant, refugee, BIPOC, and Latino families) were identified. These determinants include unique explanatory models of health and COVID-19 (intervention PRISM domain), structural and systemic challenges to vaccination and preventive services engagement (implementation and sustainability infrastructure PRISM domain), FDA-approved or authorized COVID-19 vaccines available to all adults (external environment PRISM domain), and the varying needs and characteristics of underserved communities located in our target geographic regions (recipients PRISM domain). It is hypothesized that the selected implementation strategies will exert their impact on implementation and clinical outcomes through vaccine confidence and service navigation support (mechanisms). Implementation outcomes are based on RE-AIM and expected to affect our clinical outcomes (effectiveness).
Clinical outcomes are based on the Office of Disease Prevention and Health Promotion’s Healthy People 2020 objectives . A hybrid type 3 design was selected because there is strong evidence for our clinical interventions (highly effective COVID-19 vaccines and evidence-based preventive services) and effectiveness data for the selected implementation strategies delivered individually. The study’s experimental design follows a SMART randomized structure to introduce the more intensive care coordination component only to those who are not responding to usual care or the lower intensity and less-resource intensive mHealth messaging component allowing for more efficient resource allocation. The Community Weaver-led co-creation component will be included in both mHealth and care coordination since this component is at the heart of the meaningful community engagement and the Community Weaver’s role will be critical for both strategies.
The multicomponent strategy is an adaptive approach that utilizes tailored mHealth messaging and care coordination to increase COVID-19 vaccine uptake. The SMART assignment arms are described in Table 2. The trial will span 4 years inclusive of 3 years of active implementation and 1 year of sustainment/maintenance. As outlined in Fig. 2, both randomization and analysis will be conducted at the individual participant level. At enrollment, participants will be stratified by current COVID-19 vaccination status and status of other preventive services vis-à-vis a risk score and then randomized into the first-stage intervention (mHealth outreach or standard of care). Response or lack of response after the completion of the first stage intervention will determine if participants continue in their initial first-stage intervention or re-randomized to increased intensity intervention. Nonresponders in the standard of care arm will be equally re-randomized into either mHealth outreach or care coordination for stage 2. Nonresponders in mHealth outreach will be re-randomized into mHealth or mHealth plus care coordination for stage 2. Nonresponse will be defined as not being up to date on COVID-19 vaccination OR having at least one outstanding preventive service need.
This study will be conducted in partnership with the second largest FQHC serving low-income and uninsured San Diegans and has a long-standing relationship with UC San Diego. This study will focus on clinics in Central, North, and East San Diego that serve refugee, immigrant, BIPOC, and Latino populations. San Diego has historically been one of the California counties with the highest number of immigrant and refugee families . This FQHC has been successful in the early phases of COVID-19 vaccine campaigns in many regions but anticipates challenges with sustaining the level of support needed to keep up with guidelines as policy changes supporting COVID-19 vaccine efforts may wane in coming months and years. As shown in Table 1, the three target clinics serve a racially/ethnically diverse patient population whose primary preferred languages are Vietnamese (clinic 1), Arabic (clinic 2), and Spanish (clinic 3). Engagement in preventive services as shown by recommending screenings is lower than recommended in all clinics. In addition, the FQHC has a robust care coordination program in place as part of their PCMH accreditation. Taken together, these factors well position the partnering FQHC to collaboratively deliver a multi-level community-focused intervention to support equitable vaccine uptake and preventive services engagement.
A total of 300 adult patients (100 per clinic), from a sampling frame of approximately 5000 patients, will be recruited to participate in the study. Patient recruitment will occur at each of the three participating FQHC clinics through collaborative efforts between the FQHC’s care coordinators and the Community Weavers. The CABs will be invited to co-develop culturally appropriate recruitment methods to ensure participation from Spanish, Arabic, and Vietnamese-speaking patients connected with the partnering FQHCs. A weighted sampling approach will be considered if patient volume varies considerably between the three clinics during the baseline period. Furthermore, to understand reach of the program, characteristics of those participating in the study will be compared to the population in each of the three geographic regions recruited into the study.
Community engagement through CABs and Community Weavers
The multicomponent strategy will be developed and optimized using the team’s established co-creation approach with guidance from the Community Weavers and engagement of the community through CABs. Community Weavers will be hired and supervised by the study’s community partner, the Global ARC, based on their established relationships with ethnically based community organizations. Hiring criteria for Community Weavers include that they are fluent in English and at least one of the three priority languages (Spanish, Arabic, Vietnamese); are engaged with communities living near one of the three partnering FQHC clinics; have experience with community engagement, leading and organizing; and have the capacity to share back knowledge and information with their community as well as share the community’s perspectives and concerns with the CABs.
Community Weavers will have a critical role in developing and refining tailored mHealth messages . They will also be instrumental in facilitating the existing care coordination efforts at the FQHC by focusing on immigrant and refugee communities that can be difficult to engage in care. The Community Weavers will lead three CABs that include five members each inclusive of the three primary languages spoken by community members of our clinics of interest. CAB members will be compensated for their time. In addition to the five community members, we will include the two care coordinators, a mHealth Specialist, and two clinical investigators as nonvoting members in each CAB.
During the first year, each CAB will meet up to four times. These initial meetings will be tasked with co-creating culturally and linguistically appropriate content and frequency for mHealth outreach and identifying topics, strategies, and processes for the care coordination program. mHealth outreach messages will be developed in the native languages of the CABs, and we will use professional translation as needed. To facilitate mHealth message development, Community Weavers will use a storytelling/narrative framework approach inviting residents to join open dialogue and share their journey through the decision-making process towards getting vaccinated, reflecting on the anxieties they navigated, the misinformation they were confronted with, and any structural or technological barriers they had to overcome to enhance the cultural resonance and tailoring of the mHealth messages . In subsequent years, the CABs will continue meeting quarterly to revise and update mHealth content and engage in an Appreciative Inquiry Process to evaluate the implementation of the multicomponent health program and determine when refinements for optimization are needed. The co-creation and optimization of the program components will be guided by the contextual domains of PRISM (i.e., intervention characteristics, recipient characteristics, implementation and sustainability infrastructure, external environment) and will build on the team’s prior work with the community [24, 31] and the CAB members’ lived experience as members of their respective communities.
mHealth development and outreach
mHealth-related communications will include personalized short messaging service (SMS) and automated phone calls recorded by community leaders in their respective languages aligned with the language preferences of our target clinics. The content of mHealth outreach will include information such as how and where to get vaccinated, myths about COVID-19 and other vaccines, the importance, timing, and scheduling of priority preventive services, and provide reminders for clinical care appointments. The study team includes an expert consultant in health communications to guide message development. mHealth distribution frequency will be determined by each CAB to align with their community’s preferences and needs. New messages will be developed and added to the message library based on updated public health guidance and feedback from participants. The mHealth outreach system will be user tested and refined prior to launch. See Fig. 3 for a schematic of the mHealth outreach system.
A mHealth specialist will lead the development of the mHealth delivery system by consulting with programming experts through Twilio to design and build a fully integrated, Health Insurance Portability and Accountability Act (HIPAA) compliant communications platform to manage the distribution of SMS and voice communications. Twilio will allow customized communications, develop production-ready applications, and run serverless applications on one application programming interface-powered platform. A centralized study database connected with the mHealth delivery system will track the number and timing of text and voice messages sent to individual participants. User analytics will be captured such as individual reach, responsiveness, and engagement. Example messages may include the following: “Proteja a sus seres queridos con vacunarse contra el COVID-19”/ “Protect your loved ones by getting the COVID-19 vaccine” and “Recuerde pedir su vacuna contra la Influenza”/ “Remember to ask for your flu shot.”
The partnering FQHC has an established process to provide care coordination (see standard of care in Table 2). The partnering FQHC will identify study-specific care coordinators who are members of the three study communities. Care coordination for the intervention will be tailored to the needs and cultural preferences of the respective patient community through input from our CABs. The content of care coordination will include engagement of patients in preventive services, developing a patient health action plan, support with adhering to the health action plan, scheduling preventive and care management appointments, communicating with the patient’s care team to ensure continuity of care, and COVID-19 vaccination.
Data will be collected using multiple sources to represent diverse perspectives using a QUANT + QUAL concurrent mixed-method design approach. Data sources, type, source, and timing of collection as well as key implementation and effectiveness outcomes are summarized in Tables 3 and 4. Surveys and interviews with participants will be conducted in the participant’s preferred language. Participant surveys will be drafted with input from the CABs, pilot tested prior to implementation, and use electronic or paper formats based on participant preferences. Interviews will be led by Community Weavers using a semi-structured interview guide reflecting PRISM domains adapted from prior studies [58, 72] and vetted by the CABs. Interviews will be recorded and translated to English. Study data for mHealth message delivery, care coordination, and Community Weavers will be stored in Research Electronic Data Capture (REDCap). Periodic reflections will be collected as weekly diary entries from members of the research team. Partner engagement will be evaluated using a concurrent mixed-methods approach developed through the team’s NIH-funded COVID-19 projects . A partner engagement survey (Goodman et al.) will be distributed following each CAB meeting to assess how well and how often eight engagement principles were perceived during the meeting . Ethnographic documentation of the CAB process will be conducted by trained research staff using a structured documentation form identifying various aspects of engagement (e.g., topics covered, time spent contributing) .
Primary effectiveness outcomes will be COVID-19 vaccine (initial and bivalent booster) completion at a sustained rate of 70% (consistent with Healthy People 2020 influenza vaccination goal for adults aged 18 + years) . Secondary effectiveness outcomes will be an increase in preventive service engagement defined as completing a physical exam and at least one recommended preventive care screening (e.g., mammogram, A1c screening, colorectal cancer screening) annually and increases in vaccine confidence. Data on other recommended adult immunizations (e.g., influenza, shingles) will be extracted from patient electronic health records (EHRs). Primary implementation outcomes will be reach, adoption, implementation, and maintenance up to 2-year post-implementation. Target sample size was determined using methods described by Kidwell et al.  for a SMART design with a binary outcome. A power analysis was conducted to inform our targeted sample size based on the following assumptions: (1) probability of success for adaptive health interventions of 0.70; (2) estimated response rate for standard of care of 0.45 (current vaccination rate among target population); and (3) equal re-randomization after first-stage intervention for nonresponders. Allowing for a 5% type-1 error and 80% power, a sample size of at least 252 is needed to evaluate an effect size of 0.25. Assuming a 15% loss to follow-up during the 2-year intervention period, our target enrollment of 300 participants will ensure sufficient sample size for analysis.
Quantitative data from surveys, study databases, and EHRs will be summarized using simple descriptive statistics along with data visualization methods to understand patterns and characteristics of responders and nonresponders and to guide adaptive implementation. Parametric and nonparametric tests will be employed to compare the survey responses between the arms of the trial variables and clinic locations. Factors will initially be assessed at the univariate level and then used to build multivariable models to assess independent associations of multiple factors on vaccination uptake, including community level factors such as prevalence of COVID-19 vaccination, history of COVID-19 infection, and primary community language group. Multinomial logistic regression models will explore the relation between standard of care, mHealth outreach, and mHealth plus care coordination and COVID-19 vaccine uptake, while controlling for a variety of demographic and socioeconomic variables and comorbidities, as well as FQHC clinic characteristics. Additionally, the association between adaptive interventions and the secondary clinical composite outcome, preventive services engagement, including age and gender-specific services will be evaluated. Descriptive analysis will be used to characterize within subgroup differences. Mixed-effects logistic regression models will be used to identify factors associated with COVID-19 vaccine uptake and vaccine confidence using EHR and demographic data collected with participant evaluation surveys.
Qualitative data will be culled from interviews, periodic reflections, and study databases. All qualitative data will be coded using a rapid qualitative data analytic approach guided by the domains of PRISM, informed by prior work of the lead and senior authors across diverse contexts [58, 72, 73]. This approach involves developing a templated matrix of summary responses from interviews, followed by coding by a qualitative analytic team based on pre-defined themes while allowing for new sub-themes to emerge . Group coding for consensus building, team-based review of codes, and double coding to verify acceptable inter-rater reliability will occur throughout the coding process. Community Weavers and clinic partners will be invited to review the coded data to support sense-making and develop action items based on the data.
A matrix approach will be used to triangulate data from quantitative and qualitative sources using the key domains of PRISM. A joint display analysis will be produced to support the integration of data sources [75, 76]. Unique and common themes across language groups will be identified and described.
This implementation study capitalizes on the unique and complementary methods from community engagement, implementation science, health equity, health communication, infectious disease, and public health to co-create a multicomponent strategy to promote sustained uptake of effective public health approaches of COVID-19 vaccination and preventive services engagement for underserved communities in San Diego. This proposal extends the team’s current work [23, 24, 77] using a generalizable co-creation approach to focus on scaling and sustaining COVID-19 and broader health equity for immigrant, refugee, Latino, and BIPOC communities.
This study will contribute to the limited evidence base in implementation science on public health scaling and sustainment strategies by conducting a hybrid type 3 effectiveness-implementation study on a multicomponent strategy co-created with community leaders, residents, and frontline providers. Limited research exists on sustainment strategies so the explicit focus on scaling and sustaining will contribute to the field of implementation science by adding evidence to health equity-promoting sustainment strategies. This study will include a multi-level mixed-methods data collection approach at multiple time points guided by an implementation science model and is designed to assess sustained impact of our multicomponent program with a 12-month follow-up period. An adaptive implementation approach operationalized in a SMART design will be used to allow deployment of various components of our multicomponent program over time depending on the response to prior components. This approach will facilitate understanding the impact of various intervention components and delivering more intensive support those who benefit from it most. Finally, while the mHealth messaging will target COVID-19 vaccines and boosters, generalizable knowledge will be garnered that may support development of messages for other immunizations if this intervention is to be expanded and scaled.
Balanced with these innovative features are potential challenges and limitations. Based on the team’s ongoing COVID-19 work, a primary challenge will be balancing reach with resource constraints. The underserved communities who reside in the partnering FQHC clinic regions are diverse in their cultures, languages, explanatory models of health and COVID-19, and healthcare needs. It will not be possible to engage all immigrant, refugee, and BIPOC communities through our co-created health program. However, the team will rely on the decades of community engaged participatory action research led by the Global ARC to successfully reach a diverse range of immigrant and refugee communities in ways that are meaningful, sustainable and resource appropriate, and leverage the quality care and clinical research supported by the partnering FQHC to reduce health disparities in their patient population.
This study has great potential to contribute to implementation science through the operationalization of a relatively infrequently operationalized SMART design, the use of longitudinal, multi-level, and mixed-methods data collection informed by an implementation science framework, and the use of innovative implementation strategies of co-creation through CABs led by Community Weavers and care coordination. It is expected that the study will have meaningful population health impact by increasing COVID-19 vaccine and preventive services uptake for key historically underserved populations. Furthermore, findings of this study have potential to uniquely unpack the phenomenon of health and vaccine misinformation that was exacerbated through the COVID-19 pandemic . In closing, this study has potential implications beyond a narrow focus on COVID-19 disparities reduction to influence broader public health implementation efforts for underserved communities.
Availability of data and materials
Black, Indigenous, and People of Color
Centers for Disease Control and Prevention
Electronic health records
Federal Drug Administration
Federally Qualified Health Center
- Global ARC:
Global Action Research Center
Health Insurance Portability and Accountability Act
Low- or middle-income country
National Committee for Quality Assurance
National Institutes of Health
Patient-centered medical home
Practical, robust implementation and sustainment model
Reach, effectiveness, adoption, implementation, and maintenance
Research Electronic Data Capture
Sequential multiple assignment randomized trial
Short message/messaging service
Theory of change
Romano SD, Blackstock AJ, Taylor EV, Felix SE, Adjei S, Singleton C-M, et al. Trends in racial and ethnic disparities in COVID-19 hospitalizations, by region - United States, March-December 2020. Morb Mortal Wkly Rep. 2021;70(15):560.
Magesh S, John D, Li WT, Li Y, Mattingly-app A, Jain S, et al. Disparities in COVID-19 outcomes by race, ethnicity, and socioeconomic status. JAMA Network Open. 2021; 4(11). https://doi.org/10.1001/jamanetworkopen.2021.34147.
County of San Diego, Health & Human Services Agency and Live Well San Diego. Epidemiology and Immunization Services Branch (EISB) COVID-19 vaccinations report. SanDiegoCounty.gov. https://www.sandiegocounty.gov/content/dam/sdc/hhsa/programs/phs/Epidemiology/COVID-19%20Vaccinations%20Demographics.pdf. Access 20 April 2023.
Ndugga N, Pham O, Hill L, Artiga S, Haldar W. Latest data on COVID-19 vaccinations race/ethnicity. KFF. 2022 Feb 05. https://www.kff.org/coronavirus-covid-19/issue-brief/latest-data-on-covid-19-vaccinations-by-race-ethnicity/. Accessd 20 April 2023.
Ojikutu BO. Shifting power, creating demand and requiring accountability: achieving vaccine equity in the US. Harvard Health Policy Review. 2021 April 4. http://www.hhpronline.org/articles/2021/4/1/shifting-power-creating-demand-and-requiring-accountability-achieving-vaccine-equity-in-the-us. Accessed 20 Apr 2023.
County of San Diego, Health & Human Services Agency, Public Health Services, COVID-19 Watch. Covid 19 Vaccines Administered Dashboard. SanDiegoCounty.gov. 2023. https://www.sandiegocounty.gov/content/sdc/hhsa/programs/phs/community_epidemiology/dc/2019-nCoV/status/COVID19_Vaccines_Administered_Dashboard.html. Accessed 20 Apr 2023.
Centers for Disease Control and Prevention. COVID-19 vaccinations in the United States. CDC. 2023. https://covid.cdc.gov/covid-data-tracker/#vaccinations_vacc-total-admin-rate-total. Accessed 20 Apr 2023.
Christie A, Mbaeyi SA, Walensky RP. CDC interim recommendations for fully vaccinated people: an important first step. JAMA. 2021;325(15):1501–2.
Elder JP, Ayala GX, Parra-Medina D, Talavera GA. Health communication in the Latino community: issues and approaches. Annu Rev Public Health. 2009;30:227–51.
MacDonald NE. Vaccine hesitancy: definition, scope and determinants. Vaccine. 2015;33(34):4161–4.
Vijaykumar S, Rogerson DT, Jin Y, de Oliveira Costa MS. Dynamics of social corrections to peers sharing covid-19 misinformation on WhatsApp in Brazil. J Am Med Inform Assoc. 2021;29(1):33–42.
Centers for Disease Control and Prevention. What is vaccine confidence? CDC. 2022. https://www.cdc.gov/vaccines/covid-19/vaccinate-with-confidence/building-trust.html. Accessed 20 Apr 2023.
Webb Hooper M, Nápoles AM, Pérez-Stable EJ. No populations left behind: vaccine hesitancy and equitable diffusion of effective COVID-19 vaccines. J Gen Intern Med. 2021;36(7):2130–3.
Sanchez GR, Bennett C. Why Spanish-language mis- and disinformation is a huge issue in 2022. Brookings. 2023. https://www.brookings.edu/blog/fixgov/2022/11/04/why-spanish-language-mis-and-disinformation-is-a-huge-issue-in-2022/. Accessed 20 Apr 2023
Mochkofsky G. The Latinx community and Covid-disinformation campaigns. The New Yorker. 2022. https://www.newyorker.com/news/daily-comment/the-latinx-community-and-covid-disinformation-campaigns. Accessed 20 Apr 2023.
Abrajano MA, Garcia M, Pope A, Vidigal R, Tucker J, Nagler J. Analysis | Latinos who use Spanish-language social media get more misinformation. The Washington Post. WP Company; 2022.https://www.washingtonpost.com/politics/2022/11/08/latino-midterms-votes-misinformation/. Accessed 20 Apr 2023.
Gold A, Contreras R. Spanish-language social media misinformation thrives, raising alarms. Axios. 2022. https://www.axios.com/2022/02/08/social-media-misinformation-latinos. Accessed 20 Apr 2023.
Avalere Health. The COVID-19 pandemic: impact on US adolescent and adult vaccine utilization across marketstients and families in the medical home. Avalere Report. 2021. https://avalere.com/wp-content/uploads/2021/02/The-COVID-19-Pandemic_Impact-on-US-Adolescent-and-Adult-Vaccine-Utilization-Across-Markets_White-Paper.pdf. Accessed 20 Apr 2023.
Hong K, Zhou F, Tsai Y, Jatlaoui TC, Acosta AM, Dooling KL, et al. Decline in receipt of vaccines by medicare beneficiaries during the COVID-19 pandemic—United States, 2020. Morb Mortal Wkly Rep. 2021;70(7):245.
U.S. Centers for Medicare & Medicaid Services. Preventive services. HealthCare.gov. https://www.healthcare.gov/glossary/preventive-services/. Accessed 20 Apr 2023.
Habersaat KB, Jackson C. Understanding vaccine acceptance and demand—and ways to increase them. Bundesgesundheitsblatt. 2020;63(1):32–9.
Bridges KM. Implicit bias and racial disparities in health care. American Bar Association. https://www.americanbar.org/groups/crsj/publications/human_rights_magazine_home/the-state-of-healthcare-in-the-united-states/racial-disparities-in-health-care/. Accessed 20 Apr 2023.
Salgin L, Ayers LO, Burola ML, Engler AM, Osuna A, Gay L, et al. Perceived COVID-19 risk and testing experiences in the San Ysidro U.S./Mexico border region. Transl Behav Med. 2023. https://doi.org/10.1093/tbm/ibac120.
Stadnick NA, Cain KL, Oswald W, Watson P, Ibarra M, Lagoc R, et al. Co-creating a theory of change to advance COVID-19 testing and vaccine uptake in underserved communities. Health Serv Res. 2022;57(Suppl 1):149–57. https://doi.org/10.1111/1475-6773.13910.
Rabin BA, Cain KL, Salgin L, Watson PL, Oswald W, Kaiser BN, et al. Using ethnographic approaches to document, evaluate, and facilitate virtual community-engaged implementation research. BMC Public Health. 2023; 23(1). https://doi.org/10.1186/s12889-023-15299-2.
Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26. https://doi.org/10.1097/MLR.0b013e3182408812.
Curran GM, Landes SJ, McBain SA, Pyne JM, Smith JD, Fernandez ME, et al. Reflections on 10 years of effectiveness-implementation hybrid studies. Frontiers in Health Services. 2022;2. https://doi.org/10.3389/frhs.2022.1053496.
WHAT WORKS increasing appropriate vaccination evidence-based interventions for your community. The Community Guide. 2017. https://www.thecommunityguide.org/sites/default/files/assets/What-Works-Factsheet-Vaccination.pdf. Accessed 20 Apr 2023.
Purnell TS, Fakunle DO, Bone LR, Johnson TP, Hemberger N, Pitts SB, et a. Overcoming barriers to sustaining health equity interventions: insights from the National Institutes of Health Centers for Population Health and Health Disparities. Journal of health care for the poor and underserved. 2019; 30(3): 1212–1236.
Wallerstein N, Duran B. Community-based participatory research contributions to intervention research: the intersection of science and practice to improve health equity. Am J Public Health. 2010;100(S1):S40–6.
Stadnick NA, Cain K, Oswald W, Watson P, Ibarra M, Lagoc R, et al. Engaging underserved communities in COVID-19 health equity implementation research: an analysis of community engagement resource needs and costs. Frontiers in Health Services. 2022; 2.
Harrington C, Erete S, Piper AM. Deconstructing community-based collaborative design: towards more equitable participatory design engagements. Proceedings of the ACM on Human-Computer Interaction. 2019;3:1–25.
The National Academies of Sciences, Engineering, Medicine. Community based solutions to promote health equity in the United States. https://www.nationalacademies.org/our-work/community-based-solutions-to-promote-health-equity-in-the-united-states (2017). Accessed 20 Apr 2023.
GentleWays. Leaders as weavers and facilitators: for a culture of inclusion. Gentle Ways For Our Planet. 2022. https://www.gentlewaysforourplanet.org/leaders-as-weavers-and-facilitators-for-a-culture-of-inclusion/. Accessed 20 Apr 2023.
Springer AE, Evans AE, Ortuño J, Salvo D, Arevalo MT. Health by design: interweaving health promotion into environments and settings. Front Public Health. 2017;5:268.
Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving research impact through co-creation in community-based health services: literature review and case study. Milbank Q. 2016;94(2):392–429.
Arnold M. Fostering sustainability by linking co-creation and relationship management concepts. J Clean Prod. 2017;140:179–88. https://doi.org/10.1016/j.jclepro.2015.03.059.
Mauser W, Klepper G, Rice M, Schmalzbauer BS, Hackmann H, Leemans R, et al. Transdisciplinary global change research: the co-creation of knowledge for sustainability. Current Opinion in Environmental Sustainability. 2013;5(3–4):420–31. https://doi.org/10.1016/j.cosust.2013.07.001.
Sharma P. Customer co-creation, COVID-19 and sustainable service outcomes. Benchmarking: an international journal. 2021;28(7):2232–58. https://doi.org/10.1108/bij-10-2020-0541.
Zedtwitz MV, Corsi S, Søberg PV, Frega R. A typology of reverse innovation. J Prod Innov Manag. 2014;32(1):12–28. https://doi.org/10.1111/jpim.12181.
Daley MF, Narwaney KJ, Shoup JA, Wagner NM, Glanz JM. Addressing parents’ vaccine concerns: a randomized trial of a social media intervention. Am J Prev Med. 2018;55(1):44–54. https://doi.org/10.1016/j.amepre.2018.04.010.
Sorkin DH, Janio EA, Eikey EV, Schneider M, Davis K, Schueller SM, et al. . Rise in use of digital mental health tools and technologies in the US during the COVID-19 pandemic. Journal of Medical Internet Research. 2021; 23(4). https://doi.org/10.2196/26994.
Borghouts J, Eikey E, Mark G, De Leon C, Schueller SM, Schneider M, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. Journal of Medical Internet Research. 2021;23(3). https://doi.org/10.2196/24387.
Graham AK, Lattie EG, Powell BJ, Lyon AR, Smith JD, Schueller SM, et al. Implementation strategies for digital mental health interventions in health care settings. Am Psychol. 2020;75(8):1080–92. https://doi.org/10.1037/amp0000686.
Tricco AC, Antony J, Ivers NM, Ashoor HM, Khan PA, Blondal E, et al. Effectiveness of quality improvement strategies for coordination of care to reduce use of health care services: a systematic review and meta-analysis. Canadian Medical Association Journal. 2014;186(15). https://doi.org/10.1503/cmaj.140289.
Jackson GL, Powers BJ, Chatterjee R, Prvu Bettger J, Kemper AR, Hasselblad V, et al. The patient-centered medical home. Ann Intern Med. 2013;158(3):169. https://doi.org/10.7326/0003-4819-158-3-201302050-00579.
Scholle SH, Torda P, Peikes D, Han E, Genevro J. Engaging patients and families in the medical home. Agency for Heakthcare Research and Quality. 2010. https://pcmh.ahrq.gov/page/engaging-patients-and-families-medical-home-table-1. Accessed 20 Apr 2023
Moreno L, Peikes D, Krilla A. Necessary but not sufficient: the HITECH Act and Health Information Technology’s Potential to Build Medical Homes. Agency for Healthcare Research and Quality. 2010. https://pcmh.ahrq.gov/page/necessary-not-sufficient-hitech-act-and-health-information-technologys-potential-build-medic-1. Accessed 20 Apr 2023
Stange KC, Nutting PA, Miller WL, Jaén CR, Crabtree CF, Flocke SA. Defining and measuring the patient-centered medical home. J Gen Intern Med. 2010;25(6):601–12. https://doi.org/10.1007/s11606-010-1291-3.
McDonald KM, Sundaram V, Bravata DM, Lewis R, Lin N, Kraft SA, et al. Closing the quality gap: a critical analysis of quality improvement strategies.Rockville, MD: Agency for Healthcare Research and Quality; 2007. Report No.: 04(07)-0051–7.
Patient-Centered Medical Home (PCMH). NCQA. https://www.ncqa.org/programs/health-care-providers-practices/patient-centered-medical-home-pcmh(2021). Accessed 20 Apr 2023.
Carey TS, Bekemeier B, Campos-Outcalt D, Koch-Weser S, Millon-Underwood S, Teutsch S. National Institutes of Health Pathways to Prevention Workshop: Achieving Health Equity in preventive services. Ann Intern Med. 2020;172(4):272. https://doi.org/10.7326/M19-3171.
Nelson HD, Cantor A, Wagner J, Jungbauer R, Quiñones A, Stillman L, et al. Achieving health equity in preventive services: A systematic review for a National Institutes of Health Pathways to Prevention Workshop. Ann Intern Med. 2020;172(4):258. https://doi.org/10.7326/M19-3199.
Nodora JN, Gupta S, Howard N, Motadel K, Propst T, Rodriguez J, et al. The COVID-19 pandemic: identifying adaptive solutions for colorectal cancer screening in underserved communities. JNCI: Journal of the National Cancer Institute. 2020;113(8):962–8. https://doi.org/10.1093/jnci/djaa117.
Coryn CL, Noakes LA, Westine CD, Schröter DC. A systematic review of theory-driven evaluation practice from 1990 to 2009. Am J Eval. 2010;32(2):199–226. https://doi.org/10.1177/1098214010389321.
Anderson, A. The community builder’s approach to theory of change: a practical guide to theory development. Aspen Institute Roundtable on Community Change website. 2005. http://www.theoryofchange.org/pdf/TOC_fac_guide.pdf. Accessed 20 Apr 2023.
Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. The Joint Commission journal on Quality and Patient Safety. 2008;34(4):228–43. https://doi.org/10.1016/s1553-7250(08)34030-6.
McCreight MS, Rabin BA, Glasgow RE, Ayele RA, Leonard CA, Gilmartin HM, et al. Using the practical, robust implementation and sustainability model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Translational Behavioral Medicine. 2019;9(6):1002–11. https://doi.org/10.1093/tbm/ibz085.
Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Frontiers in Public Health. 2019;7. https://doi.org/10.3389/fpubh.2019.00064.
Lundmark R, Hasson H, Richter A, Khachatryan E, Åkesson A, Eriksson L. Alignment in implementation of evidence-based interventions: a scoping review. Implementation Science. 2021;16(1). https://doi.org/10.1186/s13012-021-01160-w.
Holtrop JS, Estabrooks PA, Gaglio B, Harden SM, Kessler RS, King DK, et al. Understanding and applying the RE-AIM framework: clarifications and resources. Journal of Clinical and Translational Science. 2021;5(1). https://doi.org/10.1017/cts.2021.789.
Landes SJ, McBain SA, Curran GM. An introduction to effectiveness-implementation hybrid designs. Psychiatry Research. 2019;280:112513. https://doi.org/10.1016/j.psychres.2019.112513.
Kilbourne AM, Smith SN, Choi SY, Koschmann E, Liebrecht C, Rusch A, et al. Adaptive school-based implementation of CBT (ASIC): clustered-smart for building an optimized adaptive implementation intervention to improve uptake of mental health interventions in schools. Implement Sci. 2018;13(1). https://doi.org/10.1186/s13012-018-0808-8.
Healthy People 2020. https://www.healthypeople.gov/2020/(2021). Accessed 20 Apr 2023.
Refugee arrival data. Health&Human Services Agency. https://www.sandiegocounty.gov/content/sdc/hhsa/programs/sd/community_action_partnership/OfficeofRefugeeCoord2.html. Accessed 20 Apr 2023.
Vu CM, Nguyen D, Tanh DB, Chun J. Case study of an ethnic community-based organization in the United States. Nonprofit Volunt Sect Q. 2016;46(1):199–217. https://doi.org/10.1177/0899764016654220.
Lee H, Fawcett J, DeMarco R. Storytelling/narrative theory to address health communication with minority populations. Appl Nurs Res. 2016;30:58–60. https://doi.org/10.1016/j.apnr.2015.09.004.
Salgin L, Cain K, Stadnick NA, Watson P, Oswald W, Kaiser B, et al. Virtual ethnographic approaches to facilitate community engaged implementation research. Poster presentation at the Colorado Pragmatic Research in Health Conference; 2021 May.
Goodman MS, Ackermann N, Bowen DJ, Thompson V. Content validation of a quantitative stakeholder engagement measure. J Community Psychol. 2019;47(8):1937–51. https://doi.org/10.1002/jcop.22239.
Immunization and infectious diseases. Healthy People 2020. https://www.healthypeople.gov/2020/topics-objectives/topic/immunization-and-infectious-diseases/objectives. Accessed 20 Apr 2023.
Kidwell KM, Seewald NJ, Tran Q, Kasari C, Almirall D. Design and analysis considerations for comparing dynamic treatment regimens with binary outcomes from sequential multiple assignment randomized trials. J Appl Stat. 2017;45(9):1628–51. https://doi.org/10.1080/02664763.2017.1386773.
Pittman JO, Lindamer L, Afari N, Depp C, Villodas M, Hamilton A, et al. Implementing escreening for suicide prevention in VA post-9/11 transition programs using a stepped-wedge, mixed-method, hybrid type-II implementation trial: a study protocol. Implementation Science Communications. 2021;2(1). https://doi.org/10.1186/s43058-021-00142-9
Stadnick NA, Brookman-Frazee L, Mandell DS, Kuelbs CL, Coleman KJ, Sahms T, et al. A mixed methods study to adapt and implement integrated mental healthcare for children with autism spectrum disorder. Pilot and Feasibility Studies. 2019;5(1). https://doi.org/10.1186/s40814-019-0434-5
Hamilton A. Rapid qualitative analysis. HSRD Veterans Affairs Seminars. 2021. https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/3846-notes.pdf. Accessed 20 Apr 2023.
Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs-principles and practices. Health Services Research. 2013;48(6pt2):2134–56. https://doi.org/10.1111/1475-6773.12117.
Guetterman TC, Fetters MD, Creswell JW. Integrating quantitative and qualitative results in health science mixed methods research through joint displays. Ann Family Med. 2015;13(6):554–61. https://doi.org/10.1370/afm.1865.
Casillas A, Rosas LG, Carson SL, Orechwa A, North G, AuYoung M, et al. STOP COVID-19 CA: community engagement to address the disparate impacts of the COVID-19 pandemic in California. Front Health Serv. 2022;2:935297. https://doi.org/10.3389/frhs.2022.935297.
Knudsen J, Perlman-Gabel M, Uccelli IG, Jeavons J, Chokshi DA. Combating misinformation as a core function of public health. NEJM Catalyst Innovations in Care Delivery. 2023;4(1):CAT-22. https://doi.org/10.1056/CAT.22.0198
We would like to extend our gratitude for the time and wisdom shared by our partners at the Global Action Research Center (Marina Castelo, Raphael Lagoc) and the participating Federally Qualified Health Center (Sharon Velasquez, Noora Hammi, Nikol Lang).
This study was funded through the National Institute of Minority Health and Health Disparities R01 MD017222.
Ethics approval and consent to participate
This study obtained ethics approval under a protocol approved by the University of California San Diego Institutional Review Board (no. 807142).
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Rabin, B.A., Cain, K.L., Watson, P. et al. Scaling and sustaining COVID-19 vaccination through meaningful community engagement and care coordination for underserved communities: hybrid type 3 effectiveness-implementation sequential multiple assignment randomized trial. Implementation Sci 18, 28 (2023). https://doi.org/10.1186/s13012-023-01283-2
- Implementation science
- Hybrid type 3 effectiveness-implementation study
- Sequential multiple assignment randomized trial
- Underserved communities
- Health equity
- Care coordination