Skip to main content

Cluster randomized trial of a multilevel evidence-based quality improvement approach to tailoring VA Patient Aligned Care Teams to the needs of women Veterans



The Veterans Health Administration (VA) has undertaken a major initiative to transform care through implementation of Patient Aligned Care Teams (PACTs). Based on the patient-centered medical home (PCMH) concept, PACT aims to improve access, continuity, coordination, and comprehensiveness using team-based care that is patient-driven and patient-centered. However, how VA should adapt PACT to meet the needs of special populations, such as women Veterans (WVs), was not considered in initial implementation guidance. WVs’ numerical minority in VA healthcare settings (approximately 7–8 % of users) creates logistical challenges to delivering gender-sensitive comprehensive care. The main goal of this study is to test an evidence-based quality improvement approach (EBQI) to tailoring PACT to meet the needs of WVs, incorporating comprehensive primary care services and gender-specific care in gender-sensitive environments, thereby accelerating achievement of PACT tenets for women (Women’s Health (WH)-PACT).


EBQI is a systematic approach to developing a multilevel research-clinical partnership that engages senior organizational leaders and local quality improvement (QI) teams in adapting and implementing new care models in the context of prior evidence and local practice conditions, with researchers providing technical support, formative feedback, and practice facilitation. In a 12-site cluster randomized trial, we will evaluate WH-PACT model achievement using patient, provider, staff, and practice surveys, in addition to analyses of secondary administrative and chart-based data. We will explore impacts of receipt of WH-PACT care on quality of chronic disease care and prevention, health status, patient satisfaction and experience of care, provider experience, utilization, and costs. Using mixed methods, we will assess pre-post practice contexts; document EBQI activities undertaken in participating facilities and their relationship to provider/staff and team actions/attitudes; document WH-PACT implementation; and examine barriers/facilitators to EBQI-supported WH-PACT implementation through a combination of semi-structured interviews and monthly formative progress narratives and administrative data.


Lack of gender-sensitive comprehensive care has demonstrated consequences for the technical quality and ratings of care among WVs and may contribute to decisions to continue use or seek care elsewhere under the US Affordable Care Act. We hypothesize that tailoring PACT implementation through EBQI may improve the experience and quality of care at many levels.

Trial registration, NCT02039856


Women Veterans’ (WVs) numerical minority in Veterans Health Administration (VA) healthcare settings has created logistical challenges to delivering gender-sensitive comprehensive primary care (PC). Women commonly must access an array of VA and non-VA providers, usually requiring multiple separate visits, to achieve the same basic level of care which male veterans can achieve through a single on-site PC visit [13]. The VA has invested in a variety of resources intended to improve WVs’ care, establishing women’s health clinics, designating Women’s Health (WH) PC providers with the requisite training/experience, and contracting with non-VA obstetrics and gynecology services. Despite these investments, WVs’ quality of care in VA continues to lag behind that of male veterans [4]. Contributing to these outcomes is the lack of gender sensitivity prevalent in many VA care settings, which may be linked to low WV retention rates in VA care [5, 6].

The VA has undertaken a major initiative to transform care through mandated implementation of Patient Aligned Care Teams (PACTs), which hold promise for addressing many of the gaps in WVs’ care. The PACT model is based on the concept of patient-centered medical homes (PCMHs), widely endorsed by PC professional societies and shown to improve quality of care and patient, provider, and staff satisfaction, while reducing costs [7, 8]. PACT focuses on development of high-performing “teamlets” comprised of PC providers, nurses, and administrative support who together manage care of a defined panel of patients. These teamlets operate within a larger team that includes, for example, pharmacists, social workers, mental health (MH) providers, and dietitians, and link to specialists and hospital care in their medical “neighborhood.” Through these teams/teamlets, PACT aims to achieve improvements in accessibility, continuity, coordination, and comprehensiveness using team-based care that is patient-driven and patient-centered [9]. These improvements, in turn, should translate into better chronic illness care and prevention and lower costs.

However, how VA should adapt this major reorganization to meet the needs of special populations, such as WVs, is yet to be fully worked out. The PACT model itself does not include specific accommodations for gender-specific care or improved gender sensitivity. Current WV care is also out of step with PACT priorities and emphasis on “one-stop shopping” for care [10]. For example, WVs are more likely to be outsourced to the community for gender-specific services now than they were 10 years ago, and, while the number of women’s clinics is up, over 40 % do not deliver comprehensive PC [3]. Therefore, improving VA PC alone or “beefing up” women’s clinics is unlikely to achieve what an integrated WH-PACT model must reconcile to improve care for WVs and thereby reduce persistent gender disparities in VA care [11, 12].

Research aims

We propose to use evidence-based quality improvement (EBQI) in the context of the Chronic Care Model to develop and test achievement of WH-PACT in a cluster randomized trial [13, 14]. EBQI is a systematic approach to developing local research-clinical partnerships to produce tailored, evidence-based care models or redesigns [15]. The resulting WH-PACT redesign will make use of local WH resources (e.g., women’s clinics, designated providers), while linking them to the broader PACT initiative and medical center resources. We will also explore the extent to which receiving care that meets WH-PACT tenets translates into higher value (better quality, lower costs) for individual WVs, evaluate local implementation, and develop tools for sustaining and spreading WH-PACT.

Our aims are:

  1. 1.

    To assess the effectiveness of EBQI for developing a WH-PACT model using a cluster randomized trial design. WH-PACT model achievement includes (a) PACT features (accessible, continuous, coordinated, team-based, patient-driven, and patient-centered), (b) comprehensive WH care (PC, gender-specific care, and integrated MH), and (c) gender-sensitive care delivery. We will (a) survey providers/staff on achievement of WH-PACT model attributes, (b) survey WV patients on WH-PACT model care experiences, and (c) analyze WH-PACT achievement (e.g., continuity) using secondary data.

  2. 2.

    To examine impacts of receipt of WH-PACT concordant care on WVs’ outcomes.

    We will explore impacts on quality of chronic disease care and prevention, health status, utilization and costs.

  3. 3.

    To evaluate the processes of EBQI-supported WH-PACT implementation. We will assess pre-post practice contexts; document EBQI activities and their relationship to provider/staff and team actions and attitudes; document WH-PACT implementation; and examine barriers and facilitators to EBQI-supported WH-PACT implementation using mixed methods (e.g., semi-structured interviews, brief progress narratives).

  4. 4.

    To develop implementation and evaluation tools for use in EBQI-supported WH-PACT model adaptation, implementation, sustainability, and spread to additional VA facilities.


Setting, site selection, trial design, and participants

The VA healthcare system is currently organized into 21 regional Veterans Integrated Service Networks (VISNs), with administrative and clinical authority over VA medical centers (VAMCs) and their affiliated programs in geographically distinct regions of the USA [16].

Study sites are members of the VA Women’s Health Practice-Based Research Network (WH-PBRN), a 37-VAMC network spanning 17 VISNs and comprising over 270 geographically distinct sites of care and nearly 600 designated WH providers [17]. Together, PBRN facilities serve over 100,000 WVs (about one third of WVs seen in VA) and span diverse patient populations. We identified five VISNs with three or more WH-PBRN sites; we excluded one that was already participating in a VISN-wide EBQI stepped-wedge trial. We approached leadership at the other four VISNs and their WH-PBRN member site leads about study participation. All agreed to participate. One VISN dropped out ahead of randomization, which was replaced with a VISN that had two WH-PBRN sites; we then worked with VISN leadership to identify a third non-PBRN VAMC, which subsequently joined the WH-PBRN (since the trial began, the WH-PBRN has expanded to 60 VAMCs in 20 of 21 VISNs). VAMCs participating in the trial span nine states (Connecticut, Illinois, Iowa, Massachusetts, Minnesota, North Dakota, Pennsylvania, West Virginia, and Wisconsin).

This study is designed as a parallel two-arm, cluster randomized controlled trial (cRCT), blocked on VISN (Fig. 1). We randomly assigned the 12 VAMCs to EBQI or usual PACT implementation in an unbalanced 2:1 ratio within VISN, supporting appraisal of variations in EBQI implementation in the context of differences of VISN geography, resources, and oversight (Fig. 1). No site stratification or matching criteria were used. The study biostatistician used with a randomized permuted block of three (simple block of three VAMCs within each VISN) and a seed of 15,356 to start the random allocation sequence. The eight intervention VAMCs will engage in EBQI, while the four control VAMCs will receive standard PACT and WH care delivery handbooks and guidance that all VA facilities receive. The study biostatistician assigned the VAMCs to EBQI or control, while the study PI (EMY) enrolled and launched EBQI with the resulting eight VAMCs.

Fig. 1
figure 1

CONSORT flow diagram for cluster randomized controlled trial of evidence-based quality improvement

Local quality improvement (QI) teams, their WH and PC leaders, and VAMC facility leadership, as well as VISN leaders and other key stakeholders (e.g., national VA partners), will be the focus of this study, given their varying levels of responsibility for WH, primary care, and PACT implementation. Outcomes for the PC providers and WVs seen in each participating VAMC’s primary care/PACT programs, whether sited in a women’s clinic or general primary care clinic, will also be examined.

Ethical approval and informed consent

The VA supports a large number of Institutional Review Boards (IRBs) across the USA among chiefly academically affiliated VAMCs, in addition to a Central IRB (cIRB) for multisite research. While this study represents a 12-VAMC cluster randomized trial, the cIRB determined that participating sites were not engaged in research (i.e., the intervention is QI, and all research activities are performed centrally at the Principal Investigator’s site in the VA Greater Los Angeles (GLA) Healthcare System), permitting the use of the local GLA IRB. Evaluation and data collection activities are submitted serially to the GLA IRB for review as discrete human subject research component projects (e.g., patient survey component, provider/staff survey component). Two components involved contractors at local affiliates (teamlet interviews with UCLA, provider/staff survey with RAND Health), whose IRBs also reviewed and approved their respective activities.

The study is registered with (NCT02039856).

VA partnerships

This study is one of the five inter-related projects that together comprise a WH-focused research-clinical partnership funded by VA Health Services Research and Development (HSR&D) Service under the Collaborative Research to Enhance and Advance Transformation and Excellence (CREATE) initiative [18]. Central to CREATE is the involvement of policy and/or practice partners in the design, conduct, and dissemination of study results, with an emphasis on implementation and spread of successful strategies. Within the VA Office of Patient Care Services, Women’s Health Services (WHS) is the WH CREATE’s primary partner, with close involvement of Mental Health Services (MHS) in assessing within- and across-project mental health analyses and themes. The WH CREATE is also advised by an Executive Steering Committee, comprised of WHS and MHS partners, representatives in national VA clinical quality reporting, regional network leadership, public affairs, WH clinical care delivery, WH policy, and implementation science and economics, as well as WVs who use the VA for care.

Conceptual framework for application of EBQI

This study is guided conceptually and practically by the Chronic Care Model (CCM) [19]. Developed more than a decade ago, the CCM has been widely adopted to help guide clinical QI initiatives in the context of practices or teams in the USA and abroad [20]. The CCM depicts the health system linked with complementary community resources, while providers and teams within each healthcare organization (regardless of size) aim to deliver care that is characterized by consistent, evidence-based assessment, treatment, and follow-up, with clinical decision support (often via information technology) and support for patient self-management [21]. Applied to EBQI, local QI teams work on improvement plans that consider one or more CCM elements.

The CCM is a particularly strong fit for QI around medical homes, with an easy crosswalk to PACT goals. For example, for care coordination, PACT teamlets should link patients with community resources to facilitate referrals and respond to social service needs; provide care management services for high-risk patients; integrate behavioral health and specialty care through structured collaboration, co-location, or referral protocols; track and support patients when they obtain services outside the practice; follow up with patients within a few days of an emergency room visit or hospital discharge; and communicate test results and care plans to patients/families [22].

Addressing CCM elements, however, requires substantial stakeholder buy-in, local knowledge and skills in QI, availability of needed technical support, and continual guidance. Previous research has shown that evidence-based programs require adaptation to organizational values, needs, and resources prior to dissemination [23]. More structured than continuous quality improvement (CQI), which has had mixed results, EBQI is a systematic approach to developing a multilevel research-clinical partnership approach to QI, using top-down/bottom-up features to engage senior organizational leaders and local QI teams in implementing improvements in the context of prior evidence, provider behavior change methods, and local practice structure and resources [24]. National strategic directives serve as guides, while regional expert panels set innovation design priorities [25]. Local interdisciplinary QI teams design and implement local activities, while researchers serve as technical experts and guides. EBQI also uses team-based CQI methods to help teams structure their aims and measures and conduct plan-do-study-act (PDSA) cycles, in addition to convening topic-focused workgroups with research/clinical expertise with periodic across-site meetings for training and sharing data and lessons learned. EBQI’s value-added contribution is an emphasis on (a) applying objective evidence, with (b) theory review and synthesis integrated into aspects of innovation design and implementation, (c) valid and reliable measurement, and (d) formal measurement feedback to stakeholders at all levels [26]. Effectively applied to a series of VA implementation studies [2729], EBQI uses well-established implementation strategies, such as local priority setting among key stakeholders, adaptation of the evidence to local context (practice tailoring), audit-and-feedback of QI data to support rapid cycle improvements, and practice facilitation to support implementation into practice [30].

EBQI implementation strategies applied to WH-PACT and hypotheses

In this study, we propose to use EBQI to convene multilevel stakeholder panels (presenting panels with evidence on factors associated with improved WH care in the context of national VA WH policy); facilitate local practice QI team design meetings, while providing QI training/education and iterative QI data feedback; and sponsor within and across-practice QI collaboration calls (Fig. 2). We will continually foster coverage of CCM elements.

Fig. 2
figure 2

Evidence-based quality improvement (EBQI) conceptual model

We hypothesize that the initial results of EBQI will be the specification of the design choices for WH-PACT implementation, conduct of local QI activities, and improved provider/staff (and team) QI orientation, WH knowledge and attitudes, and gender awareness. We further hypothesize that EBQI will lead to higher achievement of WH-PACT (more practices achieve more features) in EBQI vs. control practices (aim #1). We anticipate that practice context (e.g., local leadership support, pre-EBQI factors) will moderate the relationships between EBQI and implementation of WH-PACT. We will then explore the extent to which receipt of WH-PACT concordant care (PACT features, comprehensive WH care, gender sensitive care delivery) is associated with improved WH-PACT outcomes (aim #2).

For aim #3 (processes), we will document EBQI activities among intervention practices and examine barriers and facilitators to WH-PACT achievement in all participating practices. For aim #3, we will also rely on Diffusion of Innovation theory [31, 32] to frame our assessment of EBQI-supported WH-PACT adaptation, implementation, and sustainability, considering, for example, the trialability, complexity, and relative advantage of EBQI-supported WH-PACT, as well as concordance between WH-PACT achievement and CCM principles. Aim #4 (tools) will build on our conceptual model and benefit from our practical experience implementing evidence-based care models [15].

In this study, EBQI implementation will focus on six main activities: (1) conduct of four VISN-level interdisciplinary stakeholder planning meetings using expert panel techniques to come to consensus on PACT QI priorities for women Veterans (“QI roadmaps”); (2) development and training of a local QI champion and QI team members at EBQI-assigned VAMCs to pursue one or more QI projects from the “roadmaps”; (3) formative feedback from patient, provider, staff, and practice survey data; key stakeholder and teamlet interview data; and utilization and cost-related administrative data; (4) ongoing practice facilitation and expert review and feedback on local QI proposals and progress; (5) monthly across-intervention VAMC calls to facilitate collaboration and spread of effective QI innovations; and (6) technical work groups designed to provide additional evidence-based support in priority areas (Table 1).

Table 1 Core components of evidence-based quality improvement (EBQI) implementation strategy


As shown in Fig. 3, we have planned a comprehensive approach to evaluating the processes and outcomes of EBQI for tailoring PACT to the needs of WVs. Baseline data collection (as well as 12-month follow-up in the case of the patient surveys) will be used for formative feedback to EBQI-assigned VAMCs. Table 2 provides an overview of the data sources, samples, and measures planned for each evaluation component.

Fig. 3
figure 3

Evaluation components of the Women’s Health PACT trial. Legend: PC primary care, WH women’s health, CATI computer-assisted telephone interviews

Table 2 Formative and summative evaluation data sources, samples, and measures

Sample size calculation and power analysis

We based our power calculations on the study’s unequal (2:1) intervention-to-control ratio and clustered sample design. To detect a moderate (0.20) effect size between WVs at intervention and control sites, we considered data from the Primary Care Satisfaction Survey for Women’s Care Coordination and Comprehensiveness scale, [33] with 12 practices (8 intervention-to-4 control sites), cluster adjustment (ICC = 0.03) [34], and two-tailed 5 % significance level. Based on these parameters, we will target enrollment of a minimum of 40 WVs at each of the 12 study sites over the 2-year period (baseline to 24-month follow-up), for a total of 480 WVs who complete a baseline and at least one of the two follow-up surveys (12- and/or 24-month). To achieve this target, we will create a baseline sample of 6665 WVs who have received PC or WH care at one of the study’s 12 participating sites during the 12-month period prior to the baseline survey (555 per site). Allowing for a conservative 20 % response rate and 40 % attrition over the 2-year period, we aim to enroll 111 WVs at each of the 12 sites at baseline, for a total of 1333 interviews at the completion of the first wave.

Primary outcome measures

The primary outcome measure, focused on achievement of the PACT model of care for WVs, will include changes in levels of achievement of individual PACT features: access, continuity, care coordination, team-based care, patient-driven care, patient-centeredness, comprehensiveness (including gender-specific services and integrated MH), and gender sensitivity (Fig. 2). These measures will be examined at the practice, provider/staff, and patient levels and investigated qualitatively using the key stakeholder and teamlet interviews.

Secondary outcome measures

Secondary outcome measures include WH-PACT outcomes that should result from achievement of the full PACT model, including improvements in quality of care and health status, and reductions in utilization and costs. For quality of care, we will examine chronic disease quality (e.g., process measures such as foot sensation exams or eye exams among diabetics) and preventive practice (e.g., women’s breast and cervical cancer screening and gender-neutral prevention, such as influenza immunizations and colorectal cancer screening). We will determine whether total annual costs per patient and utilization by type of care (e.g., primary care, specialty care, hospitalizations, emergency department visit rates) have changed in EBQI vs. routine PACT implementation VAMCs and based on level of WH-PACT achievement. We anticipate, however, that acute and emergency care may be too rare in our practice populations to have the power to detect differences over time, so these analyses will be exploratory.

We will also examine impacts on provider and staff WH knowledge and attitudes, QI orientation/culture, and gender awareness, as well as changes in practice contextual factors as a result of EBQI exposure (e.g., changes in leadership support, local resources applied to WH and primary care QI, training).

Statistical analyses

We will use multivariate regression to determine EBQI effectiveness, adjusting for covariates, clustering, and enrollment and attrition weights at the patient and provider/staff levels. The primary regressor of interest is being at an EBQI vs. routine PACT implementation site. We will examine the potential moderating effects of practice context and provider/staff knowledge/attitudes. Patient-level controls will include sociodemographic characteristics, health status, comorbidity, and utilization. Adjustment for clustering will be performed using Stata v13. We will evaluate goodness-of-fit using Mallow’s statistic (Cp). We will use multiple imputation methods to address missing data patterns among covariates, although CATI procedures will significantly reduce item non-response in the patient survey, and the web-survey methods will do the same for the provider/staff surveys. Hot deck imputation will be used for imputation of missing values within scales as needed. To address the potential for response bias, patient and provider/staff survey data will be weighted to the inverse of the probability of selection based on available characteristics in the administrative data used to randomly sample them. We will use factor analysis and cluster analysis using PACT features (e.g., access), comprehensive women’s health care (e.g., gender-specific care availability), and gender-sensitive care delivery (e.g., availability of same-gender providers or gender-aware providers) for WH-PACT achievement variable creation.

All semi-structured interviews will be audio-recorded and transcribed, and qualitative analysis will be conducted using ATLAS.ti version 7. Interview domains are noted in Table 2. Initially, a top-level codebook will be developed for the baseline interviews based on the interview guide. The codebook will then be elaborated based on emergent themes using a constant comparison analytic approach, adjusting as each round of interviews is reviewed. Interviews will be compared within and across facilities and over time. In baseline key stakeholder and teamlet transcripts, we will conduct targeted coding of PACT-related knowledge, attitudes, beliefs, and experiences; expectations of WH-PACT’s effectiveness; and practice contextual factors that influence how care is delivered (and changed) locally. In follow-up key stakeholder transcripts, we will identify factors that facilitated and/or impeded EBQI and WH-PACT achievement and strengths/weaknesses of the WH-PACT model as implemented.

Trial timeframe

The WH-PACT trial is planned from March 2013 through February 2017 (Fig. 1). Phase 1 network-level stakeholder panel meetings occurred as half-day in-person meetings from May to August 2014. Phase 2 local QI launched at all eight EBQI-assigned VAMCs in October 2014. Baseline key stakeholder interviews in all participating VISN and VAMC followed within days of each respective network-level meeting, followed by teamlet interviews at EBQI sites only within one quarter of each meeting. Provider and staff surveys were fielded to PC and WH providers and staff in all participating VAMCs starting September 2014, while baseline patient surveys were launched in January 2015, yielding formative feedback in the following year. EBQI will run for 24 months from launch of local QI teams, followed by 24-month follow-up interview and survey waves.

Trial status

Baseline data collection and EBQI implementation phase.


This trial represents an important opportunity to tailor VA’s PCMH to the needs of WVs while also providing new implementation science insights. This study is one of the first of its kind to test EBQI in the context of the VA WH-PBRN, building on lessons learned from a pilot EBQI study in the founder sites conducted in the first few years of PBRN implementation [35]. The value-added of PBRN use for sustainable practice-based changes in VA settings adds another dimension to this work. We anticipate that we will glean important new knowledge of what is needed to accomplish multilevel stakeholder engagement in evidence-based organizational changes.

The VA has undergone significant changes in the past 30 years since the Government Accounting Office first identified deficits in care for women in 1982 [1]. There is a growing awareness of the vulnerability of women in the military, [36, 37] with substantial MH burdens, [38] and increasing demands in an array of new mandated WH services (such as maternity benefits and newborn care), all occurring in a healthcare system with a characteristically low volume of women. While VA managers have economic incentives to increase market share among women, an underlying tension remains unresolved on how best to achieve the tenets of gender-sensitive comprehensive WH care [39, 40].

We anticipate that study results will demonstrate the value of EBQI approaches to adapting PACT to the needs of special populations, ensuring PACT not only meets the needs of WVs but also serves as an exemplar for other special populations (e.g., recently returned veterans who may need specific post-deployment health care, veterans with HIV who sometimes obtain PC in infectious disease clinics). Given the growing number of WVs relying on VA care, the expansion of services for which WVs are now eligible [41], and the pressing need to reduce persistent gender disparities in VA care, the VA will reap important benefits from determining how to adapt PACT to enhance WVs’ outcomes. And, by systematically engaging frontline leaders, managers, and providers, our approach will substantially increase the relevance and thereby potential sustainability and spread of the resulting WH-PACT model.

In this study, we will develop and test methods for providing the kind of organizational supports that the Institute of Medicine’s Crossing the Quality Chasm referred to as central to transformative change [42]. This study reflects the first time these methods will be deployed to address the needs of WVs, tied to careful, comprehensive measurement and evaluation, which in turn will inform an evidence-based roadmap for policy/practice action and a foundation for developing evidence-based tools to support scale-up and spread to different VA types in different contexts. The strength of our partnerships with WH and primary care leadership will also ensure that we are continually grounded in fiscal, management, and strategic issues of consequence as the study is conducted. Tackling the fundamental tenets of PACT care in gender-sensitive and patient-centered ways will clearly be essential to retaining WVs in care and fulfilling the promise to ensure that WVs get “the best care anywhere” [43].


CCM, Chronic Care Model; CQI, continuous quality improvement; EBQI, evidence-based quality improvement; ICC, intraclass correlation coefficient; LPN, licensed practical nurse; LVN, licensed vocational nurse; MD, medical doctor; MH, mental health; NP, nurse practitioner; PA, physician’s assistant; PACT, Patient Aligned Care Teams; PBRN, practice based research network; PC, primary care; PCMH, patient-centered medical homes; PDSA, Plan, Do, Study, Act; QI, quality improvement; RN, registered nurse; VA, US Department of Veterans Affairs or Veterans Health Administration healthcare system; VAMC, VA medical center; VISN, Veterans Integrated Service Network; WH, women’s health; WHS, Women’s Health Services; WVs, women Veterans


  1. U.S. General Accounting Office. Actions needed to ensure that female veterans have equal access to VA benefits. Washington, DC: GAO Publications; 1982. HRD-82-98.

    Google Scholar 

  2. Veterans Health Administration. Sourcebook: women veterans in the Veterans Health Administration. Volume 1: sociodemographic characteristics and use of VHA care. Washington, DC: Department of Veterans Affairs; 2010.

    Google Scholar 

  3. Yano EM, Rose D, Bean-Mayberry B, Canelo I, Washington DL. Impact of practice structure on the quality of care for women veterans (phase 2) final report. Sepulveda, CA: VA HSR&D Center of Excellence for the Study of Healthcare Provider Behavior; 2010.

    Google Scholar 

  4. Veterans Health Administration. Report to the Appropriations Committee of the U.S. House of Representatives in response to House Appropriations Report No. 110–186, accompanying Public Law 110–161, The Consolidated Appropriations Act. Washington, DC: Department of Veterans Affairs; 2008.

    Google Scholar 

  5. Vogt DS, Stone ER, Salgado DM, King LA, King DW, Savarese VW. Gender awareness among veterans administration health-care workers: existing strengths and areas for improvement. Women’s Health. 2001;34(4):65–83.

    Article  CAS  Google Scholar 

  6. Murdoch M, Bradley A, Mather S, Klein R, Turner C, Yano EM. Women and war: what physicians should know. J Gen Intern Med. 2006;21 Suppl 3:S5–10.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Starfield B, Shi L, Macinko J. Contribution of primary care to health systems and health. Milbank Q. 2005;83:457–502.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Starfield B, Shi L. The medical home, access to care, and insurance: a review of evidence. Pediatrics. 2004;113:1493–8.

    PubMed  Google Scholar 

  9. Klein S. The Veterans Health Administration: implementing patient centered medical homes in the nation’s largest integrated delivery system. Commonw Fund. 2011;1537:16.

    Google Scholar 

  10. Veterans Health Administration. Health care services for women veterans. VHA Handbook 1330.01. Washington, DC: Department of Veterans Affairs; 2010.

    Google Scholar 

  11. Yano EM, Haskell S, Hayes P. Delivery of gender-sensitive comprehensive primary care for women veterans: implications for VA patient aligned care teams. J Gen Intern Med. 2014;29 Suppl 2:S703–7.

    Article  PubMed  Google Scholar 

  12. Veterans Health Administration. Gender differences in performance measures, VHA 2008–2011. Washington DC: Women Veterans Health Strategic Health Care Group, Office of Patient Care Services; 2012.

    Google Scholar 

  13. Wagner EH, Austin BT, Davis C, Hindmarsh M, Schaefer J, Bonomi A. Improving chronic illness care: translating evidence into action. Health Affairs. 2001;20(6):64–78.

    Article  CAS  PubMed  Google Scholar 

  14. Glasgow RE, Orleans CT, Wagner EH. Does the chronic care model serve also as a template for improving prevention? Milbank Q. 2001;79(4):579–612.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Rubenstein LV, Stockdale SE, Sapir N, Altman L, Dresselhaus T, Salem-Schatz S, Vivell S, Ovretveit J, Hamilton AB, Yano EM. A patient-centered primary care practice approach using evidence-based quality improvement: rationale, methods, and early assessment of implementation. J Gen Intern Med. 2014;29 Suppl 2:S589–97.

    Article  PubMed  Google Scholar 

  16. Yano EM. The VA health care delivery system. In: Yih Y, editor. Handbook of healthcare delivery systems. New York: CRC Press Taylor & Francis Group Publishers; 2010. p. 3. 1-3.20.

    Google Scholar 

  17. Frayne SM, Carney DV, Bastian L, Bean-Mayberry B, Sadler AN, Klap R, et al. The VA women’s health practice-based research network: amplifying women veterans’ voices in VA research. J Gen Intern Med. 2013;28 Suppl 2:S504–9.

    Article  PubMed  Google Scholar 

  18. Yano EM. A partnered research initiative to accelerate implementation of comprehensive care for women veterans: the VA women’s health CREATE. Med Care. 2015;4 Suppl 1:S10–14.

    Article  Google Scholar 

  19. Bodenheimer T, Wagner EH, Grumbach K. Improving primary care for patients with chronic illness. JAMA. 2002;288(14):1775–9.

    Article  PubMed  Google Scholar 

  20. Coleman K, Austin BT, Brach C, Wagner EH. Evidence on the chronic care model in the new millennium. Health Aff (Millwood). 2009;28(1):75–85.

    Article  Google Scholar 

  21. Bodenheimer T, Wagner EH, Grumbach K. Improving primary care for patients with chronic illness: the chronic care model, part 2. JAMA. 2002;288(15):1909–14.

    Article  PubMed  Google Scholar 

  22. The Chronic Care Model and Patient Centered Medical Homes. Improving Chronic Illness Care. Accessed 10 May 2016.

  23. Gold PB, Glynn SM, Mueser KT. Challenges to implementing and sustaining comprehensive mental health service programs. Eval Health Prof. 2006;29:195–218.

    Article  PubMed  Google Scholar 

  24. Parker LE, dePillis E, Altschuler A, Rubenstein LV, Meredith LS. Balancing participation and expertise: a comparison of locally and centrally managed health care quality improvement within primary care practices. Qual Health Res. 2007;17:1268–79.

    Article  PubMed  Google Scholar 

  25. Rubenstein LV, Fink A, Yano EM, Simon B, Chernof B, Robbins AS. Increasing the impact of quality improvement on health: an expert panel method for setting institutional priorities. Jt Comm J Qual Improve. 1995;21:420–32.

    Article  CAS  Google Scholar 

  26. Rubenstein LV, Meredith LS, Parker LE, Gordon NP, Hickey SC, Oken C, Lee ML. Impacts of evidence-based quality improvement on depression in primary care: a randomized experiment. J Gen Intern Med. 2006;21(10):1027–35.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Chaney EF, Rubenstein LV, Liu C-F, Yano EM, Bolkan C, Lee ML, et al. Implementing collaborative care for depression treatment in primary care: a cluster randomized evaluation of a quality improvement practice redesign. Implement Sci. 2011;6:121.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Yano EM, Rubenstein LV, Farmer MM, Chernof BA, Mittman BS, Lanto AB, et al. Targeting primary care referrals to smoking cessation clinics does not improve quit rates: implementing evidence-based interventions into practice. Health Serv Res. 2008;43:1637–61.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Brown AH, Cohen AN, Chinman MJ, Kessler C, Young AS. EQUIP: implementing chronic care principles and applying formative evaluation methods to improve schizophrenia. Implement Sci. 2008;3:9.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Rubenstein LV, Chaney EF, Ober S, Felker B, Sherman SE, Lanto A, Vivell S. Using evidence-based quality improvement methods for translating depression collaborative care research into practice. Fam Syst Health. 2011;28(2):91–113.

    Article  Google Scholar 

  31. Rogers EM. Diffusion of innovation. 4th ed. New York: The Free Press; 1995.

    Google Scholar 

  32. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Scholle SH, Weisman CS, Anderson R, Weitz T, Freund KM, Binko J. Women’s satisfaction with primary care: a new measurement effort from the PHS national centers of excellence in women’s health. Women’s Health Issues. 2000;10(1):1–9.

    Article  CAS  PubMed  Google Scholar 

  34. Adams G, Gulliford MC, Ukoumunne OC, Eldridge S, Chinn S, Campbell MJ. Patterns of intra-cluster correlation from primary care research to inform study design and analysis. J Clin Epidemiol. 2004;57(8):785–94.

    Article  PubMed  Google Scholar 

  35. Fox AB, Hamilton AB, Frayne SM, Wiltsey-Stirman S, Bean-Mayberry B, Carney D, DiLeone B, Giersch JM, Goldstein KM, Romodan Y, Sadler A, Wiltsey-Stirman S, Yano EM, Yee E, Vogt D. Effectiveness of an evidence-based quality improvement approach to cultural competence training: The Veterans Affairs’ “Caring for Women Veterans” program. J Contin Educ Health Prof. 2016; [in press].

  36. Suffoletta-Maierle S, Grubaugh AL, Magruder K, Monnier J, Frueh BC. Trauma-related mental health needs and service utilization among female veterans. J Psychiatr Pract. 2003;9(5):367–75.

    Article  PubMed  Google Scholar 

  37. Murdoch M, Polusny MA, Hodges J, O'Brien N. Prevalence of in-service and post-service sexual assault among combat and noncombat veterans applying for VA PTSD disability benefits. Mil Med. 2004;169(5):392–5.

    Article  PubMed  Google Scholar 

  38. Magruder KM, Frueh BC, Knapp RG, Johnson MR, Vaughan 3rd JA, Carson TC, et al. PTSD symptoms, demographic characteristics, and functional status among veterans treated in VA primary care clinics. J Trauma Stress. 2004;17(4):293–301.

    Article  PubMed  Google Scholar 

  39. Weisman CC, Curbow B, Khoury AJ. Women’s health centers and managed care. Women’s Health Issues. 1996;6:255–63.

    Article  CAS  PubMed  Google Scholar 

  40. deKleijn M, Lagro-Janssen TA, Canelo I, Yano EM. Creating a roadmap for delivering gender-sensitive comprehensive care for women Veterans: results of a national expert panel. Med Care. 2015;53(4 Suppl 1):S156–64.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Washington DL, Caffrey C, Goldzweig C, Simon B, Yano EM. Availability of comprehensive women’s health care through Veterans Affairs Medical Centers. Women’s Health Issues. 2003;13:50–4.

    Article  PubMed  Google Scholar 

  42. Institute of Medicine. Crossing the quality chasm. Washington DC: National Academy Press; 2004.

    Google Scholar 

  43. Longman P. The best care anywhere. Washington Monthly. 2005. Accessed 10 May 2016

Download references


We wish to acknowledge the contributions of Martin Lee, PhD, for the study design and power/sample size calculation consultation and for conducting the randomization process; Alicia Bergman, PhD, for the EBQI implementation and intervention site management; Alissa Simon, MA, for the extensive support on IRB applications, management, tracking, and follow-up; Britney Chow, MPH, for the EBQI management and project direction; Anneka Oishi, BA, for the survey measure documentation, data entry/validation, and technical/administrative support for the data collection; and Kristina Oishi, BA, for the overarching project support, scheduling, and team management.


This study was funded by the VA HSR&D Service, Office of Research & Development, Washington DC (Project # CRE12-026), and enabled further by conduct within the Women’s Health Practice-Based Research Network funded by VA HSR&D Service (Project # SDR 10–012). Development and implementation of the EBQI intervention is being supported through annual Memoranda of Understanding with funding from VA Women’s Health Services (WHS) in the Office of Patient Care Services, Veterans Health Administration. Dr. Yano’s effort was covered by a VA HSR&D Senior Research Career Scientist award (RCS #05-195).

Funders from VA HSR&D Service have not had direct roles in this trial. They request regular reporting of progress, notification of accepted manuscripts or scientific abstracts, and other routine grant administration requirements. VA WHS support of EBQI implementation has resulted in similar reporting mechanisms (quarterly to annual summaries of progress and activities). The partnered roles of WHS, other VA leaders, and non-VA experts on the WH CREATE Executive Steering Committee (ESC) have constituted implementation-focused dialog on actions that might be organized or taken in response to early project findings.

Availability of data and materials

Consents associated with primary data collection for patient and clinician/staff surveys used to evaluate this implementation strategy did not include permission to share data in publicly available repositories. Re-identification is a particular concern in the key stakeholder and teamlet interviews because of the specific nature of the roles included in a select number of VA networks and VA medical centers, precluding data sharing outside the VA. De-identified administrative datasets may be eligible for future data sharing once national VA guidance on request and distribution processes are provided (in process).

Final datasets will be maintained locally until enterprise-level resources become available for long-term storage and access. The analytical datasets and statistical code used in subsequent publications will be retained for a minimum of 6 years, in accordance with VA record retention policy. Any future public release data datasets determined to meet VA guidelines will be maintained so that a VA-approved auditor or the PI could conduct or facilitate validation if needed.

Authors’ contributions

EMY (principal investigator) and LVR (co-principal investigator) conceived the study and acquired the funding. EMY wrote the first draft of the manuscript. ABH, EC, LSM, JD, and IC revised and augmented the methodological detail of their respective sections of the study methods. EMY, AH and JD reviewed and refined the draft. All coauthors have been involved in the iterative review, critique, and refinement of the resulting manuscript and approved the final version.

Authors’ information

The work represented by this study protocol benefitted from the prior work evaluating the effectiveness and impact of EBQI strategies in the VISN 22 PACT Demonstration Laboratory (also called the Veterans Assessment and Improvement Laboratory or VAIL), part of a five-Lab consortium funded by the VA Office of Patient Care Services to develop and test PACT innovations in five regions of the US. LVR and EMY were the PI and Co-PI of VAIL, respectively, while AH, LSM, and JD were the integral VAIL investigators and staff. EMY is a health care epidemiologist with nationally recognized implementation evaluation expertise. AH is a medical anthropologist with similarly nationally recognized implementation evaluation expertise with a focus on qualitative inquiry. IC is a public administration trained project director with nearly two decades of experience managing women Veterans’ research. EC is a health care organizational researcher with expertise in studying health care teams. LSM is a social psychologist and leading scholar in studying provider attitudes and behavior, who also led the VAIL provider/staff surveys. JD is a survey specialist, who served as a VAIL survey director and led to the VAIL analyses of national VA patient survey data, and led to the comparable efforts in the current protocol. LVR is an internationally known implementation scientist, with emphasis on primary care-based organizational improvement, and is a general internist who also practices in PACT.

Competing interests

The authors declare that they have no competing interests.

Ethics approval and consent to participate

This study protocol has been approved by the VA Greater Los Angeles Healthcare System IRB-B Subcommittee (IRB Chair, Paul Shekelle, MD, PhD) (Approval Number 2013-040589). Individual study components (e.g., qualitative methods (key stakeholder and teamlet interviews), patient survey, provider/staff survey) were reviewed and approved by the IRB-B as serial project modifications to an umbrella IRB application under the study title. For each study component, we obtained IRB approval for waivers of documentation of consent for all data collections. Consent was provided verbally and recorded in the case of interviews, verbally provided and recorded in the data by an interviewer for the patient surveys, and indicated by answering survey questions in the self-administered web-based provider/staff surveys. For retrospective administrative data pulls and analysis, we obtained a human subjects research exemption.

The VA Central IRB (Washington DC) also determined that the participating VA medical centers (n = 12) were not engaged in research and that the intervention (or implementation strategy, evidence-based quality improvement (EBQI)) was QI. Documentation of IRB approval (and in VA settings, Research & Development Committee approval as well) and Central IRB determination of non-research for the implementation strategy are available on request. Two of the study components involved contractors at local affiliates (teamlet interviews with UCLA Fielding School of Public Health and provider/staff surveys with RAND Health), whose IRBs also reviewed and approved their respective activities in addition to the VA IRB.

No animals are or will be involved in this study protocol. Similarly, the use of animal or human data or tissue is not applicable to this study protocol.

No individual person’s data in any form are or will be included in this or subsequent manuscripts. Only aggregated patient, clinician, and staff survey data will be reported. For key stakeholder and/or teamlet interview data, non-attributed or identifiable quotes may be reported in addition to thematic summaries.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Elizabeth M. Yano.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yano, E.M., Darling, J.E., Hamilton, A.B. et al. Cluster randomized trial of a multilevel evidence-based quality improvement approach to tailoring VA Patient Aligned Care Teams to the needs of women Veterans. Implementation Sci 11, 101 (2015).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: