- Study protocol
- Open Access
Study protocol: transforming outcomes for patients through medical home evaluation and redesign: a cluster randomized controlled trial to test high value elements for patient-centered medical homes versus quality improvement
© Dorr et al.; licensee BioMed Central. 2015
- Received: 9 December 2014
- Accepted: 5 January 2015
- Published: 22 January 2015
Health care in the United States is in the midst of a near perfect storm: strong cost pressures, dramatic redesign efforts like patient-centered medical homes and accountable care organizations, and a broad series of payment and eligibility reforms. To date, alternative models of care intended to reduce costs and improve outcomes have shown mixed effects in the U.S., in part due to the difficulty of performing rigorous evaluation studies that control for the broader transformation while avoiding other biases, such as organizational or clinic effect on individual patient outcomes. Our objective is to test whether clinics assigned to achieve high value elements (HVEs) of practice redesign are more likely than controls to achieve improvements in patient health and satisfaction with care and reduction in costs.
To prepare, we interview stakeholders, align with health reform, and propose a pilot. Participants are primary care clinics engaged in reform. Study protocol requires that both arms receive monthly practice facilitation, IT-based milestone reporting, and small financial incentives based on self-determined quality improvement (QI) goals; intervention receives additional prompting to choose HVEs. Design is a cluster randomized controlled trial over 1 year with pre- and post-washout periods. Outcomes are unplanned utilization and costs, patient experience of care, quality, and team performance. Analysis is a multivariate difference-in-difference with adjustments for patient risk, intraclinic correlation, and other confounders.
The TOPMED study is a cluster randomized controlled trial focused on learning how primary care practices can transform within health reform guidelines to achieve outcomes related to the Triple Aim.
ClinicalTrials.gov registration: NCT02106221.
- Primary Care Clinic
- Health Reform
- Intracluster Correlation
- Accountable Care Organization
- Practice Facilitation
United States health care costs per capita are more than 50% larger than other developed countries, yet our health outcomes lag in many respects . To combat this disparity, the U.S. is engaged in a number of health reform efforts. For instance, billions are being invested in accountable care organizations (ACOs)—intended to virtually integrate health systems, align incentives, and optimize care for populations—and patient-centered medical homes (PCMHs)—intended to change the incentives and impact of primary care settings to better engage patients, improve quality, and improve care coordination . However, the evidence about how to reform care in a complex and fragmented system is unclear [3,4] and the outcomes of previous efforts are mixed [5–8]. Although trial design has grown increasingly sophisticated over the last 20 years, designing and analyzing trials in the midst of large health reform efforts is still challenging. Analyses have shown that intervention definitions, contemporaneous trends, sample size estimates, and unit of analysis all confuse results . In this paper, we describe the protocol of a complex cluster randomized controlled trial named Transforming Outcomes for Patients through Medical Home Evaluation and reDesign or TOPMED.
TOPMED is intended to test whether primary care clinics in Oregon engaged in both PCMH- and ACO-like redesign efforts can improve outcomes. In Oregon, coordinated care organizations are ACO-like models that bundle care for Medicaid patients, creating or adapting regional entities whose goals are to reduce cost of care while meeting benchmarks in quality and patient experience of care. Their primary approach involves care coordination, increased population management systems with stratification, and combining previously disparate systems like mental health, housing, and health systems in novel ways . Similarly, for PCMH, Oregon chose a state recognition process (called the patient-centered primary care home) and payment reform through efforts like the Comprehensive Primary Care (CPC) initiative, funded through the Centers for Medicare and Medicaid Services (CMS), and primary care clinics are receiving a combination of proactive care management payments and shared savings to increase population management, care coordination, quality improvement, and engaging patients and families.
Previous evidence of the effectiveness of these models is mixed. Studies at Group Health, Geisinger, and Intermountain Healthcare demonstrated success [5,6,10], but several others, including the national demonstration project (NDP), a cluster randomized trial of 36 primary care clinics, improved quality metrics but failed to affect the key outcomes of cost, utilization, and patient experience . Since the definitions of PCMHs are complex, we and others have identified key concepts—what we call ‘high-value elements’—that may help lead to success. In the Care Management Plus (CMP) trial at Intermountain, which studied the use of care managers embedded in seven primary care clinics compared to a set of matched control clinics, success factors included an integrated care manager, an IT-based population management system, training in complex illness care, motivational interviewing, self-management and goal setting, and how to address issues for older adults and patients identified at higher risk. In all, 1,440 patients with complex illness needs were referred during the 2-year study period. For similar patients with multiple chronic illnesses, hospitalizations were reduced by 24%–40%, patient satisfaction was higher, and patient health status was improved for those referred to care management compared to controls. Similarly, Brown et al. identified seven characteristics that defined the effective Medicare chronic care demonstration sites: 1) care coordinators with face-to-face contact with patients; 2) strong relationships with physicians; 3) patient education; 4) managing care transitions; 5) acting as a communications hub; 6) medication management; and 7) addressing psychosocial issues .
A number of these concepts—care coordination, care management, population management, use of IT, focusing on complex patients—are used in PCMH and ACO-like models, but most of the elements do not directly relate to known drivers of cost reduction . Thus, in pursuing health reform, clinics have many different paths to choose to achieve transformation and potentially impact these outcomes. Assistance in transformation such as learning collaboratives and practice facilitation may help but does not appear to be sufficient to consistently achieve desired outcomes . We wanted to understand if we could use practice facilitation coupled with IT and incentives to drive clinics towards more effective components. However, trial design in health reform is notoriously complex, with previous reviews demonstrating multiple biases and confounders that threaten study validity and success . In addition, pragmatic trial design requires a protocol that can work in the real world; that is, where clinical teams need flexibility, to maintain autonomy and decision-making as much as possible, and are not burdened with additional study requests.
In our trial design process for TOPMED, we attempted to understand how one could overcome the weaknesses and mixed outcomes of previous trials. Building upon recent efforts, TOPMED integrates lessons from model implementation and trial design into health reform efforts while revising study design to account for biases. Thus, we create a trial aligned with these efforts to understand if a more targeted approach to primary care transformation is more effective than the general approach. We use a cluster randomized control trial design because our intervention involves quality improvement efforts at the organizational level of practice sites.
The objective of this study is to understand whether focused versus general quality improvement efforts in a pragmatic cluster randomized trial can facilitate improvement in patients’ health and satisfaction with care while reducing costs. The trial outcomes and hypotheses are both at the cluster and patient level.
Would a study aligned with major reform efforts with focused practice facilitation, targeted incentives, and IT-based reporting enhance quality improvement capability and likely achievement of selected health reform goals within the clinics?
Would a focus on a set of predetermined ‘high-value elements’ from the many choices in health reform can improve outcomes more than general quality improvement?
The proposed trial design is a cluster randomized controlled trial with a high-value element (intervention) and a general quality improvement arm (control), where randomization is at the clinic level and matching is used to minimize baseline clinic differences between arms.
We took several complementary approaches to explore different trial design options in the complex health reform environment for the TOPMED trial. First, we matched the trial design to the PRECIS pragmatic trial framework and reviewed trial literature. Second, we aligned trial implementation and components with major health reform elements. Third, a series of stakeholder interviews and focus groups were held to understand the values, understandings, perceptions, and intentions in regard to the health reform efforts and the potential for the trial design to be nested within these efforts. Finally, a pilot was completed to test various elements prior to the study start.
Preparation steps for trial design
PRECIS (pragmatic trial) review
Initial high-value elements were too focused and prescriptive
Broadened to allow flexibility and health professional judgment
Alignment with health reform via multistakeholder panel (N = 12)
Incentives too complex; lack of alignment with other health reform efforts; team and clinic culture will drive results
Generated alignment document; revised intervention metrics and approach to better align with initiatives; extensive team and clinic assessment
Chronic illness patient focus group (N = 9)
Access to the clinic and the specific services need to be improved; information gaps about what is newly available (like care coordination) was common
Look carefully for patient experience measures that examine access and information gathering; expect different experiences from patients than reported by clinics; improve education
Insurer focus group (N = 22)
Each insurer had their own initiative or had a separate take on the current initiative set
Aligned principles were agreed to by payers; encourage payer alignment
Pilot clinic interviews (N = 6)
Expectations of the clinic too vague and time-consuming
Generated a memorandum of understanding that clinics revised and implemented
A 6-month pilot at a single practice site was performed to test the various interventions and refine efforts. At the pilot site, preparation and protocol elements were tested in a series of six monthly practice facilitation visits. At each visit, an aspect of the trial design was tested for feasibility, clarity, and usability. The pilot team was interviewed for the results, and results demonstrated that the expectations were too vague and time-consuming. A memorandum of agreement was drawn up with specific expectations for both the research team and the clinic team. These included a role-based specification by clinic of time expected, use of data, and communication required to participate, as well as clear roles for researchers. The protocol visits by practice facilitators were honed to focus on key drivers of quality improvement and transformation: leadership, team-based approach, QI infrastructure, setting new improvement targets, and follow-up on previous improvement targets. Reports were assessed for comprehension and revised, and use of the IT system for study participation was reduced, allowing clinics to use their own protocols and tools when they matched the function of the IT system.
Participants are eight primary care clinics (clusters) that are deeply engaged in health reform, are willing to use a population management registry tool, are willing to receive practice facilitation and improve their performance, and use an electronic health record (EHR) system at baseline. A goal was set to recruit a diverse set of participant clinics by ownership, location (rural versus urban), and size (from 3 to 20 providers). Patients, providers, and staff of these clinics are consented separately when their specific opinions or viewpoints are elicited (via surveys or interviews), with no penalty for non-participation. Data is collected at the clinics during facilitation sessions, from the EHR through automated extraction and from patients, providers, and staff via paper and online surveys.
Intervention ‘high-value’ elements
Specific example metric*
Alignment with reform
Identification of at-risk populations and care management
Care plan utilization
Tier 3—PCMH provides care plans to >50% of high-risk patients.
Advance directive utilization
Tier 3—The PCMH offers advance directives to at least 50% of patients over 65.
Based on need
Care management outreach
Tier 2—PCMH’s care coordination outreach reaches 50% of high-risk patients.
Patient engagement and proactive goal setting
Education and self-management resources
Tier 2—More than 10% of all unique patients are provided patient-specific education resources and self-management services.
Tier 3—PCMH sends appropriate reminders to at least 20% of all eligible patients.
Integrated information and procedures across settings
Clinical information exchange
Tier 1—PCMH exchanges structured clinical information and tracks critical elements (e.g., hospitalizations).
Utilization monitoring and follow-up
Tier 3—PCMH follows up on patient hospitalizations and ED visits 70% of the time (when they have the information).
Population management tools
Performance data utilization
Tier 1—The PCMH uses performance data to identify opportunities for improvement and acts to improve clinical quality, efficiency, and patient experience.
Receive and respond to electronic requests
Tier 3—The PCMH provides a response to online or electronic queries within two business days.
Tier 1—PCMH offers access to in-person care at least 12 h weekly outside traditional business hours.
Tracking 3rd next available appointments
Trial implementation and monitoring
Clinics are eligible for the same amount of incentives. Practice facilitation is offered monthly to both intervention and control since recent summary literature suggests that practice facilitation both makes it possible to transform and directly affects outcomes of trials. At each visit, clinics are free to select from a wide variety of choices based on their current needs; while the control arm has 35, the intervention arm has 12 options. Previous goals are assessed by the practice facilitator and new ones set. The IT system is available at all times, with functions of risk stratification (built in scores), care management tracking, and self-assessment of all PCMH elements available for clinical team review. The incentive reports are built into the IT system, with the intervention and control receiving a general score based on setting and meeting QI goals, and intervention receiving a multiplier based on the number of high-value elements met. The multiplier, whose maximum was 2, set 30% of the potential incentive aside for HVE achievement. Once the clinics meet a HVE threshold of 2/3 of the elements achieved, they earn up to a 10% bonus for achieving additional elements. Additional file 1 shows the proposed IT reports for intervention and control.
The process to generate the scores was tested during the pilot study. In general, effort at QI is rewarded on a 0%–100% scale; factors included in the score generation are organization around QI, leadership involvement, team function, and setting and follow-up on goals. From the practice facilitator’s notes each month, a team of three investigators review each element and decide on a total score. This score is fed back at each visit, with a summary report of overall performance versus other clinics each quarter. The pilot clinic demonstrated a strong response to the reward of effort and the competition engendered by a ‘grading’ approach; they did require significant explanation of the scoring rubric and requested the opportunity to offer feedback to get a score adjusted.
Outcomes and measures
Primary, secondary, and monitoring outcomes
Alignment with health reform
Primary outcome goals
Reduce utilization by 10%
Total cost of care
All Payer All Claims data source; patient level
Strong; primary goal
Hospitalization rate per patient per month
ED visit rate per patient per month
Improve experience of care by 10%
CG-CAHPS: 4/11 composite scores
Survey to 1,600 random stratified patients
Secondary outcome goals
Improve quality of care by 10%
Composite of nine composite measures
EHR-based electronic clinical quality measures; clinic level
Strong: same measures used
Improve teamness, collaboration, and patient-centered care
Composites of various tools
CSQ, TDM, CPAT, and novel collaboration tool; clinic level
Clusters engage in quality improvement
Monthly ‘score’ based on QI progress and goal setting
Novel practice facilitation-based measure; clinic level
HVE passed monthly
Monthly count of HVEs passed
HVEs derived from PCMH; clinic level
Other secondary outcomes include measurements at the cluster and patient level based on stakeholder suggestion. For clinics, culture, microsystem team performance, coordination and collaboration, and adaptive reserve were felt by stakeholders and seen in the literature to be crucial to measure before and after the trial; thus, we will use the clinician staff questionnaire (CSQ), the Team Development Measure (TDM), the Collaborative Practice Assessment Tool (CPAT), and a novel collaboration tool for external collaboration between clinical teams. Qualitative interviews with patients and with clinics will be performed to understand the impact of study more broadly. Monitoring goals were discussed above but required monthly assessments of clinic engagement with QI goal setting and follow-up as well as the high-value elements will provide protocol monitoring metrics. Deviations from protocol will be recorded and used to calculate protocol adherence.
The sample size was based on eight hypothetical clinics with an average of 10,000 patients each. The intracluster correlation was estimated at .02, and the necessary sample size to detect a change of 10% in cost and utilization over the year was <6,000 patients with average costs of $730 per month (average Medicare spend for our region) and average hospitalization rates of 14% at a power of > .80. Besides the monitoring metrics, above, no interim analyses or stopping guidelines were planned due to the low risk of the study.
For randomization, with a small number of clusters (8), the study requires significant work to assure baseline similarities, where possible, and analytic techniques to adjust for site-specific differences. To that end, we used a sequence allocation that was computer-generated and used matching by key characteristics to place the clustered clinics into match groups. Match criteria included organization type and location; total patient population, and the number of patients that were high risk according to a standardized risk scoring system based on comorbidities that predict hospitalization. Sequence generation was done by a random number seeded by time; matched pairs of clinics are randomly allocated by this seed to either intervention or control. For the patient experience of care surveys, patients are randomly chosen by a computer program, with oversampling of ‘high risk’ at 50% of the overall sample. The principal investigator (PI) wrote the randomization algorithm, the matching criteria, and had multiple team members approve them as fair, unbiased, and without error prior to implementation. Once the program was run, it was sent directly to the other team members without intervention by the PI.
Blinding and consent
For blinding, incentives vary by arm and participants are shown their own incentives. To partially blind them, the terms ‘intervention’ and ‘control’ are not used, and all reports are only shown to individual clinics without noting differences between arms. Similarly, blinding for the study team (to avoid experimenter’s bias) is not possible, since the reports are seen across clinics by the team. To reduce bias, the practice facilitator is required to log all interactions and have these reviewed by a third party for time equality (same resource amounts provided to each), for minimal discussion of differences between clinics and arms, and for similar internal milestone achievement by each clinic. Blinding for the analyst is planned, with intervention and control arms labeled only as A and B, and clinic and patient identifiers are masked. Consent was sought from clinics, clinicians participating, and patients receiving the CG-CAHPS survey; other data collection is completed, done under a waiver of consent.
Statistical methods and analysis
Descriptive analysis will compare baseline characteristics of clinics and individuals across outcomes and primary confounders (demographics, chronic illnesses, previous medical care, previous adherence to quality measures). A difference in difference calculation will compare changes in rates of outcomes for intervention versus control. Primary analysis will be a multi-level, two-stage analysis to account first for patient level and clinic level differences and then changes over time. Modeling will be done using generalized estimating equations; these can be adjusted for the level and the outcome variable specifics. The risk of selection bias by cluster is high, so to account for this, we pre-assess the population by risk of hospitalization using a standardized risk score prior to trial start. This group becomes the analytic cohort, and the size of each high-risk group is used to stratify clusters. We will generate the intracluster correlation, account for it by adjusting multivariate analysis, and perform propensity matching within the analytic cohort to assure similarity of baseline cohorts at the clinics. Overfitting will be tested using Copa’s test and model fit using Hosmer-Lemeshow, receiver operating curves, and Akaike information criterion for logistic-based models and Box-Cox test, GLM family test, Link and RESET test in addition to R2, and mean-squared error for cost-based models.
Ethical approval and trial status
Ethical approval was granted by the Oregon Health and Science University (OHSU) Institutional Review Board. At the time of manuscript submission, the trial is ongoing. While primary data collection has concluded, we are currently collecting data from CG-CAHPS surveys and have not yet begun analysis on primary outcomes.
The TOPMED trial is a cluster randomized controlled trial to test whether focused quality improvement would be superior to general quality improvement in achieving cost and experience of care outcomes, especially for high-risk patients, across eight clinics. The trial has a pragmatic design, focusing on providing incentives, IT support, and practice facilitation that are intended to work in the real world of health reform and still allow choice, autonomy, and flexibility for the clinical teams.
Several trial design aspects proved challenging and have limitations. Sample size for number of clusters is low, while the patient-to-cluster ratio is high. A report by Mathematica on patient-centered medical home studies completed a quantitative analysis that concluded 200 clinics (cluster level) with 10 patients each would have significantly more power than 10 clinics with 200 patients each . However, stakeholders reflected to our study team that the ability to understand what is happening in each cluster can be extraordinarily complex—a clinic is an organization made up of many different microsystems, changing processes, and subpopulations of patients who may self-select a particular microsystem based on their needs; failing to measure these microsystems leads to errors. In addition, significant resources are required to try to alter processes within a single clinic, making studies with large number of clusters expensive. Finally, measurement of processes, structure, and other aspects of performance (e.g., culture) at the cluster level (in addition to the patient level) is also resource consuming and yet may be the single most important aspect of cluster RCTs—understanding processes and culture seems to dictate success in many implementation science reviews. Blinding for a trial design like this is not wholly possible. However, partial blinding was used and independent review of facilitator performance and scoring is intended to reduce this bias. For analysis, multiple publications have discussed the frequent need for post hoc adjustments in analyses, including intracluster correlations, two-stage analysis, instrumental variables, and propensity scoring as ways to adjust for baseline differences in the individual patients. Although we use two-stage analysis, differences in clusters or panels may still persist and affect outcomes.
The Transforming Outcomes for Patients through Medical home Evaluation and reDesign (TOPMED) trial was designed as a pragmatic cluster randomized controlled trial with community and stakeholder input. Final trial design is a two-arm trial randomized at the clinic level with matching to improve balance. Intervention and control arms both receive equal incentives, IT support and integration, and practice facilitation; Intervention is focused on a set of literature- and stakeholder-identified high-value elements, while control will be encouraged in general quality improvement activities. Outcomes will be the triple aim of cost, patient experience of care, and population health.
accountable care organizations
Clinician and Group—Consumer Assessment of Healthcare Providers and Systems
Care Management Plus
Centers for Medicare and Medicaid Services
Collaborative Practice Assessment Tool
Comprehensive Primary Care initiative
clinician staff questionnaire
electronic health record
health information technology
Integrated Care Coordination Information System
national demonstration project
patient-centered medical homes
pragmatic-explanatory continuum indicator summary
randomized controlled trial
Team Development Measure
Transforming Outcomes for Patients through Medical home Evaluation & reDesign
This study was funded by the Gordon and Betty Moore Foundation.
- McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348:2635–45.View ArticlePubMedGoogle Scholar
- Devore S, Champion RW. Driving population health through accountable care organizations. Health Aff (Millwood). 2011;30:41–50.View ArticleGoogle Scholar
- Brown RS, Peikes D, Peterson G, Schore J, Razafindrakoto CM. Six features of Medicare coordinated care demonstration programs that cut hospital admissions of high-risk patients. Health Aff (Millwood). 2012;31:1156–66.View ArticleGoogle Scholar
- Peikes D, Dale S, Lundquist E, Genevro J, Meyers D. Building the evidence base for the medical home: what sample and sample size do studies need? White Paper (Prepared by Mathematica Policy Research under Contract No. HHSA290200900019I TO2). AHRQ Publication No. 11-0100-EF. Rockville, MD: Agency for healthcare Research and Quality; 2011.Google Scholar
- Tufano JT, Ralston JD, Tarczy-Hornoch P, Reid RJ. Participatory (re)design of a sociotechnical healthcare delivery system: the group health patient-centered medical home. Stud Health Technol Inform. 2010;157:59–65.PubMedGoogle Scholar
- Kremitske DL, West Jr DJ. Patient-focused primary care: a model. Hosp Top. 1997;75:22–8.View ArticlePubMedGoogle Scholar
- Nutting PA, Miller WL, Crabtree BF, Jaen CR, Stewart EE, Stange KC. Initial lessons from the first national demonstration project on practice transformation to a patient-centered medical home. Ann Fam Med. 2009;7:254–60.View ArticlePubMedPubMed CentralGoogle Scholar
- Dorr DA, Wilcox AB, Brunker CP, Burdon RE, Donnelly SM. The effect of technology-supported, multidisease care management on the mortality and hospitalization of seniors. J Am Geriatr Soc. 2008;56:2195–202.View ArticlePubMedGoogle Scholar
- Robeznieks A. Off-trail in Oregon. Armed with CMS grant, Ore. lawmakers aim to cut Medicaid costs with coordinated ACO. Mod Healthc. 2012;42:6–7. 1.Google Scholar
- Dorr DA, Wilcox A, Burns L, Brunker CP, Narus SP, Clayton PD. Implementing a multidisease chronic care model in primary care using people and technology. Dis Manag. 2006;9:1–15.View ArticlePubMedGoogle Scholar
- Patient-Centered Medical Home [http://www.ncqa.org/Programs/Recognition/Practices/PatientCenteredMedicalHomePCMH.aspx]
- Grumbach K. The patient-centered medical home is not a pill: implications for evaluating primary care reforms. JAMA Intern Med. 2013;173:1913–4.View ArticlePubMedGoogle Scholar
- Peikes D, Chen A, Schore J, Brown R. Effects of care coordination on hospitalization, quality of care, and health care expenditures among Medicare beneficiaries: 15 randomized trials. JAMA. 2009;301:603–18.View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.