Designing a valid randomized pragmatic primary care implementation trial: the my own health report (MOHR) project

Overall design

The MOHR study is a paired, cluster (practice level) randomized, non-blinded, practical, implementation study that uses a delayed intervention design with 9 pairs of primary care practices (18 total). The design combines elements of pragmatic trials, implementation science, systems science, and mixed methods approaches, and engages stakeholders as partners throughout the study design, implementation, evaluation and subsequent dissemination. One practice in each pair will be randomized to early implementation, and the second to delayed implementation four months later. Implementation will consist of each practice defining a patient population and workflow and then fielding the systematic collection of the patient reported measures in daily care. The primary outcomes will be implementation measures and costs that are relevant to potential adopting and funding agencies [15–19]. Patient goal-setting and clinical follow-up in the 10 health behavior and psychosocial domains will be compared between early and delayed intervention sites prior to implementation of the MOHR assessment in delayed intervention sites, thus delayed intervention practices effectively serve as control practices. Feasibility will be assessed by monitoring implementation in both early and delayed implementation sites (see Figure 1 for project overview and timeline). The project is funded through a unique partnership of grant supplementation from the National Cancer Institute (NCI), the Agency for Healthcare Research and Quality (AHRQ), and the Office of Behavioral Social Sciences Research (OBSSR).

Research teams and study settings

To carry out this pragmatic trial, partnerships were created between local research teams at eight nationally distributed academic institutions that manage a practice-based research network (PBRN) or participate in the Cancer Prevention and Control Research Network (CPCRN) [20, 21]. Given the network affiliations, each research team has expertise in partnering with their practice and patient stakeholders. Accordingly, research teams are charged with helping their partner practices implement the patient-reported assessment and carry out the MOHR study protocol, while balancing the need for adaptation to ensure implementation is relevant to local culture and practice flow (see ‘Balancing fidelity and adaption,’ below) [22]. To coordinate activities across the project, a MOHR planning committee composed of representatives from all local research teams and practices will have joint bimonthly meetings for planning (pre-implementation) and to share successes and challenges (post-implementation). The organization and coordination of activities is depicted in Figure 2.

Research teams identified one or two pairs of primary care practices within their network that were similar with respect to practice type (e.g., FQHC or PBRN, family practice or internal medicine), practice ownership, geographic region, EHR infrastructure, and patient population served. Practice pairs were purposefully selected to represent the diversity of primary care settings and populations to ensure greater generalizability of results. Practices were randomly allocated to the early or delayed condition by coin toss. To conceal allocation, the researchers randomizing practices (i.e., project coordination members) were blinded to practice name; practices were labeled by number in paired blocks (by local research teams), and identity was revealed to all participants one week later during a joint call.

Work groups

To facilitate rapid, collaborative decision making on important issues relevant to the MOHR study design and implementation, work groups were formed consisting of research team members, clinicians, funding agency representatives, and content experts. Each work group is responsible for effectively solving issues related to their specific MOHR study area and bringing proposals to the larger collaborative for approval. Work groups include: Web Tool and Practice Training Work Group – to design and test an electronic version of the comprehensive behavioral and psychosocial risk assessment, http://www.MyOwnHealthReport.org, for practices to use and to create training sessions to prepare practices to address topics identified by the MOHR assessment; Patient Experience Survey Work Group – to design the Patient Experience Survey (see ‘Outcomes assessment’ , below) for assessing main patient-level outcomes; Clinical Context Work Group – to identify and evaluate how key practice, patient, and environmental contextual issues impact implementation; Cost Collection and Analyses Work Group – to measure the practice costs of implementing the MOHR assessment; and Publication, Presentation, and Relations Work Group – to promote opportunities for participants to present and publish findings, coordinate with relevant stakeholder entities, and create a repository of materials, tools, and disseminated findings.

Intervention

The intervention being studied is the systematic implementation and fielding of the MOHR health behavior and psychosocial assessment, including the delivery of brief customized patient and provider feedback reports. Each local research team worked with their practice pair champions and key stakeholders to determine the most effective workflow for administering the MOHR assessment, including: (a) which patients would be invited to complete the assessment, (b) when the assessment would be completed (e.g., two weeks before a visit, immediately before a visit), (c) where the assessment would be completed (home or practice), (d) whether the assessment would be electronic or paper based, and (e) how clinicians would receive MOHR feedback summaries within the context of office visits. This is consistent with the original NIH consensus panel’s goal of identifying and standardizing patient reported elements to be incorporated into EHRs while minimizing staff burden and enhancing consistency of implementation. As a result, the practice pairs were encouraged to field the electronic version of the MOHR assessment if possible [8].

Because it was not feasible to fully integrate the MOHR assessment into each of the diverse practice’s EHR and patient portal, a stand alone, publically available website (http://www.MyOwnHealthReport.org) was created. The MOHR website is partially integrated with the study practices’ existing EHRs and administers the 17 health behavior and psychosocial screening questions (including basic demographics) as defined by the NIH consensus panel [6, 8]. In response to positive depression, anxiety, alcohol, and drug screening questions, the website seamlessly prompts the patient to complete more in-depth assessments, including the Patient Health Questionnaire (PHQ9) [23], the Generalized Anxiety Disorder (GAD) questionnaire [24, 25], the Alcohol Use Disorders Identification Test (AUDIT-C) [26], and the Drug Abuse Screening Test (DAST-10) [27], respectively. The MOHR website scores the patient’s responses and characterizes them as being of ‘no concern’ , ‘some concern’, or ‘high concern’ based on norms and national guidelines. For measures with ‘some’ or ‘high concern’ , patients are asked to select which topics they are ready to change and/or discuss with their doctor and select the one topic they feel is most important to address [28–31].

After completing this process, patients are taken to a feedback page to review, download and print (see Figure 3). Health domains in which they are doing well are highlighted and reinforced, initial recommendations for changes and improvements are given for domains where improvements are needed, and space is provided to list any questions, decide on and create up to three SMART goals (specific, measurable, achievable, realistic and timely), and outline follow-up steps to help achieve their goals [32, 33]. When patients have finished reviewing their feedback page and leave the website, a clinician summary is also sent to the patient’s primary care team.

Practice implementation preparation

Practices are being prepared to implement the MOHR assessment through a series of calls and meetings with the site champions, an educational webinar with an interactive ‘call in’ and case scenarios, support materials, and on-site consultation for follow-up. The webinars have been designed using the principles of adult learning, including multiple modalities, reinforcement, modeling and observation. The follow-up consultations have been modeled on academic detailing that entails a brief face-to-face intervention with the clinician, repeated at periodic intervals [34–36]. Academic detailing has been found effective in improving other aspects of clinical care such as cancer screening and chronic disease management [37–43].

The webinars present the evidence supporting the MOHR assessment, strategies for successful practice implementation of risk assessments, and approaches to counseling and supporting patients as they address identified problem behaviors and psychosocial issues. The webinars, as academic detailing, distill the evidence into brief, practical messages that are relevant to and actionable for primary care clinicians. The webinar and local research team consultations seek to change staff attitudes and beliefs toward implementing the MOHR assessment though persuasive communications, including case examples, tailored feedback and reinforcement [44]. Feedback and reinforcement will include weekly practice reports on the number of patients completing the MOHR assessment and bimonthly meetings between practice champions and local research teams to share cross project implementation experiences and review the local implementation process.

Data sources

Outcomes will be collected from three sources: practice appointment records, the Patient Experience Survey, and patient responses to the MOHR assessment. Appointment records will be used for calculating Reach (see ‘Primary outcomes’, below). Reach measures are collected when practices enter the intervention phase, months two to six for early intervention practices and months seven to nine for delayed intervention practices. Appointment records are queried by the practices (electronically or manually) for patients who meet inclusion criteria, are documented, de-identified, and transferred to a central data coordination center every two weeks. The Patient Experiences Survey will be used to compare the early versus delayed intervention practices on patient reports of patient-provider discussions, recommendations for referrals, and collaborative goal-setting across the 10 health domains. The patient responses to the MOHR assessment will be used for calculating patient health behavior and psychosocial changes over time. This will be assessed by repeated administration of the MOHR assessment for the cohort of patients who complete the assessment on the website and provide email addresses (see ‘Outcomes assessment’ , below).

The content of the Patient Experience Survey and fielding protocol were developed by the working group, building on earlier pilot phase experiences and practice input, with the intention of balancing scientific rigor and patient needs, practice capabilities, and resource availability across the diverse settings. The survey questions were standardized. The 5As framework (Assess, Advise, Agree, Assist and Arrange) served as a guide for developing questions for each health domain [45]. Questions from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) assessing patient trust in their healthcare team and perceived provider communication style were also included [46]. Practices will mail the Patient Experience Survey to patients that the early and delayed intervention practices defined as eligible to receive the MOHR assessment two weeks after the patient’s office visit. To enhance survey response rates, surveys will be mailed using a modified Dillman technique in the practice’s envelope with a cover letter from the patient’s provider [47, 48]. Study information will be included with the survey, and completion of the survey will be deemed as consent for participating in the survey – our Institutional Review Board approved a waiver of written consent.

Outcomes assessment

The primary outcomes for this study are implementation measures [15]. The measures include the practices’ implementation Reach and Effectiveness [49]. Reach measures include the percent and characteristics of patients that: are defined as eligible to be invited to complete the MOHR assessment; are invited to complete the MOHR assessment; and complete the MOHR assessment. The primary Effectiveness measures are a comparison of early versus delayed intervention practice patient report on whether they: were asked about the health behavior and psychosocial domains; set a goal or action plan with their clinician; and arranged any follow-up contact regarding their plan. As early intervention practices will have implemented the MOHR assessment and delayed intervention practices will not yet have implemented the MOHR assessment, they will respectively serve as intervention and control conditions. The impact on health behaviors and psychosocial problems will be assessed by repeated administration of the MOHR assessment for the subset of patients providing an email address. Patient responses will be compared at baseline and four months after baseline. Multivariate regressions will assess for influences of practice implementation characteristics (e.g., electronic versus paper) and patient demographics on each outcome. The overall CONSORT diagram depicting study participation and data capture are depicted in Figure 4.

Power and sample size analysis

To determine power, we focused on the patient report as to whether any practice personnel worked with her/him to set specific goals for change for the 10 behavior and mental health topics. For the base power analysis, we assumed 9 matched pairs of sites, 300 patients mailed the Patient Experience Survey from each site, and a 50% response rate (2,700 total respondents). Imputation analyses will be conducted to evaluate the impact of missing data, but here we assume the loss of up to 50% of the sample for power calculation purposes. For the first power analysis, we focus on the binary event that at least one goal was established (as opposed to none). Assuming a delayed intervention response rate of 0.3, there is 90% power to detect a 17.5% improvement in the early versus delayed intervention group, and 80% power to detect a 15% improvement.

For the second power analyses, we focus on the continuous outcome of the number of goals established using a hierarchical analysis of covariance. We adjust for any baseline differences in variables related to the outcome as well as clustering effects of patients within clinic. Assuming an intraclass correlation coefficient (ICC) of 0.03, there is 90% power to detect a between conditions effect size of 0.47 with 9 sites per treatment and a 50% patient response rate, and 80% power to detect an effect size of 0.40. Assuming a larger ICC of 0.05, we have 80% power to detect an effect size of 0.51 and 90% power to detect a difference of 0.60, given the assumptions above. All power analyses were performed using the ‘Optimal Design Software’ package (version 3.0).

Contextual assessments

An important aspect of a pragmatic trial is that it is implemented in ‘real-world’ contexts, making it essential that these contexts be described to understand differential outcomes [50]. Accordingly, local research teams will collect data on multiple levels of contextual factors [51–53]. Levels will include national/state/local, community, healthcare system, practice and patient [54]. Contextual factors will include practice culture and staffing, patient panel characteristics, community characteristics and resources, and organizational features [55]. Local research teams will collect relevant data through structured interviews with clinicians and through field observations of clinical, social and organizational processes and interactions as practices field the MOHR assessment at the baseline, middle and end of the intervention to assess changes in context. Quantifiable characteristics will be incorporated from initial practice descriptions made during recruitment. Teams will use a standardized template to systematically elicit and record contextual data.

Cost assessment

The MOHR cost assessment will evaluate both the implementation costs and the replication costs using cost assessment procedures that are minimally intrusive and burdensome [19]. Using standardized a procedure, local research teams will collect each practice’s cost estimate following brief training. This person will use a combination of procedures, including observation of the intervention and brief standardized structured questions of key practice implementation staff. Cost data will be collected at two time points – early in the implementation and once a steady state has been reached. Calculations will be made using established expenditure methods, and replication costs will be categorized based upon different practice and implementation characteristics [56–59]. The cost assessments will be conducted from the practice perspective and will include costs of recruitment, promotion, training, supervision, implementation and follow-up. Data collected during the trial will be used to estimate the cost of a practice replicating the MOHR assessment setting under different conditions and assumptions. Research costs, downstream costs of increased or reduced healthcare utilization, and costs from the patient perspective will not be considered.

Balancing fidelity and adaption

To maintain internal validity, certain components of the intervention were standardized. All participating practices will implement these components, and fidelity to implementation will be assessed. However, to reflect the pragmatic nature of the trial, practices were encouraged to adapt aspects of the intervention to their workflow and setting [22]. This is necessary because the MOHR study purposively includes practices that vary widely with respect to patients (mostly minority versus non-minority), location (rural versus urban), and EHR and patient portal use. Therefore, to ensure successful implementations as well as high assessment and survey response rates, each practice pair considered its patients’ needs. The elements that needed to be standardized were decided by the overall project steering committee and working groups, while decisions about local tailoring were made by the practice pairs with support and direction from the local research team.

Standard study elements include the MOHR health behavior and psychosocial assessment questions, provision of patient and provider feedback summaries based on the MOHR assessment, goal-setting materials and brief training to support clinicians with counseling, and Patient Experience Survey questions. Practice pairs are collaboratively deciding which patients to target for the MOHR assessment, while attempting to include the largest percentage of adult patients that is feasible; timing of the MOHR assessment (one to two weeks prior to a visit at the patient’s home or immediately prior to a visit at the practice); mode of the MOHR assessment (website or paper), delivery of clinician summary (integrated into EHR, faxed, printed); and mode of administering the Patient Experience Survey (postal, web-based, telephone or a combination).

Trial status

The MOHR trial began January 2013 and is expected to complete data collection in November 2013. The trial is proceeding according to the timeline outlined in Figure 1. Currently, the early implementation clinics are in the early stages of fielding the MOHR assessment to their patients. The delayed intervention clinics will begin implementation between July and September 2013.