Skip to main content

Table 2 Key partner-based evaluation protocol design decisions

From: Partnering with health system operations leadership to develop a controlled implementation trial

Design element Operational considerations Researcher considerations
Sites and population
 The BHIP operational initiative has already begun. Need for results to inform continuing process.
Can capitalize on momentum of the system to engage and motivate sites, promulgate best-practice models.
Helps to sell the project to facilities.
Increases likelihood of incorporation into sustainable practice.
The stepped-wedge design can assess secular trends.
 Identifying the population of facilities to target Slower-to-adopt facilities are of concern. Working with this population avoids ceiling effects (high performers) and insufficient commitment to change (laggards).
 Site recruiting via operational structures Hierarchical communications and reporting structure enhance facility identification and endorsement of program. Provides access beyond “usual suspect” volunteer facilities and “friends of friends” facilities to enhance external validity.
Intervention and design
 Need for all participating sites to receive implementation support Harder to justify the project on policy level if not all sites receive support.
Can be a site recruiting tool.
Stepped wedge can accommodate this, though analysis is more complicated than traditional parallel-groups design
Additionally, stepped wedge can enhance formative evaluation and evaluate secular trends.
 Balance in randomization Experience-based expertise contributes identifying characteristics relevant to success. Sophisticated statistical expertise provides site alancing techniques.
 Control condition Sites seek as much active support as possible, as soon as possible. Researchers develop a credible contrast condition by which to evaluate the impact of the implementation strategy.
 Length of implementation support Experience-based expertise suggests one year of support needed. Pilot data agree, but the need for timely data provision requires steps in wedge of 4 rather than 12 months.
 Need to work with existing VAMC staff without external research-funded support besides external facilitators Resource limitations preclude deploying additional clinical or administrative staff (limitation of both OMHO and QUERI funding). Makes sustainability more likely.
Provides distinct scientific contribution enhancing effectiveness data beyond that from more traditional CCM clinical trials to date.
 Delineating the interface between quality improvement program evaluation and research The BHIP initiative is nationwide in scope and facility participation is not optional.
However, a facility’s participation in this implementation project is optional.
Medical center participation in the project is the decision of the medical center director and mental health leadership, not individual provider.
However, providers can choose not to participate in qualitative interviews.
Patients can choose not to participate in health status and perception of care assessments.
 Use of videoconference and telephone as main modalities for external facilitation Budget (OMHO or QUERI) will not support frequent site visits by external facilitators. Provides greater likelihood of spread of intervention strategy if successful.
Outcome assessment and analysis
 Identification of outcome domains and appropriate instruments Program fidelity measures must be streamlined and targeted, and wherever possible benchmarked against national data. Patient-level measures must be psychometrically valid and feasible in a heterogeneous patient population.
 Both quality and health status impacts are important Operational priority issues are (a) whether CCM can be implemented into BHIP teams and (b) whether CCM-guided BHIP teams have impact on the target population. Hybrid type III designs can accommodate implementation outcomes and health status outcomes.
 Data must both be scientifically valid and reported in a time frame useful to operational partners. Three-year outcomes can help plan strategy for next initiatives, but are too late to make tactical improvements to this phase of BHIP roll-out. Design and analysis accommodate “early looks” at the data on semi-annual basis, using adjustment of significance testing parameters.
 Ethical and regulatory issues A non-voluntary national initiative receives expert support from researchers in order to optimize their roll-out based on valid empirical data. Researchers gather a broader range of data to answer relevant research questions from voluntary subjects.
  1. Conceptual organization adapted from Bauer et al. [69]