Table 2 Key partner-based evaluation protocol design decisions

 The BHIP operational initiative has already begun.

Need for results to inform continuing process.

Can capitalize on momentum of the system to engage and motivate sites, promulgate best-practice models.

Helps to sell the project to facilities.

Increases likelihood of incorporation into sustainable practice.

The stepped-wedge design can assess secular trends.

 Identifying the population of facilities to target

Slower-to-adopt facilities are of concern.

Working with this population avoids ceiling effects (high performers) and insufficient commitment to change (laggards).

 Site recruiting via operational structures

Hierarchical communications and reporting structure enhance facility identification and endorsement of program.

Provides access beyond “usual suspect” volunteer facilities and “friends of friends” facilities to enhance external validity.

 Need for all participating sites to receive implementation support

Harder to justify the project on policy level if not all sites receive support.

Can be a site recruiting tool.

Stepped wedge can accommodate this, though analysis is more complicated than traditional parallel-groups design

Additionally, stepped wedge can enhance formative evaluation and evaluate secular trends.

 Balance in randomization

Experience-based expertise contributes identifying characteristics relevant to success.

Sophisticated statistical expertise provides site alancing techniques.

 Control condition

Sites seek as much active support as possible, as soon as possible.

Researchers develop a credible contrast condition by which to evaluate the impact of the implementation strategy.

 Length of implementation support

Experience-based expertise suggests one year of support needed.

Pilot data agree, but the need for timely data provision requires steps in wedge of 4 rather than 12 months.

 Need to work with existing VAMC staff without external research-funded support besides external facilitators

Resource limitations preclude deploying additional clinical or administrative staff (limitation of both OMHO and QUERI funding).

Makes sustainability more likely.

Provides distinct scientific contribution enhancing effectiveness data beyond that from more traditional CCM clinical trials to date.

 Delineating the interface between quality improvement program evaluation and research

The BHIP initiative is nationwide in scope and facility participation is not optional.

However, a facility’s participation in this implementation project is optional.

Medical center participation in the project is the decision of the medical center director and mental health leadership, not individual provider.

However, providers can choose not to participate in qualitative interviews.

Patients can choose not to participate in health status and perception of care assessments.

 Use of videoconference and telephone as main modalities for external facilitation

Budget (OMHO or QUERI) will not support frequent site visits by external facilitators.

Provides greater likelihood of spread of intervention strategy if successful.

 Identification of outcome domains and appropriate instruments

Program fidelity measures must be streamlined and targeted, and wherever possible benchmarked against national data.

Patient-level measures must be psychometrically valid and feasible in a heterogeneous patient population.

 Both quality and health status impacts are important

Operational priority issues are (a) whether CCM can be implemented into BHIP teams and (b) whether CCM-guided BHIP teams have impact on the target population.

Hybrid type III designs can accommodate implementation outcomes and health status outcomes.

 Data must both be scientifically valid and reported in a time frame useful to operational partners.

Three-year outcomes can help plan strategy for next initiatives, but are too late to make tactical improvements to this phase of BHIP roll-out.

Design and analysis accommodate “early looks” at the data on semi-annual basis, using adjustment of significance testing parameters.

 Ethical and regulatory issues

A non-voluntary national initiative receives expert support from researchers in order to optimize their roll-out based on valid empirical data.

Researchers gather a broader range of data to answer relevant research questions from voluntary subjects.