Table 2 Overview of validation analyses for primary aims

Aim 1

Inter-rater reliability

The consistency of measurement results across different raters given identical conditions

ICC calculated from HLM to determine if respondents have higher agreement within facility and project than between.

Individual-level, baseline ORCA data from partner projects

k = 208

n = 53

Internal consistency reliability

The consistency of items within a given scale, with the same rater

Cronbach's alpha, and item-rest correlation to determine if items within subscales, and subscales within scales, correlate more strongly than between subscales/scales.

Individual-level, baseline ORCA data from partner projects

k = 208

n = 53

Aim 2

Content validity

A check of the instrument's items against the content domain of the construct

Expert panel review of conceptual domains, and Delphi survey on ORCA items assessing (a) degree of match to conceptual domain, and (b) importance for understanding organizational readiness;

Transcripts of expert panel discussion and structured Delphi survey

n = 14 (panel members)

Confirmatory factor analysis to match items to subscales, and subscales to scales.

Individual-level, baseline ORCA data from partner projects

k = 208

n = 53

Aim 3

Predictive validity

Degree to which an instrument predicts a theoretically meaningful outcome.

Multivariate regression in which the ORCA scales serves as IV, and implementation effect size as the DV.

Site-level, baseline ORCA data, and individual-level implementation outcomes

k = 146

n = 30

Concurrent validity

Degree to which an instrument distinguishes groups it should theoretically distinguish (e.g., low false positives and low false negatives).

Multivariate regression in which external facilitation intervention is the IV and the ORCA scales are the DV.

Site-level, follow-up ORCA data, and intervention cohort (external facilitation vs. control site)

k = 122

n = 28

Convergent validity

The degree to which an instrument performs in a similar manner to other instruments that purportedly measure the same construct.

Multivariate regression with ORCA scales as IVs, and JSI items on satisfaction with direct supervision and senior leadership serve as DVs.

Individual-level, baseline ORCA and job satisfaction data

k = 158

n = 33

Discriminant validity

Degree to which an instrument performs in a different manner to other instruments that measure different constructs.

Multivariate regression with ORCA scales as IVs, and overall JSI and satisfaction with pay as DVs.

Individual-level, baseline ORCA and job satisfaction data

k = 158

n = 33