Skip to main content

Table 2 Overview of validation analyses for primary aims

From: Predicting implementation from organizational readiness for change: a study protocol

Type of validation Definition Analysis Data Source Observations
Aim 1     
Inter-rater reliability The consistency of measurement results across different raters given identical conditions ICC calculated from HLM to determine if respondents have higher agreement within facility and project than between. Individual-level, baseline ORCA data from partner projects k = 208
n = 53
Internal consistency reliability The consistency of items within a given scale, with the same rater Cronbach's alpha, and item-rest correlation to determine if items within subscales, and subscales within scales, correlate more strongly than between subscales/scales. Individual-level, baseline ORCA data from partner projects k = 208
n = 53
Aim 2     
Content validity A check of the instrument's items against the content domain of the construct Expert panel review of conceptual domains, and Delphi survey on ORCA items assessing (a) degree of match to conceptual domain, and (b) importance for understanding organizational readiness; Transcripts of expert panel discussion and structured Delphi survey n = 14 (panel members)
   Confirmatory factor analysis to match items to subscales, and subscales to scales. Individual-level, baseline ORCA data from partner projects k = 208
n = 53
Aim 3     
Predictive validity Degree to which an instrument predicts a theoretically meaningful outcome. Multivariate regression in which the ORCA scales serves as IV, and implementation effect size as the DV. Site-level, baseline ORCA data, and individual-level implementation outcomes k = 146
n = 30
Concurrent validity Degree to which an instrument distinguishes groups it should theoretically distinguish (e.g., low false positives and low false negatives). Multivariate regression in which external facilitation intervention is the IV and the ORCA scales are the DV. Site-level, follow-up ORCA data, and intervention cohort (external facilitation vs. control site) k = 122
n = 28
Convergent validity The degree to which an instrument performs in a similar manner to other instruments that purportedly measure the same construct. Multivariate regression with ORCA scales as IVs, and JSI items on satisfaction with direct supervision and senior leadership serve as DVs. Individual-level, baseline ORCA and job satisfaction data k = 158
n = 33
Discriminant validity Degree to which an instrument performs in a different manner to other instruments that measure different constructs. Multivariate regression with ORCA scales as IVs, and overall JSI and satisfaction with pay as DVs. Individual-level, baseline ORCA and job satisfaction data k = 158
n = 33
  1. IV = independent variable, DV = dependent variable, ORCA = Organizational Readiness for change Assessment, JSI = job satisfaction index, HLM = Hierarchical Linear Modeling, k = number of individual respondents, n = number of sites.