In this article, we provide psychometric evidence in support of a new, brief, theory-based measure of organizational readiness for change, which we call Organizational Readiness for Implementing Change (ORIC). Content adequacy assessment indicated that the items that we developed to measure change commitment and change efficacy reflected the theoretical content of these two facets of organizational readiness and distinguished these two facets from hypothesized determinants of readiness. Exploratory and confirmatory factor analysis in the lab and field study revealed two correlated factors, as expected, with good model fit and high item loadings. Reliability analysis in the lab and field study showed high inter-item consistency for the resulting individual-level scales for change commitment and change efficacy. Inter-rater reliability and inter-rater agreement statistics supported the aggregation of individual level readiness perceptions to the organizational level of analysis.
As expected, the lab study provided stronger evidence than the field study for the reliability and validity of organization-level means as representations of organization-level measures of readiness. In the lab study, we manipulated and standardized the information that study participants received about the organizational readiness of the hospitals depicted in the vignettes to implement meaningful use of electronic health records. In the field study, we made no effort to present study participants with consistent information about their organization’s readiness to use mobile phone technology to monitor and evaluate international health and development programs. Likewise, we made no effort to select international non-governmental organizations that might be expected a priori to vary widely in organization readiness. To our knowledge, organizational leaders made no effort to shape organizational members’ perceptions of readiness. Even under these conditions, organizational members exhibited ‘strong agreement’ in their perceptions of organizational readiness, and the overall level of readiness among the set of participating organizations was high.
The discrepant results in the inter-rater reliability statistics [i.e., ICC(1) and ICC(2)] and inter-rater agreement statistics (r
WG(J)
and AD
M(J)
) for change efficacy highlight an important difference in how these two types of statistics measure ‘similarity’ in organizational members’ ratings. Inter-rater reliability indicates the relative consistency (or rank order consistency) in ratings, whereas inter-rater agreement indicate the absolute consensus (or interchangeability) in ratings. LeBreton and colleagues [15] observe that strong levels of inter-rater agreement can be masked by subtle inconsistencies in the rank orders of ratings, especially when the between-unit variance is restricted (e.g., all organizations are rated high or low). For example, two sets of ratings on a seven-point scale (rater one = 7, 6, 6, 7, 7, 6, 6, 7, 7; rater two = 6, 7, 7, 7, 7, 6, 6, 6, 7, 7) would generate a mean r
WG(J)
of 0.94 and an ICC(1) of only 0.04. LeBreton et al. encourage investigators to examine multiple indicators of inter-rater reliability and inter-rater agreement, but caution that one type of statistic may be more relevant than another depending on the research question. In this case, we were interested in the psychometric question of whether individual perceptual data on readiness could be aggregated to the organization-level of analysis. Our field study results suggested that sufficient consensus existed within the INGOs to measure readiness at the organizational level; however, our results also suggested, for the participating INGOs, between-group variation in change efficacy scores might be insufficient to warrant an organization-level analysis of the determinants or outcomes of this facet of readiness.
Although ORIC shows promise, further psychometric assessment is warranted. Specifically, the measure should be tested for convergent, discriminant, and predictive validity. Convergent validity could be assessed by comparing ORIC to other reliable, valid, but much longer measures, such as the Organizational Readiness for Change Assessment [5] or the Texas Christian University (TCU) Organizational Readiness for Change instrument [28]. Discriminant validity could be assessed by comparing ORIC to measures of constructs related to, yet distinct from, organizational readiness for change (e.g., organizational culture). Finally, predictive validity could be assessed by examining the association of ORIC with hypothesized outcomes of readiness, such as championing change and implementation effectiveness [8]. Assessment of predictive validity is particularly important for determining whether organizational level readiness should be a key priority for leaders of organizational change efforts.
Limitations
This study had a few limitations. First, one item in the Change Commitment Scale was dropped inadvertently in the field test in study four. Results from our previous three studies suggest that including the item would not have adversely affected the reliability and validity of the scale in study four. Nevertheless, future field studies should check this.
Second, we carried forward to study two one item (i.e., ‘We want to perform this change’) that did not satisfy the conditions for content adequacy in study one. We did so because we believed participants may not have had sufficient semantic context to determine the meaning of ‘want.’ We obtained encouraging results for this item from the exploratory and confirmatory factor analysis in the laboratory study (study two and 3) and the confirmatory factor analysis in the field study (study four). Nevertheless, further testing on this item is warranted.
Third, in study two and study three we asked graduate and undergraduate students to assess organizational readiness as if they were an employee of the hospital described in the vignette. This approach may raise concerns about the validity of these data because the students are not in fact hospital employees. However, we believe this approach is appropriate for our study because it has been used in several previous studies [29, 30], the students were enrolled in programs (health policy and management or health behavior) that familiarize students with healthcare settings, and the results of our field test with actual employees (study four) support findings from study two and study three.
Fourth, we could not test for a higher-order organizational readiness for change construct because the structural component of the model would be under-identified with only two factors. Our field study results suggest, however, that constructing a higher-order factor might not be advisable given the moderate correlation of the Change Commitment and Change Efficacy scales and the differences they exhibited in inter-rater reliability. Researchers might wish to retain the scales rather than combine them because they capture related, yet distinct facets of organizational readiness to implement change.
Finally, in study four we did not collect information about the efforts organizational leaders undertook to increase readiness of employees. Such information could have proved useful for assessing why inter-rater reliability for the Change Efficacy Scale did not support aggregation of the individual-level data into an organizational-level mean. For example, it is possible that some individuals were provided more information about the impending change than others, resulting in different views on the organization’s readiness. Finally, because each organization in study four exhibited a high level of readiness, it would be useful to test ORIC in a sample with more variation in readiness between organizations.