Implementation success for healthcare quality and safety interventions can vary significantly across teams. Assessing differences in team context and progress can help QI team members make adjustments over the course of the intervention and help researchers design more effective interventions. In addition, identifying factors associated with successful teams can increase the likelihood of implementation success for future teams. However, measures used for these assessments must be reliable, valid, and responsive in order to be useful for these purposes.
In this study, we examined the measurement properties of the TCT, a short instrument that has been used to track implementation progress for an intervention to reduce bloodstream infections within ICUs . TCT measures evaluated in this study included participation in intervention components, perceptions of unit performance on infection prevention behavior, and key barriers to team progress including lack of leadership support and physician engagement, which are often central to implementation success [6, 19–21]. Our study found support for the temporal stability, construct validity, and responsiveness of QI team member reports on intervention activities, perceived unit-level behaviors, and barriers to team progress from this instrument. At both the item and aggregate level, the TCT measures we evaluated were responsive during periods of high implementation activity and stable during periods of low implementation activity. Furthermore, the general trajectories in behaviors and barriers were as predicted, with greater number of prevention behaviors and fewer barriers observed over the course of the intervention. The greatest changes took place within the first six months of the intervention. These findings attest to the value of these measures for detecting change and tracking the course of implementation progress.
Construct validity for the evaluated measures was generally good. A hypothesized overall relationship between infection prevention behaviors and barriers was supported by a significant moderate correlation. Individual barrier items also demonstrated significant associations, with items related to buy-in from physician, nursing, and other clinical staff being the strongest. This finding appears reasonable given that improved ICU behavior depends on the commitment of the entire ICU staff, not just of the QI team members.
Convergent and discriminant validity results were also as hypothesized. The TFS , which assesses aspects of team effectiveness, provides a good overall match to the TCT barrier items and the expected association is supported by a significant and moderately strong correlation. Similarly, moderate to strong correlations were observed for TCT items to corresponding TFS subscales. Discriminant validity is also good, with the TCT barrier items demonstrating weak and non-significant association with the overall and staffing/resource subscale of the PES, a more general measure of nursing work environment. Together, these findings indicate that the TCT barrier items provide valid measures of team effectiveness and functioning.
However, barriers to team progress at baseline did not predict time to zero CLABSI rate as hypothesized. Constraint in the variance of this outcome may have limited our ability to detect an association between our TCT measures and the key clinical outcome of interest in the intervention. In addition, while barriers on the QI team very likely affect intervention outcomes, they may not be the strongest influences. Although these results did not confirm our original hypothesis regarding predictive validity, most findings from this study support the reliability, validity, and responsiveness of the TCT measures we examined. Furthermore, the low burden of this instrument, requiring less than 10 minutes to complete and reporting only once a month, contributes to the instrument's feasibility for use in busy clinical settings such as ICUs.
Demonstration of a practical, valid, and responsive measure of context is important given the growing interest in better theoretical and practical understanding of how context affects implementation success of effective QI interventions [5, 7, 22, 23]. Among the contextual elements that may influence QI success are staffing, work environment, safety climate, teamwork, implementation activities, organizational culture, learning and mindfulness, leadership support, and engagement of participants. The TCT emphasizes key internal processes and group psychosocial traits of the QI team. Although there is little consensus around how to define and measure implementation context, team functioning and effectiveness have been examined in a number of QI studies . Unfortunately, few studies of team effectiveness have used validated instruments.
Additional work to more clearly define and validate measures of implementation context will be important for advancing the research in this area. Several instruments, although not specific to healthcare, have been validated. Anderson and West developed and validated the factor structure, internal consistency, predictive validity, and within-team consensus for the Team Climate Inventory, which focused on the climate for innovation within work groups . Wheelan and Hochberger developed and validated the Group Development Questionnaire, a general measure of the trajectory of group formation and development . These instruments are clearly valuable to the understanding of team-based interventions. However, the generalized concepts, while useful for research purposes, may be less relevant for QI team members seeking opportunities to achieve success in implementing a specific intervention. Furthermore, these instruments are often longer and, therefore, less feasible to complete on a routine basis in busy clinical settings. Finally, the concepts they measure may not reflect the rapid changes that take place within a QI intervention. Teams and collaborative faculty often need more frequent feedback on the effects of changes they have made to improve care.
Measures of implementation context specific to QI collaboratives have been developed and are being validated, including a 14-item instrument by Dückers et al.  measuring team organization, internal support (leadership and organization), and external support (external change agents). The TFS provides a measure of team functioning with demonstrated reliability that has been used to assess team effectiveness in QI collaborative . Our study contributes to this literature on team context measures for QI efforts by demonstrating the responsiveness of the tool to changes in barriers to team progress and unit-level infection prevention behaviors over the course of the intervention. We also demonstrated the impact of team functioning on intervention effectiveness through the significant negative association between barriers to team progress and unit-level behaviors in infection behaviors. Finally, our study demonstrated reliability for TCT through good temporal stability during a stable phase of the intervention period. These are critical properties to assess when tracking progress of a team-based intervention.
Our study has several limitations. First, not all items included in the TCT were evaluated as part of this study. Validity of items related to the number of times the teams met with executives and hospital boards, nursing turnover, and diffusion practices are challenging to assess for the measurement properties of interest, because they represent events that may not consistently take place from month to month and gold standard data for validation are not readily available. Second, many items included in this instrument-such as the elements of CUSP, educational activities, and infection prevention behaviors-are specific to the interests and needs of the target intervention. This will limit the applicability of this measure and our findings to other QI interventions. However, there are also benefits of this approach to assessing implementation context. Specifically, tailoring measures to QI intervention can make the assessments more relevant and useful for teams participating in a specific intervention. For example, team members can discuss specific barriers with each other and senior leaders and resolve issues that hamper implementation progress. As we expect the diffusion of the bloodstream infection prevention intervention to expand, the findings from this study should offer useful insights for future QI teams. Third, the lack of significance for several correlations with absolute r value between 0.21 and 0.33 that we observed in the discriminant validity analyses may be due to the small number of ICUs included. However, given that most of the correlations were weak, we expect that our overall conclusion would be robust even if a larger sample of ICUs had been available. Fourth, the findings of validity analyses that relied solely upon TCT data (e.g., the association of prevention behaviors and team barriers) may be artificially strengthened due to common method bias. Fifth, we did not first establish the convergence of member reports for the team level measures, because we relied on a conceptual basis for 'team' (i.e., assignment to team) rather than an empirically-driven one that requires substantial correlation of member reports. We reasoned that member perceptions cannot be assumed to converge because team members likely have different skills, roles, and experience and, further, may have different observations of team experience. Despite these differences, individual members are core contributors to and observers of the team. Therefore, their reports, in aggregate, provide valid reflections of team experience, irrespective of their level of convergence. Finally, as a longitudinal effectiveness project conducted under naturalistic conditions, data quality was a concern. Team-level response rate was challenging to determine because QI team composition can change over time. There was little assurance that member responses were not seen by others within or outside the QI team because conditions were not controlled by the research team. Many ICUs had months when TCT data were not submitted. This missing data led to the exclusion of some ICUs from temporal stability and responsiveness analyses. However, a comparison of the characteristics and CLABSI outcomes of included and excluded ICUs found no significant differences, suggesting that our conclusions regarding the psychometric properties of the TCT were sound.