We used an interrupted time series design as the overall intervention evaluation approach with comparisons to assess the effect of monthly feedback reports in nine LTC units in four facilities. We conducted a concurrent process evaluation to evaluate staff response to the intervention. We report these results in a separate paper, currently under review. The project received ethics approval from the Health Research Ethics Board, Committee B, at the University of Alberta and operational approval from the two LTC organizations participating in the study. The full protocol for the entire Data for Clinical Improvement and Excellence (DICE) project was published in two protocol papers . We provide brief descriptions of key elements in this paper, in which we report on the DICE-LTC (Long-Term Care) component of the full project .
As we describe in our protocol paper for a social network sub-study within DICE-LTC , we worked from an underlying conceptual model built on the Theory of Planned Behavior, in which we posited that feedback reports would work through multiple paths (attitudes and social norms in particular) to influence intention to change behavior. We were unable to measure provider behavior directly through this study, but we measured longitudinal measures of intention to change behavior, reported in the process evaluation paper. We expected an effect at the level of the LTC nursing unit, a geographic sub-unit of the entire facility, which is relatively self-contained with respect to most of the frontline, day-to-day, and shift-to-shift direct care providers, most of whom are health-care aides (HCAs). The geographic boundaries of the nursing unit are quite permeable, and other types of providers, notably those in the allied health professions including occupational, physical, and recreational therapists, social workers, and others, traverse unit boundaries. However, in our study, geographic nursing units all had a single, clearly identified unit manager and were understood as important and relatively independent components of the organization by staff in all four facilities. Our intervention included handing feedback reports to individual providers of all kinds, as there were few opportunities to deliver reports to groups of staff. We did, however, expect that providers would talk to each other about the reports, and we asked them questions to elicit an understanding of how much they discussed the reports, as well as some information about the purposes of the discussion, described in the process evaluation paper.
Settings and sample
The project was conducted in nine LTC nursing units in four continuing care facilities in Edmonton, Alberta, Canada. The four facilities were part of two organizations providing a range of continuing care services in the community. All four facilities had implemented the Resident Assessment Instrument Minimum Data Set (RAI) version 2.0 (http://www.interrai.org), although two facilities had implemented RAI 2.0 between 3 and 5 years prior to the beginning of the DICE-LTC feedback intervention, while the other two had only implemented this system between 6 and 12 months prior to the project beginning. We initially hypothesized that maturity of RAI assessments might be an important factor in the uptake and effectiveness of the feedback reports.
The feedback reports were developed during a pilot study conducted in two nursing homes in the Edmonton area in late 2007 and early 2008. We used data from the RAI 2.0 to measure resident-level outcomes, and this served as the data source for the feedback reports. The RAI 2.0 assessment tool covers a wide range of process and outcome data at the individual resident level, and assessments are updated quarterly for each resident. We reported on measures of pain frequency and intensity, risk for and occurrence of falls, and depression prevalence, all aggregated to the unit level. These three areas were among the top eight domains identified as priorities through the pilot project and were agreed upon by senior leadership in both participating organizations. Data were extracted at the resident level from vendor servers every month and stripped of personal identifiers, except for the unit on which each resident lived, before being sent to the research team.
Following definitions used by the Canadian Institute for Health Information, we derived the pain scale from two items on the RAI 2.0 assessment, one measuring pain frequency and the other measuring pain intensity. The scale is scored 0 for no pain, 1 for pain less than daily, 2 for daily pain of moderate intensity, and 3 for daily severe pain . The depression rating scale is scored on a 0 to 14 range and is derived from seven RAI 2.0 items, from making negative statements to crying and tearfulness. A score of 3 or more indicates possible depressive disorder and residents should be further evaluated . Falls were defined as any fall occurring in the 31 to 180 days prior to the current assessment, and falls risk were defined as a combination of requiring assistance for locomotion, problems with balance, dizziness or vertigo, and unsteady gait. The assessments were completed by staff in each facility following comprehensive instructions and training required for RAI 2.0 assessments. We aggregated these into proportions for residents on each of the nine units.
Reports were primarily graphic on one sheet of paper, front and back, printed in color, and included a cover sheet with details about the data sources and the comparison units. We provided a sample of this report in the DICE-LTC protocol paper . We provided feedback using monthly time points from months 2 to 11, after which we switched to showing quarterly time points for months 12 and 13. Reports were hand-delivered by project staff, who were all research assistants with minimal training, in each of the nine LTC units during a consistent week in each month for each of the 13 months of the intervention period. Each report was specific to the unit, and all direct care providers and unit managers received the unit-specific reports. Facility administrators received reports for each of their units prior to report distribution on the units. We included facility administrators, nurse managers, and frontline direct care staff, including registered nurses, licensed practical nurses, HCAs, physical therapists, recreational therapists, occupational therapists, pharmacists, social workers, and other allied health providers. The goal of the feedback report distribution was to ensure that frontline staff received the reports directly.
In addition to the feedback intervention, research assistants also distributed and collected post-feedback surveys in each intervention unit as part of an extensive process evaluation. In the first section of the post-feedback survey, we asked questions about response to the reports. In another section, we asked about intentions to change behavior with respect to assessing pain among residents. A sample survey instrument is provided in the protocol paper , and the results detailing provider response to the feedback reports are reported in the process evaluation paper.
In addition to the data included in the feedback reports delivered to participating units and facilities, we also requested data from the same period for nine units in three additional facilities, matched as closely as possible to the facilities participating in the study. All three comparison facilities came from one of the two larger organizations participating in the study. These provided comparison units to control for secular trend over the baseline, intervention, and follow-up periods. We also included pressure ulcer prevalence, a quality indicator not included on the feedback reports, as an additional check on secular trends within the participating units.
Our primary analysis used segmented regression on the interrupted time series data. In this approach, all data are aggregated to a single observation at multiple time points at equal intervals. In our case, they were in months . Given a data set aggregated to equal interval time points, we specify our linear regression model as:
is the outcome in month t;
time is the time in months at time t from the start of the observation period; it ranges from 1 to 25 months;
feedback is an indicator for time t occurring before (feedback = 0) or after (feedback = 1) the feedback report, which was implemented in January 2009 in the series;
time after intervention starts is the number of months after the intervention starts at time t, coded 0 before the feedback intervention starts and (time-6) after the feedback intervention starts;
end of intervention is an indicator for time t occurring before or after the end of the feedback intervention, which was after January 2010 in the time series;
time after intervention ends is the number of months after the intervention at time t, coded 0 before the end of the feedback report and (time-19) after the end of the feedback report;
is the error term at time t that represents the unexplained random variation in the model.
In this model, we estimated the regression parameters as follows:
0 is the baseline level of the outcome;
1 is the slope of the regression line (trend line) prior to the feedback report;
2 is the change in level immediately following the feedback report start;
3 is the change in regression slope during the feedback report period;
4 is the change in level immediately after the feedback report ends;
5 is the change in regression slope in the post-feedback report period.
We conducted separate segmented regression analyses for the intervention and comparison facilities in each of the four outcomes: proportion of residents with pain scores greater than 2, proportion with depression scores greater than 3, proportion with falls, and proportion with pressure ulcers. We provide both visual representations of the results in the form of time series graphs and tables of the parameter estimates from the regression analyses for statistical inference. We tested the residuals from each regression analysis for autocorrelation using the Durbin-Watson test, and if this was significant, we used the Prais-Winsten regression to adjust for autocorrelation ,. The Durbin-Watson test is a check of first-order serial auto-correlation due to repeated measures (AR1); the Prais-Winsten regression adjusts for first-order serial auto-correlation by assuming an AR1 error term. When the Durbin-Watson test did not show significant autocorrelation, we present the results of ordinary least squares regression of the time series data. Ordinary least squares regression is more commonly termed linear regression. In each month, we included only residents who had received a new assessment during that month, which lessened the degree of autocorrelation between months as different resident data were included each month. We used this approach in the monthly feedback reports also, to increase the degree to which the data would change from month to month and provide new information.
We initially powered our sample size on falls as the primary outcome as we designed the project, using data reported in the literature on the prevalence of falls in LTC settings similar to those included in our study, and on effect sizes for change in falls due to quality improvement interventions similar to our planned feedback intervention. We did not, in our original power calculations, focus on the interrupted time series design, but instead used a standard approach with an adjustment for clustering. However, our final sample included all available LTC units in the four facilities participating in the project. While the number of residents across all available units met our calculated sample size in designing the project, we did not take into account the interrupted time series approach to analysis. As a result, our sample size calculation may have overestimated our power. In addition to the primary analyses, we also conducted secondary analyses dividing the intervention units into those with mature RAI 2.0 systems compared to those with recent implementation of RAI 2.0. We included secondary hypotheses about the effects of mature vs. new data systems in our original proposal .
We did not risk-adjust our outcomes. We were focused on reporting observed data to the staff providing care to residents, and while risk adjustment might be necessary for comparing across units for performance management or reporting purposes, for quality improvement, staff needed to understand the experience of residents in their care. The data used in this summative evaluation are the same as those provided to staff throughout the feedback intervention. We included data from 6 months prior to the start of the feedback intervention, which began in January 2009, as pre-intervention data, and from 6 months after the end of the intervention, as post-intervention data. The timeline in Figure 1 shows the timing of data extraction, feedback report delivery, and post-intervention data.
Variables included in the analysis
Following the segmented regression analysis approach, the only variables included in these analyses are the time points based on the study timeline and outcome data at each time point. The intervention and comparison units are implicit in the different regression analyses.