Data for improvement and clinical excellence: report of an interrupted time series trial of feedback in long-term care

Long-term care (LTC) settings, also described as nursing homes or long-term care facilities, exist in some form in most developed countries. In Canada and the United States, institutional long-term care facilities are usually stand-alone facilities or sometimes part of a hospital. They provide ongoing, continuing care to physically frail, often cognitively impaired, residents who live there, for the most part, until their death. Many facilities provide both rehabilitation and long-term care [1]. The focus of our project was on LTC units, not those providing post-acute care or other rehabilitation services. LTC units usually operate under resource constraints and rely heavily on non-professional staff to provide the majority of care to physically frail and typically cognitively impaired older adults. In these settings, resources for quality improvement are difficult to obtain, and trials of quality improvement interventions are relatively rare compared with other health-care settings.

There is a growing evidence base for care in LTC settings, including evidence-based approaches to pain management, falls prevention, and other important topics [2]-[5]. However, getting evidence into practice is not well understood, and relatively little evidence exists to date on effective interventions for implementing evidence-based practices and changing provider behavior in these settings [6]. Many interventions have been rigorously tested across multiple settings and conditions, and some evidence exists for the effectiveness of specific interventions in implementing evidence-based practice through changing provider behavior [7]-[10]. One of these is audit combined with feedback reports (referred to as feedback interventions). Feedback interventions have demonstrated modest effect in promoting desired behavior change among health-care providers across settings and provider types [11],[12]. To date, few feedback interventions have been conducted in LTC settings. Those that have been conducted have included only a subset of the providers who give care to residents, typically the most educated professionals [13]. A major barrier to using feedback interventions in LTC has been the lack of data on care processes and outcomes, but this has been addressed in many jurisdictions by the adoption of a Minimum Data Set which assesses resident care processes and outcomes [14]-[17].

The probable mechanism by which feedback interventions have their main effect is in providing people with information about their own performance [18]-[20]. The results, particularly among people who have not received data-based feedback on their performance in the past, may be to increase motivation to change behavior. Feedback reports have often been used in conjunction with other interventions [11]. Our intention in conducting this study was to keep the intervention simple and relatively low cost by using feedback reports alone without other interventions. The low cost of the type of feedback intervention we studied is supported by widespread availability of data on resident status and outcomes [14],[15], which do not require significant additional cost for audit data to construct the feedback reports [14],[15],[21]. In this paper, we report the summative resident outcomes of a feedback intervention delivered to all staff in four LTC facilities. Our primary hypothesis was that a consistent, long-duration, resident-focused feedback intervention would improve resident outcomes compared with settings which did not receive the feedback intervention and compared with an outcome not included in the feedback report.

We used an interrupted time series design as the overall intervention evaluation approach with comparisons to assess the effect of monthly feedback reports in nine LTC units in four facilities. We conducted a concurrent process evaluation to evaluate staff response to the intervention. We report these results in a separate paper, currently under review. The project received ethics approval from the Health Research Ethics Board, Committee B, at the University of Alberta and operational approval from the two LTC organizations participating in the study. The full protocol for the entire Data for Clinical Improvement and Excellence (DICE) project was published in two protocol papers [22]. We provide brief descriptions of key elements in this paper, in which we report on the DICE-LTC (Long-Term Care) component of the full project [22].

As we describe in our protocol paper for a social network sub-study within DICE-LTC [22], we worked from an underlying conceptual model built on the Theory of Planned Behavior, in which we posited that feedback reports would work through multiple paths (attitudes and social norms in particular) to influence intention to change behavior. We were unable to measure provider behavior directly through this study, but we measured longitudinal measures of intention to change behavior, reported in the process evaluation paper. We expected an effect at the level of the LTC nursing unit, a geographic sub-unit of the entire facility, which is relatively self-contained with respect to most of the frontline, day-to-day, and shift-to-shift direct care providers, most of whom are health-care aides (HCAs). The geographic boundaries of the nursing unit are quite permeable, and other types of providers, notably those in the allied health professions including occupational, physical, and recreational therapists, social workers, and others, traverse unit boundaries. However, in our study, geographic nursing units all had a single, clearly identified unit manager and were understood as important and relatively independent components of the organization by staff in all four facilities. Our intervention included handing feedback reports to individual providers of all kinds, as there were few opportunities to deliver reports to groups of staff. We did, however, expect that providers would talk to each other about the reports, and we asked them questions to elicit an understanding of how much they discussed the reports, as well as some information about the purposes of the discussion, described in the process evaluation paper.

Analysis

Our primary analysis used segmented regression on the interrupted time series data. In this approach, all data are aggregated to a single observation at multiple time points at equal intervals. In our case, they were in months [25]. Given a data set aggregated to equal interval time points, we specify our linear regression model as:

$\begin{array}{l}{Y}_t=b_0+b_1*{\mathrm{time}}_t+b_2*{\left(\mathrm{feedback}\right)}_t+b_3*{\left(\mathrm{time}\mathrm{after}\mathrm{intervention}\mathrm{starts}\right)}_t+b_4*\\ {\left(\mathrm{end}\mathrm{of}\mathrm{intervention}\right)}_t+b_5*{\left(\mathrm{time}\mathrm{after}\mathrm{intervention}\mathrm{ends}\right)}_t+e_t\end{array}$

where

Y _t is the outcome in month t;

time is the time in months at time t from the start of the observation period; it ranges from 1 to 25 months;

feedback is an indicator for time t occurring before (feedback = 0) or after (feedback = 1) the feedback report, which was implemented in January 2009 in the series;

time after intervention starts is the number of months after the intervention starts at time t, coded 0 before the feedback intervention starts and (time-6) after the feedback intervention starts;

end of intervention is an indicator for time t occurring before or after the end of the feedback intervention, which was after January 2010 in the time series;

time after intervention ends is the number of months after the intervention at time t, coded 0 before the end of the feedback report and (time-19) after the end of the feedback report;

e _t is the error term at time t that represents the unexplained random variation in the model.

In this model, we estimated the regression parameters as follows:

b ₀ is the baseline level of the outcome;

b ₁ is the slope of the regression line (trend line) prior to the feedback report;

b ₂ is the change in level immediately following the feedback report start;

b ₃ is the change in regression slope during the feedback report period;

b ₄ is the change in level immediately after the feedback report ends;

b ₅ is the change in regression slope in the post-feedback report period.

We conducted separate segmented regression analyses for the intervention and comparison facilities in each of the four outcomes: proportion of residents with pain scores greater than 2, proportion with depression scores greater than 3, proportion with falls, and proportion with pressure ulcers. We provide both visual representations of the results in the form of time series graphs and tables of the parameter estimates from the regression analyses for statistical inference. We tested the residuals from each regression analysis for autocorrelation using the Durbin-Watson test, and if this was significant, we used the Prais-Winsten regression to adjust for autocorrelation [26],[27]. The Durbin-Watson test is a check of first-order serial auto-correlation due to repeated measures (AR1); the Prais-Winsten regression adjusts for first-order serial auto-correlation by assuming an AR1 error term. When the Durbin-Watson test did not show significant autocorrelation, we present the results of ordinary least squares regression of the time series data. Ordinary least squares regression is more commonly termed linear regression. In each month, we included only residents who had received a new assessment during that month, which lessened the degree of autocorrelation between months as different resident data were included each month. We used this approach in the monthly feedback reports also, to increase the degree to which the data would change from month to month and provide new information.

We initially powered our sample size on falls as the primary outcome as we designed the project, using data reported in the literature on the prevalence of falls in LTC settings similar to those included in our study, and on effect sizes for change in falls due to quality improvement interventions similar to our planned feedback intervention. We did not, in our original power calculations, focus on the interrupted time series design, but instead used a standard approach with an adjustment for clustering. However, our final sample included all available LTC units in the four facilities participating in the project. While the number of residents across all available units met our calculated sample size in designing the project, we did not take into account the interrupted time series approach to analysis. As a result, our sample size calculation may have overestimated our power. In addition to the primary analyses, we also conducted secondary analyses dividing the intervention units into those with mature RAI 2.0 systems compared to those with recent implementation of RAI 2.0. We included secondary hypotheses about the effects of mature vs. new data systems in our original proposal [22].

We did not risk-adjust our outcomes. We were focused on reporting observed data to the staff providing care to residents, and while risk adjustment might be necessary for comparing across units for performance management or reporting purposes, for quality improvement, staff needed to understand the experience of residents in their care. The data used in this summative evaluation are the same as those provided to staff throughout the feedback intervention. We included data from 6 months prior to the start of the feedback intervention, which began in January 2009, as pre-intervention data, and from 6 months after the end of the intervention, as post-intervention data. The timeline in Figure 1 shows the timing of data extraction, feedback report delivery, and post-intervention data.

We report on a feedback intervention which we were able to deliver consistently over a 13-month period across nine LTC units in four facilities. Given the low level of resources available in these facilities, completing the intervention with all nine original units, and with a relatively high rate of participation among providers, is an important achievement. Attrition rates for quality improvement interventions is a critical problem among LTC facilities, usually attributed to high rates of staff turnover, low levels of education, and high workload, and many quality improvement interventions have not been able to retain all participating units for the full project [6].

Based on our primary outcome, falls, the feedback intervention was not effective in changing resident outcomes. We had hoped to demonstrate a reduction in falls as a result of the intervention. Instead, we found no immediate change in the level or number of falls at the outset of the intervention, when the effect of feedback might be highest, and a modest but significant increase in the rate of falls over the intervention period. This latter finding was contrary to our expectation. The lack of any significant change in either the control condition (pressure ulcers) or comparison sites suggests that there were few, if any, secular trends during the intervention period which would have changed resident outcomes.

In strict terms, then, we did not find the positive effect of feedback interventions alone that we had hoped to find. This is consistent with the literature on feedback interventions, which suggests that feedback interventions are likely insufficient by themselves to effect sufficient provider behavior change to change outcomes. This is mediated significantly by the degree of uptake of the feedback report, the conditions under which it was received, and other factors, based on the Theory of Planned Behavior, which might result in behavior change following receipt of a feedback intervention. In addition to factors related to the Theory of Planned Behavior, which include attitudes and beliefs, social and professional norms, and perceived behavioral control [28], factors related to the feedback intervention itself are likely important. These include the timeliness of the data, the perceived validity of the data, the source of the feedback, and other factors such as the perceived sign of the information in the report (feedback perceived as positive, negative, or neutral) [22]. We describe findings related to a number of these factors in the process evaluation paper. Briefly, a high proportion of the target provider groups received, read, and understood the feedback report. A smaller proportion discussed it with other staff in the unit. Many participants described an interest in receiving further data as well as further information about how to make changes to benefit residents, but it was beyond the scope of this project to provide information. The lack of time and opportunity to spend time in activities other than offering direct care to residents was a serious and continual constraint on our activities.

Despite that, we note that there was a significant change in pain scores at the beginning of the intervention period, in the direction we had expected. It is notable that the post-feedback survey we administered 1 week after the feedback report distribution asked a set of questions about intention to change pain assessment behavior among direct care staff. We report on these findings in the companion process evaluation paper, but note that this may have unintentionally worked as a co-intervention with the feedback report, as it may have acted to focus participant attention on pain assessment. While this was an unexpected effect, not one we intentionally designed into the intervention, similar results have been observed previously, and there is a literature on what is called “mere measurement” or the “question-behavior effect” [29],[30]. This is an intriguing finding that deserves follow-up work in future implementation studies.

We focused on pain assessment because it had been ranked highly as an important quality improvement issue across all provider groups in our pilot study and had been the subject of attention in both organizations participating in the intervention. Managers strongly endorsed pain assessment as an important issue for measurement, and prior to initiating the feedback intervention, several sites had engaged in education about pain assessment and management. The graphical time series for this outcome depicted in Figure 3 shows that the intervention sites demonstrated a decrease in proportion of residents with high pain scores at the outset of the intervention, followed by a period of several months of low scores which slowly climbed again towards the end of the intervention period. In the comparison sites, there was a later drop in scores, but then an inconsistent pattern of increasing and decreasing scores over most of the intervention period. From the coefficient estimates, we see that the change in scores at the beginning of the intervention period was significant for the intervention units, while the changes for the comparison units were not, even though the comparison units were given the same education, unrelated to our intervention, prior to the intervention period.

An important note is that this is the first attempt, to our knowledge, to deliver feedback reports to all direct care providers in LTC settings, including health-care aides. Most previous studies have excluded aides, often without justification. The justifications that have been used frequently include feasibility, lack of educational preparation, and that the primary drivers of quality improvement are professional providers, usually registered or licensed practical nurses. We argue that because aides are the majority of providers in LTC settings, excluding them from involvement in quality improvement interventions may have unintended negative consequences and may decrease the effectiveness of the intervention. In this report, and in the companion process evaluation paper, we have demonstrated that including aides is feasible and that aides report being able to understand feedback reports.

Most prior quality improvement efforts in LTC settings have not included control or comparison groups, and most have used pre- and post-intervention designs that make causal attribution very difficult. We included two different kinds of comparisons to guard against both attention effects and secular trend. We ruled out secular trend as a cause of the significant changes we saw. We also saw relatively little change in the desired direction as a result of this modest, intentionally parsimonious intervention. The one change in the desired direction we did find may have been related to an unintended co-intervention, namely the focus on pain assessment in our measurement instruments. It is difficult, based on our findings, to argue for continuing to conduct feedback interventions alone in LTC settings, although feedback reports can be an important building block in intervention design. Especially when data already exist, this can be a relatively low-cost component to an intervention that may enhance the effect of additional components such as education, goal setting, and other relevant behavior change techniques [31],[32].

We believe that our findings should provide qualified evidence for managers on the usefulness of existing data from sources like the RAI 2.0, which are increasingly available in many jurisdictions. A major impetus for our project was the desire to use the data that staff so arduously collect on a quarterly basis, to improve the quality of care and quality of life of residents. We have shown that this work is feasible, although achieving significant outcome improvement may require additional intervention beyond feedback alone.

AES conceived of the study, drafted and revised the manuscript, and is responsible for the overall analysis. CS conceived of the study, reviewed and contributed to drafts, and shares responsibility for its conduct. MABB conducted the data analysis and reviewed and critically revised the manuscript. KDF reviewed and critically revised the manuscript. All authors read and approved the final manuscript.