- Study protocol
- Open Access
Improving patient safety by optimizing the use of nursing human resources
Implementation Science volume 10, Article number: 89 (2015)
Recent ecological studies have suggested that inadequate nurse staffing may contribute to the incidence of adverse events in acute care hospitals. However, longitudinal studies are needed to further examine these associations and to identify the staffing patterns that are of greatest risk. The aims of this study are to determine if (a) nurse staffing levels are associated with an increased risk of adverse events, (b) the risk of adverse events in relationship to nurse staffing levels is modified by the complexity of patient requirements, and (c) optimal nurse staffing levels can be established.
A dynamic cohort of all adult medical, surgical, and intensive care unit patients admitted between 2010 and 2015 to a Canadian academic health center will be followed during the inpatient and 7-day post-discharge period to assess the occurrence and frequency of adverse events in relationship to antecedent nurse staffing levels. Four potentially preventable adverse events will be measured: (a) hospital-acquired pneumonia, (b) ventilator-associated pneumonia, (c) venous thromboembolism, and (d) in-hospital fall. These events were selected for their high incidence, morbidity and mortality rates, and because they are hypothesized to be related to nurse staffing levels. Adverse events will be ascertained from electronic health record data using validated automated detection algorithms. Patient exposure to nurse staffing will be measured on every shift of the hospitalization using electronic payroll records. To examine the association between nurse staffing levels and the risk of adverse events, four Cox proportional hazards regression models will be used (one for each adverse event), while adjusting for patient characteristics and risk factors of adverse event occurrence. To determine if the association between nurse staffing levels and the occurrence of adverse events is modified by the complexity of patient requirements, interaction terms will be included in the regression models, and their significance assessed. To assess for the presence of optimal nurse staffing levels, flexible nonlinear spline functions will be fitted.
This study will likely generate evidence-based information that will assist managers in making the most effective use of scarce nursing resources and in identifying staffing patterns that minimize the risk of adverse events.
Adverse events (AEs) are injuries caused by medical management rather than by the underlying condition of the patient . AEs affect between 2.9 and 16.6 % of all hospitalized patients, and studies have estimated that 30 to 58 % of all AEs are preventable [2–9]. Preventable AEs are associated with significant morbidity and mortality, with 20 to 25 % of all events resulting in permanent disability or death [1, 6]. Preventable AEs are also very costly, representing between US$17 and US$29 billion in additional health care costs annually . Moreover, human errors have been identified as one of the largest contributors to the occurrence of preventable AEs in acute care hospitals [1, 10].
Nurses are the largest component of the acute care hospital workforce, representing 30 to 40 % of hospital staff, accounting for 25 % of the total acute care hospital operating budget and 44 % of direct care costs [11, 12]. Nurses play an important role in the prevention of AEs, as nursing actions such as the ongoing monitoring of patient conditions are related to better patient outcomes [13, 14]. To accomplish these actions, an adequate number of nurses with the appropriate clinical knowledge and skills are needed. Indeed, recent ecological studies have suggested that understaffing and the use of less qualified nursing staff may be important contributors to the incidence of AEs in acute care hospitals [13–16].
The next step in investigation is to determine the optimal nurse staffing levels (i.e., the optimal numbers of nurses and mixes of skills, education, and experience) that are needed to minimize the risk of AEs. This is especially relevant given that the current shortage of nurses is expected to worsen in the coming years  and because nursing managers are implementing a variety of staffing policies—for which little empirical evidence is available—to mitigate the shortage of nurses and maintain the safety of patient care (e.g., using overtime hours, hiring less qualified nursing staff such as nursing assistants or registered practical nurses) [18, 19]. In addition, with the current economic downturn, many hospitals are under pressure to identify less costly staffing plans, thus furthering the routine use of less skilled nursing workers and of overtime hours to meet peak staffing demands [20, 21]. To help decision makers establish safe evidence-based staffing policies, the temporal relationships between nurse staffing levels and the incidence of AEs, the thresholds for safe staffing levels, and the potential modification of these relationships by levels of patient complexity need to be determined by conducting longitudinal studies.
Until recently, the detailed daily patient and staffing data needed to conduct such an investigation were not readily available. With the advent of electronic health records (EHR) and digital capture of payroll and staffing data, an exciting opportunity has emerged to address these important questions in the management of nursing staff in acute care hospitals. Taking advantage of this new opportunity, the objectives of this study are to determine if (a) nurse staffing levels are associated with an increased risk of AEs and if particular patterns of nurse staffing are of greatest risk, (b) the risk of AEs in relationship to nurse staffing levels is modified by the complexity of patient requirements, and (c) optimal nurse staffing levels can be established.
The past two decades have witnessed the publication of a large number of studies examining the relationships between nurse staffing levels and AEs. As a group, these studies have provided ecological evidence that acute care hospitals with higher nurse staffing levels have lower rates of mortality and AEs [13–16]. Moreover, they have suggested that while using overtime hours and less qualified nursing staff may provide acceptable numbers of people at bedside, these staffing strategies are also associated with a higher risk of AEs [21–23]. While these studies have made important contributions to the field, the strength of the evidence they have provided has been challenged [13, 24].
Indeed, most of these studies used cross-sectional designs where the exposure to nurse staffing levels and the occurrence of AEs were measured at the same point in time [25, 26]. Because the temporal relationship between the exposure and the outcome cannot be ascertained from these studies, it is difficult to determine if the occurrence of an AE can truly be attributed to the antecedent exposure to low nurse staffing levels. In addition, most of these studies were multi-institutional investigations that used large national, state, or provincial administrative databases to determine if hospital-level measures of nurse staffing were related to AEs after adjusting for hospital case mix [15, 24]. While this approach can be useful for benchmarking purposes, it entails averaging staffing and AE data over relatively long periods of time (e.g. 1 year), and across all types of units (e.g., internal medicine, intensive care unit) and patients within a given hospital . As a consequence, it has been difficult to translate the results of these studies into specific recommendations regarding the optimal nurse staffing levels for a given patient at a specific point in time during the course of a hospitalization [13, 15].
To move this field forward, there is a need for longitudinal studies conducted at the patient level of analysis . Such studies are required not only to determine if nurse staffing levels are related to AEs but also to identify the particular staffing patterns that are of greatest risk, and the specific staffing thresholds that minimize the risk of AEs given the complexity of patient requirements. This is especially important given that the health care industry has lagged behind other safety-sensitive industries, such as aviation and long-distance trucking, in defining staffing policies that minimize the risk of AEs [13, 20]. As a first step towards defining such policies, novel methods for capturing variations in nurse staffing levels through time have recently been developed.
Novel methods for measuring patient exposure to nurse staffing levels
We recently used digitized payroll data to examine how variations in nurse staffing levels through time are related to the use of patient sitters, a type of unlicensed assistive health care provider (UAP) whose function is to provide close surveillance to patients at risk of an AE [27, 28]. Using a nested case-control approach, we found that patients exposed to higher volumes of overtime hours and to less experienced nursing staff during the antecedent exposure period had a higher likelihood of sitter use compared to controls (Table 1).
In the proposed study, we will build on this prior work to examine the associations between nurse staffing levels and the risk of AEs. One of the original contributions of this study, and a direct extension of our prior work in this area, will be to model the effect of nurse staffing levels on the risk of AEs as a time-varying exposure. This will allow to determine if and how the risk of AEs changes with both the duration of exposure to suboptimal staffing (e.g., extensive overtime use), but also with the intensity of exposure (e.g., low volume of nursing hours per patient). This will facilitate the identification of the particular nurse staffing patterns that are of greatest risk. An important pre-requisite to identifying such patterns is to have access to accurate measures not only of AE occurrence but also of their timing.
Current methods of AE measurement
In their systematic review of the studies examining the associations between nurse staffing levels and AEs, Kane et al.  found that 62 % of the existing studies relied on discharge diagnostic codes to identify AEs, 29 % on manual chart review, and 9 % on incident reporting systems. Discharge diagnostic codes have low sensitivity and positive predictive values (PPV) for identifying AEs [29, 30]. In addition, these codes are not dated with precision (beyond being marked as present or not on admission) [31, 32], a limitation that prevents the elucidation of the temporal relationship between nurse staffing levels and the occurrence of AEs. Accurate timing of AE occurrence is indeed a critical requirement if new knowledge is to be generated on the etiologic contribution of various nurse staffing policies on the incidence of AEs. Manual chart review is a time-consuming, resource-intensive, and costly process . As a consequence, studies using this approach are often smaller in size and underpowered for adequately examining how changes in nurse staffing levels relate to the occurrence of AEs . Incident reports are known to underestimate the true incidence of AEs by a factor of about 20 [35, 36].
In sum, the limitations in existing methods for measuring AEs have curtailed the ability to pursue important investigational work on the identification of nurse staffing policies that minimize the risk of AEs. To address these limitations, novel methods of AE detection have recently been developed and validated.
Novel methods of AE detection
With the increasing availability of rich clinical narrative documents in an electronic format (e.g., radiology reports, progress notes), researchers have started to use automated methods, such as statistical natural language processing (NLP) techniques, to identify AEs [33, 37]. Statistical NLP techniques use probabilistic approaches to automatically classify a set of documents into one of a discrete set of predefined categories (e.g., positive or negative for pneumonia) . Prior studies have shown that statistical NLP can be highly accurate for identifying AEs.
For instance, we recently conducted a pilot study to validate the accuracy of using statistical NLP for identifying venous thromboembolism (VTE), an AE that includes both deep vein thrombosis (DVT) and pulmonary embolism (PE), from electronic narrative radiology reports . The statistical NLP model predicting DVT achieved a sensitivity of 0.80 (95 %CI 0.76–0.85), a specificity of 0.98 (95 %CI 0.97–0.99), and a PPV of 0.89 (95 %CI 0.85–0.93). As for the statistical NLP model predicting PE, sensitivity was 0.79 (95 %CI 0.73–0.85), specificity 0.99 (95 %CI 0.98–0.99), and PPV 0.84 (95 %CI 0.75–0.92) . Since then, we have received funding from the Canadian Institutes of Health Research to adapt this technique for the detection of other AE indicators, including hospital-acquired pneumonia (HAP), ventilator-associated pneumonia (VAP), and in-hospital falls. Preliminary studies using NLP-based approaches for detecting these AEs have shown good prediction performances, with sensitivities ranging from 83 to 86 %, PPV ranging from 85 to 100 %, and specificities ranging from 90 to 100 % [40, 41].
An important advantage of statistical NLP, in addition to being highly accurate, is that large amounts of EHR data can be scanned with minimal human effort and cost , a major gain compared to using manual chart review. Moreover, because free-text radiology reports are dated, the timing of AE occurrence can be determined with relatively high precision, a net improvement compared to using discharge diagnostic codes. In the proposed study, we will take advantage of these characteristics and build on our prior research work to assess the temporality of the associations between nurse staffing levels and the risk of AEs.
This study will be conducted at the McGill University Health Centre (MUHC), an academic health center located in the Canadian province of Quebec. The MUHC is composed of five adult care hospitals and has more than 800 beds. It receives close to 40,000 inpatient admissions and performs more than 34,000 surgeries yearly. The MUHC has more than 11,500 employees, including 3000 registered nurses (RNs) and nursing assistants (NAs), and 800 patient care assistants (PCAs) .
Study design and population
A dynamic cohort of all adult medical, surgical, and intensive care unit (ICU) patients admitted to the MUHC between January 1, 2010, and December 31, 2015, will be followed during the inpatient and 7-day post-discharge period to assess the occurrence and frequency of four AEs (i.e., VTE, HAP, VAP, and in-hospital falls) in relationship to antecedent nurse staffing levels. A follow-up period of 7 days post-discharge will be observed to allow enough time for patients with an AE “incubating” at the time of discharge (e.g., HAP, VTE) to develop the symptoms of the disease and return to the hospital (Fig. 1) . Because in-hospital falls, by definition, cannot occur after discharge, the follow-up period for this particular AE will stop at hospital discharge . For VAP, only the subset of patients who will be intubated at any point over the course of a hospitalization will be considered at risk of developing the disease. Patients will be enrolled in the cohort if they were (a) admitted on a medical, surgical, or ICU at the MUHC; (b) not admitted for one of the AEs of interest; and (c) not hospitalized in the previous 30 days. Re-hospitalization by the same patients occurring more than 30 days after the end of the follow-up period for a given hospitalization will be eligible for inclusion (Fig. 1) [43, 44].
AEs will be ascertained by applying statistical NLP models to EHR data retrieved at the MUHC. For patients experiencing multiple AEs, or relapses of the same AE over a given hospitalization, only the first AE will be selected. AEs occurring during the follow-up period will be ascertained from EHR data at the time of hospital readmission or emergency department (ED) visit (approximately 9 % of patients return to the ED within 7 days of discharge; 86 % to the same hospital from which they were discharged) . Patients (a) with no AE until the end of the follow-up, (b) readmitted to the hospital during the follow-up period for any reason other than the AEs of interest, or (c) who die before experiencing an AE will be censored at that time (Fig. 1). Patients will be assumed to survive throughout the 7-day post-discharge period.
Nurse staffing levels for a given patient will be determined for each shift of a hospitalization using electronic payroll records. The association between nurse staffing levels and the risk of AEs will be assessed using survival analysis with time-dependent measures of nurse staffing, patient complexity as a confounder and potential modifier of the association, and patient hospitalization as the unit of analysis for AE assessment.
Data required for this study will be extracted from the MUHC clinical data warehouse and will be linked by unit, patient, and hospital admission date and time. Specifically, 12 electronic databases will be queried, including (1) laboratory and microbiology; (2) radiology; (3) vital signs; (4) ICU; (5) pharmacy; (6) admission, discharge, and transfer; (8) discharge abstracts; (9) emergency; (10) operating room; (11) incident reports; and (12) payroll.
Study outcomes: AE indicators
Four AE indicators will be measured: (a) HAP, defined as a pulmonary infection that occurs 48 h or more after hospital admission in patients with no evidence of pneumonia at the time of hospital admission ; (b) VAP, defined as a pulmonary infection occurring more than 48 h after endotracheal intubation and mechanical ventilation in patients with no evidence of pneumonia at the time of intubation  (pneumonia will be considered as a VAP up until 72 h after extubation) ; (c) VTE, defined as a DVT of the lower or upper extremities or a PE occurring 48 h or more after hospital admission ; and (d) in-hospital falls, defined as an unplanned descent to the floor with or without injury occurring during the course of a hospitalization .
These AEs were selected because their occurrence is hypothesized to be related to nurse staffing levels [13–16] and because they can result in significant morbidity, mortality, and cost [49–52]. Moreover, they all have high incidence rates compared to other AEs. Indeed, HAP represents one of the most common nosocomial infections, accounting for 15 % of all hospital-acquired infections and 25 % of all ICU-acquired infections [46, 53, 54]. HAPs are estimated to occur at a rate of between 5 and 20 cases per 1000 hospital admissions . VAP represents the most frequent ICU-acquired infection, occurring in 9 to 28 % of patients receiving mechanical ventilation [54–57]. In the absence of thromboprophylaxis, the incidence of VTE ranges from 10 to 40 % in the medical and general surgical populations to as high as 40 to 60 % in patients who have undergone major orthopedic surgical procedures [58, 59]. Falls are estimated to occur in 1.9 to 3 % of all hospitalizations and in 2 to 27 % of elderly hospitalizations .
AE occurrence will be ascertained by applying statistical NLP models to electronic narrative radiological and incident reports retrieved at the MUHC [39–41]. These models have learned from human-labeled training documents which combinations of words in narrative reports are suggestive of the presence of a given AE. The accuracy of these models is summarized in Table 2. For all patients, the date and time of the narrative report will be considered as the date and time of AE occurrence.
Nurse staffing levels
Primary attributes of nurse staffing
Nurse staffing levels in a given hospital typically vary from one unit to the next as well as within a given unit on a shift-by-shift basis as a function of the fluctuations in patients’ requirements for nursing care, as well as with unpredictable absenteeism . For this reason, patient exposure to nurse staffing levels will be measured on every shift of a hospitalization using the following three time-varying indicators:
Nurse staffing intensity will be defined as the average number of work hours per patient provided by all members of the nursing staff (i.e., RNs, NAs, PCAs, and UAPs) [60, 61]. For each patient, nurse staffing intensity will be measured by dividing the total number of nursing work hours on the unit where the patient was hospitalized on a given shift by the beginning-of-shift patient census on that unit for that shift .
Skill mix will be defined as the proportion of the total nursing work hours that were reported by RNs . For each patient, skill mix will be measured by dividing the total number of RN work hours on the unit where the patient was hospitalized on a given shift by the total number of nursing work hours on that unit for that shift.
Overtime use will be defined as time worked in excess of the standard hours and paid at least one and one-half (1.5) times the regular rate of pay [19, 21]. For each patient, overtime use will be measured by dividing the total number of overtime hours reported by all members of the nursing staff on the unit where the patient was hospitalized on a given shift by the total number of nursing work hours reported on that unit for that shift.
Secondary attributes of nurse staffing
Other nurse staffing attributes can potentially be associated with AE occurrence, including nursing staff levels of experience and education [60, 62]. Patient exposure to these staffing attributes will be measured on every shift of a given hospitalization using the following time-varying indicators:
Experience is expected to provide nurses with exposure to different patient conditions and clinical scenarios which contribute to the development of their knowledge, skills, and critical thinking [63, 64]. More experienced nurses should thus have a broader set of interventions to prevent AEs from occurring. Conversely, when the overall level of experience in a team of nurses is reduced, more AEs may occur . For each patient, nurse experience will be measured as the mean number of years of experience held by all members of the nursing staff who reported work hours on the unit where the patient was hospitalized on a given shift .
Education (at the baccalaureate degree or higher) is thought to be associated with better critical thinking and clinical judgment skills among RNs, and ecological studies have suggested that hospitals with more highly educated RNs have lower rates of AEs and mortality [34, 64]. For each patient, education will be measured as the proportion of RNs with a baccalaureate degree or higher among all RNs who reported work hours on the unit where the patient was hospitalized on a given shift .
Duration and intensity of exposure to nurse staffing
Time-varying exposure to nurse staffing levels will be measured for every shift of a given hospitalization. Four alternative measures will be used to represent nurse staffing levels: (a) current exposure, defined as nurse staffing levels on the current shift; (b) mean recent exposure, defined as mean staffing levels over the past 4 days; (c) mean exposure since hospital admission, defined as mean staffing levels since hospital admission, and (d) weighted cumulative exposure (WCE) . WCE is a novel analytical method, developed and validated in our prior research work [65, 66], which adds the dimension of timing of past exposures, along with information about duration and intensity, into a single metric. This metric is created by multiplying each exposure in the past by a weight that represents the relative importance of that exposure on the current risk of an AE, and summing the weighted past exposures up to the current time . The weighting function that assigns weights to past exposures is estimated directly from the data . This novel approach should provide additional insights about the mechanism by which time-dependent patterns of nurse staffing influence the risk of AEs. Because no data on nurse staffing levels during the post-discharge follow-up period will be available, a sensitivity analysis will be conducted to determine which, among several alternatives, provides the best estimates for each measure of staffing (e.g., assigning average expected nurse staffing values based on the patient’s destination upon discharge, carrying over the last nurse staffing values at discharge, setting all nurse staffing values to 0).
The aforementioned measures of nurse staffing exposures were selected based on the following hypotheses: For HAP and VAP, suboptimal staffing levels have been shown to result in lapses in basic infection control measures (e.g., hand washing) which may facilitate the transmission of pathogens [67, 68]. Given that the typical incubation period for HAP and VAP is at least 48 h , recent exposure to suboptimal nurse staffing levels (e.g., 4 days before the event) may be the most relevant for HAP and VAP. Prolonged immobility has been identified as an important risk factor for VTE , and studies have shown that certain thromboprophylaxis measures, such as patient mobilization, are performed less frequently when staffing levels are suboptimal . As such, sustained exposure to suboptimal staffing during a hospitalization may result in a greater risk of VTEs. Lastly, suboptimal staffing levels may also reduce nurses’ capacity to provide adequate surveillance to patients at risk of falling [64, 71]. Because only a small fraction of time is needed for a fall to occur, it is likely that reduced surveillance capacity resulting from suboptimal staffing levels on a given shift would be related to a greater risk of in-hospital falls .
Potential confounders common to all four AE indicators
Several patient and organizational characteristics may increase the likelihood of HAP, VAP, VTE, or in-hospital fall. These conditions will be measured as potential confounders.
Patient demographic characteristics and comorbidities are important modifiers of a patient’s health condition and have been associated with an increased risk of AEs [50, 57, 58]. Patient age at hospital admission and sex will be measured as fixed-in-time covariates. Comorbidities will be measured using the Charlson Comorbidity Index, a weighted index of 17 comorbidities that are associated with an increased risk of mortality [72, 73]. Comorbidities will be measured at the time of hospital admission as a fixed-in-time covariate using discharge diagnostic codes from all prior hospitalizations since 2004 (i.e., the maximum time frame for which complete data from previous hospitalizations are available).
Severity of illness may influence the risk of AEs and nurse staffing levels . To adjust for this possible source of confounding, severity of illness will be measured as a time-varying covariate using the laboratory-based acute physiology score (LAPS) [74, 75]. The LAPS is a scoring system that integrates information from 14 laboratory tests into a single continuous variable. LAPS will be measured at the beginning of every shift using the most recent laboratory results. LAPS can range from 0 (corresponding to lab results within normal physiologic ranges) to a theoretical maximum of 256. A score of 0 points will be assigned to any unperformed laboratory test [74, 75].
Admission diagnosis may increase the likelihood of AEs and will be measured at the time of hospital admission as a fixed-in-time covariate [50, 57, 58]. Admission diagnoses will be classified into 44 categories using a procedure based on diagnostic codes .
Patient turnover rates, defined as the number of admissions, discharges, and transfers on a given shift , may increase the demands on the nursing staff and result in errors or lapses in care processes that may increase the risk of AEs [76, 77]. Patient turnover rates will be measured as a time-varying covariate representing the total number of admissions, transfers, and discharges on the unit where the patient was hospitalized on a given shift divided by the start-of-shift patient census on that unit for that shift.
The type of nursing unit where the patient is located may influence both the risk of AE and nurse staffing levels . To adjust for possible confounding from measures of staffing and hospitalization on certain units (e.g., ICU vs. medicine), and for other unmeasured unit-level factors that may increase the risk of AEs (e.g., work environment characteristics) , a time-varying covariate representing the nursing unit where the patient is currently hospitalized will be created and updated every shift.
Other confounders include three additional fixed-in-time covariates which will be measured at the time of hospital admission: the hospital at which the patient was admitted (to test for differences between the five MUHC hospitals), the year, and the month in which the admission took place (to test for temporal and seasonal trends).
Potential confounders specific to each AE indicator
In addition to confounders common to all four AEs, AE-specific confounders will also be considered:
VAP/HAP: (a) Length of mechanical ventilation will be measured at the beginning of every shift as the cumulative number of shifts for which the patient is under mechanical ventilation (cumulative use) . (b) Recent surgeries will be measured every shift as a time-varying covariate indicating whether or not the patient had a surgical procedure . In addition, the type of the latest surgical procedure performed (i.e., cardiothoracic, orthopedic, abdominal, other, none) will also be captured by a categorical covariate updated every shift [53, 54]. (c) Drugs associated with an increased likelihood of pneumonia will be grouped in two therapeutic classes based on the American Hospital Formulary Service (AHFS) classification : (i) corticosteroids and (ii) gastro-protective agents [53, 54]. The usage of these drugs will be assessed as two time-varying covariates representing current use (yes or no) and will be updated every shift.
In-hospital fall: (a) Gait and mobility impairments will be measured as a fixed-in-time covariate at the time of hospital admission . Validated coding rules will be used to identify discharge diagnostic codes from all prior hospitalizations since 2004 that are suggestive of the presence of gait and mobility impairments . (b) Cognitive impairments will be measured as a fixed-in-time covariate at the time of hospital admission . Coding rules from the Dementia, Delirium and Other Cognitive Impairments subclassification of the Mental Health and Substance Abuse Clinical Classification will be used to identify discharge diagnostic codes from all prior hospitalizations since 2004 that are suggestive of the presence of cognitive impairments . (c) History of previous falls will be measured at the time of hospital admission as a fixed-in-time dichotomous covariate (yes or no) . Patients with a history of previous falls will be identified by applying our statistical NLP model for inpatient falls to radiological and incident reports from prior hospitalizations since 2004 . (d) Psychoactive drugs associated with an increased risk of falling will be grouped into six therapeutic classes based on the AHFS classification : (i) anticonvulsants; (ii) antidepressants; (iii) antipsychotics; (iv) anxiolytics, sedatives, and hypnotics; (v) benzodiazepines; and (vi) opioids. Psychoactive drug use will be assessed as six time-varying covariates representing current use (yes or no) and will be updated every shift .
VTE: (a) History of VTE will be measured at the time of hospital admission as a fixed-in-time covariate indicating whether or not the patient had a VTE in the past . History of VTE will be determined by applying our statistical NLP models to narrative reports of radiological examinations performed since 2004 . (b) Anticoagulant drug use will be assessed as a time-varying covariate updated every day and representing current use (yes or no). (c) Recent surgeries will be measured using the approach described for VAP. (d) Mobility impairments will be measured using the approach described for in-hospital fall .
Descriptive statistics will be used to summarize patient and nurse staffing characteristics. To examine the associations between nurse staffing levels and the risk of HAP, VAP, VTE, and in-hospital falls, four separate AE-specific Cox proportional hazards regression models will be used (i.e., one model for each AE indicator). Time 0 will correspond to the date of hospital admission, and the time to event will be defined as the time to the first AE specific to each of the four models. Patients who had no AE by the end of the follow-up period for a hospitalization episode or who die before experiencing an AE will be censored at that time. All models will be adjusted for patient demographic characteristics, comorbidities, and AE-specific risk factors. For continuous covariates, we will use the flexible fractional polynomials approach to test for nonlinear effects and, if necessary, account for such nonlinearities .
Within each regression model, exposure to primary nurse staffing attributes (i.e., staffing intensity, skill mix, and overtime) and secondary nurse staffing attributes (i.e., education and experience) will be defined in four alternative ways, each using a different time-varying exposure or time window: (a) current exposure, (b) mean recent exposure over the past 4 days, (c) mean exposure since hospital admission, and (d) WCE . For WCE, the weights assigned to past exposures will be estimated using a flexible cubic spline function to avoid any a priori assumptions regarding the shape of the weight function [65, 82]. Each model will be adjusted for both fixed (e.g., age, sex) and time-varying covariates (e.g., severity of illness, medication use) relevant to the AE indicator being considered. The fit of the alternative exposure models will be compared using the Akaike information criterion (AIC) , and the best-fitting (minimum AIC) model for a given AE indicator will be selected for subsequent analyses . Hazard ratios (HRs) and 95 % confidence intervals (95 %CIs) will be estimated for the best fitting models. To assess if the association between nurse staffing levels and AEs is modified by the complexity of patient requirements, the significance of interaction terms between nurse staffing attributes and selected covariates (e.g., patient type [medical vs. surgical vs. ICU], turnover rates) will be tested. Stratified models will be estimated for each modifying effect.
To assess for the presence of optimal nurse staffing thresholds, for each staffing-by-complexity stratum, a flexible extension of the Cox model that uses nonlinear spline functions to estimate how the hazard varies with increasing value of the predictor (here, nurse staffing levels) will be fitted . The null hypothesis that the effect of nurse staffing levels is linear will be tested with a nonparametric likelihood ratio test (LRT), comparing the partial deviance of the best-fitting linear models with that of the nonlinear spline model . P < .05 for the LRT will indicate that the nonlinear model provides significantly better prediction of AEs than the linear model, in which case spline functions may indicate the threshold effect of nurse staffing . The HRs and the 95 %CI for nonlinear HRs derived from splines will be estimated using bootstrap resampling .
The proportional hazards (PHs) assumption will be verified with a nonparametric LRT comparing partial deviance of the conventional PH to a flexible non-PH model . In the case of significant violation of the PH hypothesis, the flexible model will estimate how the covariate effect (adjusted HR) changes during the follow-up . To account for possible nonrandom (informative) censoring on death, inverse probability censoring weights will be used in sensitivity analyses . To account for data clustering resulting from repeated hospitalizations by the same patient through time, the marginal approach with robust standard error estimators will be used . Cox regression will be implemented with SAS, version 9.2, and flexible spline-based models , and the WCE model  will be implemented with customized programs in R.
Sample size requirements
We estimate that the Cox regression models for the current exposure to each of the three main nurse staffing attributes will have excellent 90 % power (at two-tailed alpha = 0.05) to detect reductions in the risk of HAP or VTE of (a) 3 to 5 % for every increase of 10 min per patient per shift in nurse staffing intensity and (b) 4 to 6 % for every 5 % increase per shift in skill mix. In addition, this model will have 90 % power to detect a risk increase of 5 to 7 % for every 10 additional minutes per patient per shift of overtime (Additional file 1: Tables A1 and A2). Given that in-hospital falls and VAP have higher incidence rates than HAP and VTE, the detectable HRs for these AEs are smaller (Additional file 1: Tables A3 and A4).
Current study status
At the present time, we have received both research ethics and institutional approvals to begin the study and extract the required data. Data extraction is expected to begin shortly.
Relevance and impact
This study will be the first to model the effect of nurse staffing strategies on the risk of AEs as time-varying exposures. This will allow determining if and how the risk of AEs changes with both the duration of exposure to suboptimal staffing (e.g., extensive overtime use), but also with the intensity of exposure (e.g., low volume of nursing hours per patient). This will facilitate the identification of the particular nurse staffing patterns that are of greatest risk. This study will likely generate evidence-based information that will assist managers in making the most effective use of scarce nursing resources and in identifying staffing patterns that minimize the risk of AEs.
Akaike information criterion
Deep vein thrombosis
Electronic health records
Intensive care unit
Laboratory acute physiology score
Likelihood ratio test
McGill University Health Centre
Natural language processing
Patient care assistants
Positive predictive values
Registered practical nurse
Unlicensed assistive health care providers
Weighted cumulative exposure
Kohn LT, Corrigan J, Donaldson MS. To err is human. Building a safer health system. Washington, D.C.: Institute of Medicine, National Academy Press; 2000.
Wilson RM, Runciman WB, Gibberd RW, Harrison BT, Newby L, Hamilton JD. The quality in Australian health care study. Med J Aust. 1995;163:458–71.
Thomas EJ, Studdert DM, Burstin HR, Orav EJ, Zeena T, Williams EJ, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care. 2000;38:261–71.
Thomas EJ, Studdert DM, Newhouse JP, Zbar BI, Howard KM, Williams EJ, et al. Costs of medical injuries in Utah and Colorado. Inquiry. 1999;36:255–64.
Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med. 1991;324:370–6.
Leape LL, Brennan TA, Laird N, Lawthers AG, Localio AR, Barnes BA, et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med. 1991;324:377–84.
Baker GR, Norton PG, Flintoft V, Blais R, Brown A, Cox J, et al. The Canadian adverse events study: the incidence of adverse events among hospital patients in Canada. CMAJ. 2004;170:1678–86.
Davis P, Lay-Yee R, Briant R, Ali W, Scott A, Schug S. Adverse events in New Zealand public hospitals I: occurrence and impact. N Z Med J. 2002;115:U271.
Davis P, Lay-Yee R, Briant R, Ali W, Scott A, Schug S. Adverse events in New Zealand public hospitals II: preventability and clinical context. N Z Med J. 2003;116:U624.
Committee on the work environment for nurses and patient safety. Keeping patients safe: transforming the work environment of nurses. Washington, D.C: The National Academies Press; 2004.
Kane NM, Siegrist RB. Understanding rising hospital inpatient costs: key component of cost and the impact of poor quality. 2012. https://www.heartland.org/policy-documents/understanding-rising-hospital-inpatient-costs-key-components-cost-and-impact-poor-q. Accessed Jan 21, 2014.
Welton JM, Fischer MH, DeGrace S, Zone-Smith L. Hospital nursing costs, billing, and reimbursement. Nurs Econ. 2006;24:239–45. 262, 227.
Brennan CW, Daly BJ, Jones KR. State of the science: the relationship between nurse staffing and patient outcomes. West J Nurs Res. 2013;35:760–94.
Shekelle PG. Nurse-patient ratios as a patient safety strategy: a systematic review. Ann Intern Med. 2013;158:404–9.
Kane RL, Shamliyan T, Mueller C, Duval S, Wilt TJ. Nurse staffing and quality of patient care. Evid Rep Technol Assess (Full Rep). 2007;151:1–115.
Bae SH, Fabry D. Assessing the relationships between nurse work hours/overtime and nurse and patient outcomes: systematic literature review. Nurs Outlook. 2014;62:138–56.
Buerhaus PI. The future of the nursing workforce in the United States: data, trends and implications. Boston: Johns and Bartlett; 2008.
Bae SH, Brewer CS, Kovner CT. State mandatory overtime regulations and newly licensed nurses’ mandatory and voluntary overtime and total work hours. Nurs Outlook. 2012;60:60–71.
Drebit S, Ngan K, Hay M, Alamgir H. Trends and costs of overtime among nurses in Canada. Health Policy. 2010;96:28–35.
Trinkoff AM, Johantgen M, Storr CL, Gurses AP, Liang Y, Han K. Nurses’ work schedule characteristics, nurse staffing, and patient mortality. Nurs Res. 2011;60:1–8.
Berney B, Needleman J, Kovner C. Factors influencing the use of registered nurse overtime in hospitals, 1995–2000. J Nurs Scholarsh. 2005;37:165–72.
Jacobsen C, Holson D, Farley J, Charles J, Suel P. Surviving the perfect storm: staff perceptions of mandatory overtime. JONAS Healthc Law Ethics Regul. 2002;4:57–66.
Trinkoff A, Geiger-Brown J, Brady B, Lipscomb J, Muntaner C. How long and how much are nurses now working? Am J Nurs. 2006;106:60–71.
Needleman J, Buerhaus P, Pankratz VS, Leibson CL, Stevens SR, Harris M. Nurse staffing and inpatient hospital mortality. N Engl J Med. 2011;364:1037–45.
Stone PW, Pogorzelska M, Kunches L, Hirschhorn LR. Hospital staffing and health care-associated infections: a systematic review of the literature. Clin Infect Dis. 2008;47:937–44.
Unruh L. Nurse staffing and patient, nurse, and financial outcomes. Am J Nurs. 2008;108:62–71.
Rochefort CM, Ward L, Ritchie JA, Girard N, Tamblyn RM. Patient and nurse staffing characteristics associated with high sitter use costs. J Adv Nurs. 2012;68:1758–67.
Rochefort CM, Ward L, Ritchie JA, Girard N, Tamblyn RM. Registered nurses’ job demands in relation to sitter use: nested case-control study. Nurs Res. 2011;60:221–30.
Kaafarani HM, Borzecki AM, Itani KM, Loveland S, Mull HJ, Hickson K, et al. Validity of selected patient safety indicators: opportunities and concerns. J Am Coll Surg. 2011;212:924–34.
Goto M, Ohl ME, Schweizer ML, Perencevich EN. Accuracy of administrative code data for the surveillance of healthcare-associated infections: a systematic review and meta-analysis. Clin Infect Dis. 2014;58:688–96.
Houchens RL, Elixhauser A, Romano PS. How often are potential patient safety events present on admission? Jt Comm J Qual Patient Saf. 2008;34:154–63.
Bahl V, Thompson MA, Kau TY, Hu HM, Campbell Jr DA. Do the AHRQ patient safety indicators flag conditions that are present at the time of hospital admission? Med Care. 2008;46:516–22.
Govindan M, Van Citters AD, Nelson EC, Kelly-Cummings J, Suresh G. Automated detection of harm in healthcare with information technology: a systematic review. Qual Saf Health Care. 2010;19:e11.
Blegen MA. Patient safety in hospital acute care units. Annu Rev Nurs Res. 2006;24:103–25.
Bates DW, Evans RS, Murff H, Stetson PD, Pizziferri L, Hripcsak G. Detecting adverse events using information technology. J Am Med Inform Assoc. 2003;10:115–28.
Classen DC, Resar R, Griffin F, Federico F, Frankel T, Kimmel N, et al. ‘Global trigger tool’ shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood). 2011;30:581–9.
Nadkarni PM, Ohno-Machado L, Chapman WW. Natural language processing: an introduction. J Am Med Inform Assoc. 2011;18:544–51.
Hastie T, Tibshirani R, Friedman J. The elements of statistical learning. Data mining, inference, and prediction. 2nd ed. New York: Springer; 2009.
Rochefort CM, Verma AD, Eguale T, Lee TC, Buckeridge DL. A novel method of adverse event detection can accurately identify venous thromboembolisms (VTEs) from narrative electronic health record data. J Am Med Inform Assoc. 2015;22:155–65.
Bejan CA, Xia F, Vanderwende L, Wurfel MM, Yetisgen-Yildiz M. Pneumonia identification using statistical feature selection. J Am Med Inform Assoc. 2012;19:817–23.
Toyabe S. Detecting inpatient falls by using natural language processing of electronic medical records. BMC Health Serv Res. 2012;12:448–56.
McGill University Health Centre (MUHC). MUHC at a glance. 2014. https://muhc.ca/homepage/page/muhc-glance. Accessed Nov 19, 2014.
Ashton CM, Wray NP. A conceptual framework for the study of early readmission as an indicator of quality of care. Soc Sci Med. 1996;43:1533–41.
Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee-for-service program. N Engl J Med. 2009;360:1418–28.
Canadian Institute for Health Information (CIHI). All-cause readmissions to acute care and return to the emergency department. Ottawa: CIHI; 2012.
American Thoracic Society and Infection Disease Society of America. Guidelines for the management of adults with hospital-acquired, ventilator-associated, and healthcare-associated pneumonia. Am J Respir Crit Care Med. 2005;171:388–416.
Cimiotti JP. Staffing level: a determinant of late-onset ventilator-associated pneumonia. Crit Care. 2007;11:154.
Maynard G, Stein J. Preventing hospital-acquired venous thromboembolism: a guide for effective quality improvement. 2008. http://www.ahrq.gov/professionals/quality-patient-safety/patient-safety-resources/resources/vtguide/vtguide.pdf. Accessed Apr 14, 2013.
Currie L. Fall and injury prevention. In: Hughes RG, editor. Patient safety and quality: an evidence-based handbook for nurses. Rockville: Agency for Healthcare Research and Quality; 2008.
Anderson DC, Postler TS, Dam TT. Epidemiology of hospital system patient falls: a retrospective analysis. Am J Med Qual. 2015. doi:10.1177/1062860615581199.
Muscedere JG, Day A, Heyland DK. Mortality, attributable mortality, and clinical events as end points for clinical trials of ventilator-associated pneumonia and hospital-acquired pneumonia. Clin Infect Dis. 2010;51 Suppl 1:S120–5.
Agrafiotis M, Siempos II, Ntaidou TK, Falagas ME. Attributable mortality of ventilator-associated pneumonia: a meta-analysis. Int J Tuberc Lung Dis. 2011;15:1154–63.
Kieninger AN, Lipsett PA. Hospital-acquired pneumonia: pathophysiology, diagnosis, and treatment. Surg Clin North Am. 2009;89:439–61.
Joseph NM, Sistla S, Dutta TK, Badhe AS, Parija SC. Ventilator-associated pneumonia: a review. Eur J Intern Med. 2010;21:360–8.
Vincent JL, Rello J, Marshall J, Silva E, Anzueto A, Martin CD, et al. International study of the prevalence and outcomes of infection in intensive care units. JAMA. 2009;302:2323–9.
Rosenthal VD, Bijie H, Maki DG, Mehta Y, Apisarnthanarak A, Medeiros EA, et al. International Nosocomial Infection Control Consortium (INICC) report, data summary of 36 countries, for 2004–2009. Am J Infect Control. 2012;40:396–407.
Barbier F, Andremont A, Wolff M, Bouadma L. Hospital-acquired pneumonia and ventilator-associated pneumonia: recent advances in epidemiology and management. Curr Opin Pulm Med. 2013;19:216–28.
Geerts WH, Bergqvist D, Pineo GF, Heit JA, Samama CM, Lassen MR, et al. Prevention of venous thromboembolism: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition). Chest. 2008;133 Suppl 1:381S–453.
Fanikos J, Rao A, Seger AC, Piazza G, Catapane E, Chen X, et al. Venous thromboembolism prophylaxis for medical service-mostly cancer-patients at hospital discharge. Am J Med. 2011;124:1143–50.
Stalpers D, de Brouwer BJ, Kaljouw MJ, Schuurmans MJ. Associations between characteristics of the nurse work environment and five nurse-sensitive patient outcomes in hospitals: a systematic review of literature. Int J Nurs Stud. 2015;52:817–35.
Van den Heede K, Clarke SP, Sermeus W, Vleugels A, Aiken LH. International experts’ perspectives on the state of the nurse staffing and patient outcomes literature. J Nurs Scholarsh. 2007;39:290–7.
West E, Mays N, Rafferty AM, Rowan K, Sanderson C. Nursing resources and patient outcomes in intensive care: a systematic review of the literature. Int J Nurs Stud. 2009;46:993–1011.
Benner P. From novice to expert: excellence and power in clinical nursing practice. Upper Saddle River: Prentice Hall; 2001.
Kutney-Lee A, Lake ET, Aiken LH. Development of the hospital nurse surveillance capacity profile. Res Nurs Health. 2009;32:217–28.
Sylvestre MP, Abrahamowicz M. Flexible modeling of the cumulative effects of time-dependent exposures on the hazard. Stat Med. 2009;28:3437–53.
Abrahamowicz M, Beauchamp ME, Sylvestre MP. Comparison of alternative models for linking drug exposure with adverse effects. Stat Med. 2012;31:1014–30.
Hugonnet S, Chevrolet JC, Pittet D. The effect of workload on infection risk in critically ill patients. Crit Care Med. 2007;35:76–81.
Blot SI, Serra ML, Koulenti D, Lisboa T, Deja M, Myrianthefs P, et al. Patient to nurse ratio and risk of ventilator-associated pneumonia in critically ill patients. Am J Crit Care. 2011;20:e1–9.
Goldhaber SZ. Risk factors for venous thromboembolism. J Am Coll Cardiol. 2010;56:1–7.
Papastavrou E, Andreou P, Efstathiou G. Rationing of nursing care and nurse-patient outcomes: a systematic review of quantitative studies. Int J Health Plann Manage. 2013;29:3–25.
Lake ET, Cheung RB. Are patient falls and pressure ulcers sensitive to nurse staffing? West J Nurs Res. 2006;28:654–77.
Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40:373–83.
Sundararajan V, Henderson T, Perry C, Muggivan A, Quan H, Ghali WA. New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality. J Clin Epidemiol. 2004;57:1288–94.
Wong J, Taljaard M, Forster AJ, Escobar GJ, van Walraven C. Derivation and validation of a model to predict daily risk of death in hospital. Med Care. 2011;49:734–43.
Escobar GJ, Greene JD, Scheirer P, Gardner MN, Draper D, Kipnis P. Risk-adjusting hospital inpatient mortality using automated inpatient, outpatient, and laboratory databases. Med Care. 2008;46:232–9.
Unruh LY, Fottler MD. Patient turnover and nursing staff adequacy. Health Serv Res. 2006;41:599–612.
Evans WN, Kim B. Patient outcomes when hospitals experience a surge in admissions. J Health Econ. 2006;25:365–88.
American Society of Health-System Pharmacists (AHFS). American Hospital Formulary Service Drug Information. Bethesda: AHFS; 2010.
Wilchesky M, Tamblyn RM, Huang A. Validation of diagnostic codes within medical services claims. J Clin Epidemiol. 2004;57:131–41.
Owens P, Myers M, Elixauser A, Branch C. Care of adults with mental health and substance abuse disorders in U.S. community hospitals (Publication No. 07-0008). 2007. http://www.ahrq.gov/research/data/hcup/index.html. Accessed Feb 1, 2015.
Royston P, Sauerbrei W. Multivariable model building: a pragmatic approach to regression analysis based on fractional polynomials for modelling continuous variables. West Sussex: John Wiley & Sons; 2008.
Abrahamowicz M, MacKenzie TA. Joint estimation of time-dependent and non-linear effects of continuous covariates on survival. Stat Med. 2007;26:392–408.
Akaike H. A new look at statistical model identification. IEEE Trans Automatic Control. 1974;19:716–23.
Benedetti A, Abrahamowicz M, Leffondré K, Goldberg MS, Tamblyn R. Using generalized additive models to detect and estimate threshold associations. Int J Biostat. 2009;5:1–26.
Efron B, Tibshirani RJ. An introduction to bootstrap. New York: Chapman & Hall; 1993.
Robins JM, Finkelstein DM. Correcting for noncompliance and dependent censoring in an AIDS Clinical Trial with inverse probability of censoring weighted (IPCW) log-rank tests. Biometrics. 2000;56:779–88.
Wei LJ, Lin DY, Weissfeld L. Regression analysis of multivariate incomplete failure time data by modeling marginal distributions. J Am Stat Assoc. 1989;84:1065–73.
Funding for this study was provided by the Canadian Institutes of Health Research (CIHR) through research fund FRN 130499. Dr. Rochefort holds a career award from the Fonds de recherche en santé du Québec (FRQ-S). These funding sources were not involved in the design of the study; in the collection, analysis, and interpretation of data; in the writing of the manuscript; or in the decision to submit the manuscript for publication.
The authors declare that they have no competing interests.
All listed authors (1) have made substantial contributions to conception and design, or acquisition of data, or analysis and interpretation of data; (2) have been involved in drafting the manuscript or revising it critically for important intellectual content; (3) have given final approval of the version to be published; and (4) agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated or resolved.
About this article
Cite this article
Rochefort, C.M., Buckeridge, D.L. & Abrahamowicz, M. Improving patient safety by optimizing the use of nursing human resources. Implementation Sci 10, 89 (2015). https://doi.org/10.1186/s13012-015-0278-1
- Nurse staffing
- Adverse events
- Electronic health record
- Acute care hospital
- Survival analysis
- Longitudinal study