Design
This mixed methods study will use quantitative (objectives 1, 3, and 4) and qualitative (objective 2) methodologies to address the research objectives. We will implement a sequential exploratory design by using the qualitative findings (from interviews) to inform the quantitative components of the study (survey and regression analysis) [23].
Theoretical framework
The Promoting Action on Research Implementation in Health Services (PARiHS) [24] framework will guide this study. The PARiHS framework contends that evidence, context, and facilitation are the key influencing factors for successful implementation of a new innovation for practice change. The most successful implementation occurs when evidence is scientifically robust and matches professional consensus and patient preferences (high evidence), when the context is receptive for change with sympathetic cultures, strong leadership, and appropriate monitoring and feedback systems (high context), and when there is appropriate facilitation of change, with input from skilled external and internal facilitators (high facilitation) [24]. Based on this framework, we hypothesize that successful adoption of the MND (a new innovation) to facilitate practice change will be dependent upon the nature and the clarity of the evidence on which it is based, the quality of the context (environment) in which it is being implemented, and the type of facilitation needed to ensure successful adoption. Concepts from the PARiHS framework will inform development of our semi-structured interview guide and the provincial survey for this study.
A description of the methods for each of the objectives is presented below.
Objective 1—to evaluate the population effect of implementing the MND across Ontario
Methods
An ITS analysis of six KPIs (Table 1) will be used. This quasi-experimental design can be used to determine the effect of a complex intervention introduced at a specific point in time [25]. It is superior to many other quasi-experimental and observational designs, such as before and after designs, in that it avoids threats to internal validity such as history and maturation [26, 27]. By using outcomes assessed at multiple time points, it allows the estimation of an underlying secular trend prior to the intervention. The estimated intercept and trend before and after the intervention can then be compared to determine both the immediate and long-term effects of the intervention. While ITS designs are useful for determining whether an intervention has had an effect on the outcome while accounting for any underlying secular trend, the possibility of confounding by temporally concurrent interventions still poses an important threat to the internal validity of this design [28]. We will use two approaches to assess “possible rival explanations” for any apparent effect of the MND [29]. First, we will assess two non-equivalent outcomes measured in the same population over the same period of time: CS rates in nulliparous women with induced labor and the proportion of low-risk women at term gestation having electronic fetal monitoring in labor. Since these outcomes should not be affected by the MND, a finding of no change in these variables will strengthen our ability to attribute any change in the KPIs to the MND. Second, we will use data from two Canadian provinces (Nova Scotia and British Columbia) as a non-equivalent control group. A suitable control group must have similar baseline characteristics and pre-implementation temporal trends, except for the MND intervention [30].
Analysis
The data to support the primary ITS analysis will come from the BIS. The MND implementation occurred from November 2012 to March 2013, allowing time for all hospitals to gain experience with data entry and use of the dashboard. MND data for the 3-year pre-implementation period (November 2009–October 2012) and the 2-year post-implementation period (April 2013–March 2015) will be available after April 2016. Thus, each site will contribute 3 years of data to the pre-implementation phase and 2 years after implementation. We will exclude data from the 5–6 months between the pre- and post-implementation phases from the analysis. For our primary ITS analyses, the six KPIs will be expressed as monthly proportions after pooling across all hospitals. Pooling is necessary to allow inclusion of low birth volume hospitals and to accommodate all six KPIs, including two that assess performance of uncommon outcomes. Time series plots will be used to visually inspect the immediate and long-term effect of the intervention and the presence of trends, cyclical patterns, and outliers. A segmented regression analysis will be completed using separate intercepts and slopes for the time periods before and after implementation of the MND [26, 31]. The presence of autocorrelation will be assessed using Durbin-Watson tests, as well as visual inspection of residual plots. If autocorrelation is present, an autocorrelation parameter will be included in the model. The results of the segmented regression analysis of each KPI will be reported as level and trend changes after the intervention, with 95 % confidence intervals (CI). All analyses will be conducted using SAS v. 9.4.
Sample size considerations
The ITS analysis will include monthly data aggregated from 94 hospitals over 5 years (60 time points total). Recommendations are to include between 40 and 50 observations for robust statistical analysis of ITS designs [32]; moreover, to avoid over-fitting of segmented regression models, at least 10 observations for each regression coefficient is required [33]. With 4 regression coefficients in our planned analyses, 60 time points is adequate. Furthermore, to help ensure stability of the monthly proportions, it is recommended that measurements be based on at least 100 observations at each time point. After pooling across all hospitals (approximately 12,000 births per month), the minimum monthly denominator will range from approximately 200 (repeat Caesarian section) to 7800 (group B Streptococcus screening). Thus, we expect to have sufficient sample sizes to ensure stable estimates.
Objective 2—to explore factors that potentially explain differences among hospitals’ use of the MND
Methods
Hospital selection
A criterion-based approach [34] will be used to identify a purposeful sample of up to 20 hospitals reflecting different levels of care (levels 1, 2, and 3) [35], annual birth volumes (<1000, 1001–2500, >2500), geographic locations, and degree of engagement with the MND (i.e., none, partial, or full). This approach to hospital selection will provide a diverse sample of hospitals from which to recruit participants for the interviews and ensure a rich source of data to inform our understanding of factors potentially associated with differential effectiveness of the MND.
Sampling procedures
In qualitative research, there are no standardized rules for sample sizes: while 6–8 participants often suffice for a homogeneous sample, 20–30 may be needed when looking for disconfirming evidence or trying to achieve maximum variation [36, 37]. We will use the concept of data saturation to determine when no additional interviews are required (i.e., no new information is emerging) [38, 39]. Directors or managers of the maternal-newborn units from up to 20 of the purposefully selected hospitals (key informants) will be invited to participate in an interview. Recruitment may be augmented through snowball sampling. We do not anticipate problems recruiting participants because of our extensive connections with these centers. Following ethical approval, we will identify individuals within each organization willing to participate, who can provide information regarding use of the MND with regard to its utility as an A&F tool for practice change. Participants will be recruited based on their familiarity with the BIS, their ability to describe practice from the perspective of the organization, their knowledge of the KPIs in the MND, and quality improvement within their organization. Participant consent will be obtained prior to scheduling the interviews.
Semi-structured interview guide
Key informant interviews will be completed using a semi-structured interview guide. Interview questions will be based on the concepts in the PARiHS framework and the Organizational Readiness for Knowledge Translation (OR4KT) Tool [40, 41], a comprehensive evidence-based instrument that was developed based on a systematic review of conceptual models/frameworks of organizational readiness for change in health care. The OR4KT has been validated in primary care settings and contains questions covering six dimensions of organizational readiness (organizational climate for change, organizational contextual factors, change content, leadership, organizational support, and motivation). The interview questions will be designed to probe participants’ perspectives about the attributes of the MND, hospital contextual factors, and facilitation/support issues that have influenced their hospital’s use of the MND. Interviews, which may last up to 1 h, will be conducted in person or by telephone and will be audiotaped, with consent. The interview guide will be pilot tested and questions revised if necessary.
Data entry and processing
Interviewing, transcription, and analysis will proceed concurrently to monitor the progress of the interviews, permit follow-up of issues that may emerge from the data, and allow probing of emerging themes in subsequent interviews [42, 43]. Digital recordings will be transcribed verbatim and verified by the interviewer prior to analysis. Data will be imported into NVivo 11™ (qualitative data management software) to facilitate management of data analysis [44].
Analysis
Within-case analysis
Initially, data from each of the cases (hospitals) will be analyzed independently. Analysis will begin with repeated reading of transcripts and field notes, summarizing key information by writing a description of each transcript [34, 45], followed by qualitative content analysis using coding, categorizing, and thematic description [43, 46, 47]. Codes will be sorted into (1) a priori categories based on PARiHS concepts (MND attributes, OR4KT dimensions [40, 41], and facilitation factors) and (2) categories that emerge during the analysis [34]. The final step in this within-case analysis will be the development of narrative descriptions of themes derived from each case. Member checking will be undertaken involving a small subgroup of participants to ensure that the themes identified through the coding process resonate with the participants’ experiences and to identify any gaps in the analysis or issues requiring further consideration [34, 48].
Cross-case analysis
Subsequently, thematic similarities and differences between cases (based on hospital selection criteria) will provide understanding of the key factors influencing the use of the MND in different practice settings. Investigators will regularly discuss the coding template, categories, and emerging themes to build consensus regarding study findings. The findings will be used to generate hypotheses about factors that explain variability in performance after implementation of the MND and to inform development of a survey to measure these factors in all maternal-newborn hospitals in Ontario.
Objective 3—to measure factors hypothesized to be associated with differential effectiveness of the MND
Methods
Sampling procedures
For each maternal-newborn hospital in the province, an individual knowledgeable about organizational structure, quality improvement, and clinical practice, such as the obstetrical director, will be invited to complete the survey. Following ethical approval, we will initiate contact with these potential respondents by email and provide information about the purpose of the study, how the survey results might be used, the confidentiality of the data, and an invitation to participate.
Questionnaire development
Information obtained from the key informant interviews (objective 2), concepts contained in the PARiHS Framework, and the OR4KT Tool will inform development of the provincial survey. The survey will be developed using REDCap (Research Electronic Data Capture), a secure, web-based application designed to support data capture for research studies, hosted at the Children’s Hospital of Eastern Ontario Research Institute (CHEO RI) [49]. The survey will have four components (demographic information, questions about the attributes of the MND, clinical behaviors related to the KPIs, and facilitation/user support needs, and the OR4KT questions). The OR4KT questions will be used to explore contextual factors potentially influencing effective use of the MND. New questions will be developed to probe participants’ perspectives about the attributes of the MND and clinical behaviors related to the KPIs (e.g., content and clarity of the information displayed in the MND, evidence supporting each KPI and benchmark, audit features and functionality, and user access). In addition, questions probing the concept of facilitation will be developed focusing on the intensity of the facilitation activities undertaken, satisfaction, and internal and external supports. We will pilot test the survey with clinicians and administrators for clarity, length, and flow of questions, and the questionnaire will be revised if necessary. The OR4KT Tool will be used in its entirety as designed.
Survey administration
To promote a high response rate, the survey will be designed and administered using Dillman’s Tailored Design Method [50] for electronic mail surveys. If the response rate has not reached 80 % by weeks 10–12 after reminders, a follow-up phone call will occur.
Analysis
Descriptive statistics will be used to summarize characteristics and factors measured in the survey. A list of 18–20 factors will be identified for further statistical analysis (objective 4). To assess the representativeness of the survey, differences in the characteristics of hospitals with and without responses will be investigated using chi-squared and two-sample t tests (or non-parametric tests where required).
Objective 4—to identify factors significantly associated with differences in hospital performance before and after implementation of the MND
Analysis
We will conduct a multivariable generalized linear mixed-effects regression analysis of the repeated indicators at each hospital to identify those factors that are most predictive of between-hospital differences in effectiveness of the MND. Unlike the pooled analysis for objective 1, this analysis will use individual hospital-level data. The analysis will be limited to hospitals with annual birth volumes >100 to avoid numerical instability due to small denominators. With an anticipated 60 % response rate for the survey, approximately 50 hospitals will be included in the analysis. Appropriate time intervals for these longitudinal analyses will be chosen to avoid instability due to low denominators: we anticipate that 4 KPIs will be analyzed using quarterly intervals (12 pre-implementation and 8 post-implementation time points) and 2 KPIs will be analyzed using annual intervals (3 pre-implementation and 2 post-implementation time points). The generalized linear mixed-effects regression analysis of the repeated proportions at each hospital will use either a log-link function with the denominator specified as an offset term or a logit link function with the outcome specified in binomial form. For quarterly measurements, time will be modeled using a semi-parametric spline function with knots separating the pre- and post-implementation phases. Random intercepts and slopes will be specified for each hospital, and hospital-specific trends will be estimated using empirical Bayes’ best linear unbiased predictors. An advantage of this model is that estimates can be obtained even for smaller hospitals, as these estimates “borrow information” from the rest of the data, resulting in individual means that are shrunken towards the population mean. For annual measurements, time will be analyzed as a categorical variable and random intercepts will be specified for each hospital. To identify factors associated with differences among hospital trends over time, the candidate predictor variables identified in the survey related to the attributes of the MND (e.g., clarity of the KPI definitions, evidence summaries, visual displays, and audit features), contextual factors (e.g., leadership, culture, formal and informal interactions, and resources), and facilitation factors (e.g., training, resources, internal and external supports) will be entered into the model, together with their product interaction terms with time. To reduce the potential number of coefficients (2 intercepts and 2 slopes, 18–20 candidate predictors, 54–60 interaction terms), we will enter each candidate predictor variable plus its interactions with time separately into the model. Only those factors that have significant main or interaction effects will be considered for the full multivariable model. To arrive at a more parsimonious final model, we will use stepwise backward elimination, first removing interaction terms as necessary and then main effects.
Study status
At the time of preparing this manuscript, we have hired research staff, obtained ethical approval, developed tools to classify hospitals, developed and piloted data collection tools, and begun the process of data collection for the ITS and provincial survey.