Search Strategy

Inclusion Criteria

A study was eligible for inclusion if: 1) it was a randomized controlled trial (RCT) or controlled before and after (CBA) design, 2) authors evaluated interventions aimed at increasing research use or evidence-based practice, 3) participants were nurses, and 4) outcomes directly and explicitly captured research use. Only studies in English were assessed.

For criterion one, we defined RCT and CBA using Cochrane definitions. To meet criterion two, investigators must have explicitly stated that the research purpose was to test an intervention aimed at increasing research or evidence-based practice. For criterion three, we included both registered and student nurses and did not exclude based on type of nurse (i.e., psychiatric nurse, license practical nurse, etc). However, we did not include studies of nurse practitioners because, we argue, their practice has more similarities to medical practice than nursing. To meet criterion four, investigators must have explicitly described how their chosen outcomes represented research use or have used an instrument designed explicitly to measure research use. We excluded studies unless authors were explicitly clear as to how chosen outcomes captured a conceptualization of research use. This was a clear decision when authors used a tool designed to measure research use. However, to be included when a change in provider behavior was the outcome, the investigator had to have clearly described how the behavior reflected research use. For example, in evaluating the implementation of a clinical practice guideline, the investigator needed to measure all recommended behaviors outlined in the guideline, identify the percentage of recommended behaviors that signified research use, or illustrate how outcomes reflected their conceptualization of research use. If this was not done, we could not be certain the investigators were measuring research use and so we excluded the study.

Screening Process

The search resulted in over 8,000 titles. One author reviewed titles, abstracts and selected studies. Two reviewers each screened 20% of the titles and abstracts. Inter-rater reliability between reviewers was greater than 90%. The initial screening process resulted in 117 studies. Manual and ancestry searching produced an additional 21 studies. Further review of the 138 studies narrowed them to 14 and the final result was four studies meeting the inclusion criteria [24–27]: three RCTs and one CBA (Figure 1).

Excluded Studies

Methodological Quality

Two reviewers assessed each study and discrepancies were resolved through discussion. Each item was scored as: done, not done, and not clear. A quality rating was assigned to each study as low, medium, or high depending whether it scored done on zero to four, five to six, or seven to eight items respectively. Unit of analysis errors and incorrect power calculations were noted. We did not use quality assessment ratings to exclude studies because we sought to explore the general state of the science in this field.

Data Extraction

We extracted data from four studies representing five experimental cohorts where an intervention was compared to a control. One reviewer independently extracted data from all four studies while two reviewers each extracted data from two of the studies. We used extraction tools and dictionaries available from EPOC [38]. Data on design, subjects, setting, interventions and outcomes were extracted.

Several studies in this review reported additional outcomes, for example, on predictors of research use, changes in knowledge or attitudes, or patient outcomes. These were not extracted or reported on as we did not consider them as measures of research use per se.

Methodological Quality of Included Studies

Overall, the quality of the studies was low (Table 3). Two had unit of analysis errors where the investigators allocated by group but did not account for clustering in the analysis [24, 25]. Of the two studies without unit of analysis errors, the investigators of one study allocated by unit and accounted for clustering [26], while the other allocated and analyzed at the provider level [27]. No authors presented power calculations. Two studies had substantial differences in outcomes prior to the intervention [24, 26]. Allocation concealment was not reported in two RCTs [24, 26]. None of the investigators used blinded or reliable outcome assessments. The CBA investigators did not protect against contamination of the intervention across study groups [27]. However, the RCT investigators all randomized by ward and attempted to protect against contamination [24–26]. The CBA investigator reported adequate provider follow up [27]. However, the RCT investigators either used separate samples [25, 26], or did not report on follow up [24].

Included Studies

Four studies representing five intervention cohorts in Canada, USA, Taiwan, and Hong Kong met our inclusion criteria (Table 4). Three were RCTs (four intervention cohorts) [24–26], and one was a CBA (one intervention cohort) [27]. All studies included nurses from inpatient clinical settings; oncology, medicine, surgery and multiple specialties.

Investigators assessed educational meetings delivered to nurses in three studies [25–27]. In one study, the investigators compared two investigator-provided educational interventions to a control [26]. Because these interventions varied in content and duration, we identified this study as having two cohorts. Another study used a combination of local experts and educators to deliver the intervention [27]. The third study that assessed educational meetings used local opinion leaders identified by the study participants to conduct a demonstration tutorial which was supplemented with education delivered by a local expert [25]. One study investigated the formation of a multidisciplinary team of practitioners and researchers [24]. Within this intervention there were components of education and marketing. However, the investigators based their conclusions on the entire intervention (the multidisciplinary team) rather than the components, therefore, we did not separate the components of this intervention.

The investigators of three studies used nurse-administered instruments to measure research use. Dufault [24] used Kim's [39] 13-item Likert-type scale that asked participants to rate their research utilization competency on a one to seven scale. Tranmer [26] used the Research Utilization Questionnaire (RUQ) developed by Champion and Leach [40, 41]. This 42-item Likert-type questionnaire measured attitudes towards research, access to research, support of the use of research and research use. The questionnaire was divided into corresponding subscales. Because Tranmer [26] reported and analyzed the results of each subscale, we extracted only the data that pertained to the use of research subscale. Finally, using an instrument based on her previous work, Tsai [27] assessed whether research utilization was implemented in nursing practice and to what degree. Tsai's instrument consisted of 11 items including single-choice, multiple-choice and open-ended questions.

In the final study by Hong, investigators used self-reporting and participant observation to assess practice compliance with all the recommendations from a clinical practice guideline [25]. This study differed from many of the excluded studies that assessed provider behavior change. Specifically, the investigators linked all eight outcomes to the eight practices recommended by the clinical guideline, which was referenced to research, thus providing support that the outcomes did reflect research use.

Findings

Educational meetings

Two studies representing three cohorts tested the effect of interactive educational meetings on research utilization [26, 27]. Tranmer measured research use both in nurses who participated and nurses from the same unit as those who participated [26]. There were no significant changes in research utilization scores in either group. This suggests that, based on this study, educational meetings are ineffective whether a nurse participates directly (attending education meetings) or indirectly (working with nurses who attended educational meetings but not attending themselves). However, no definite conclusions can be drawn due to design limitations.

Educational meetings of varying content, frequency and duration (Table 6) were also found to be ineffective. Tranmer, who did not describe frequency of their intervention, reported non-significant changes in research utilization scores regardless of whether the intervention was twenty hours and focused on literature critiquing, research design, and protocol implementation, or eight hours and focused solely research design and implementation [26]. These results are supported by Tsai's study, in which she tested a series of educational strategies focused on research use totaling 65 hours and delivered over eight weeks [27].

Interactive educational meetings did not have a delayed effect on research utilization. Tsai measured research use at two points: immediately and six months following the intervention. In both cases, there were no significant changes in research utilization [27]. Similar findings were reported by Tranmer who measured research utilization only once, one year following the start of the intervention and also reported non-significant results [26].

In summary, based on this review, educational meetings of varying content, duration, and frequency cannot be said to be effective research utilization interventions in nursing. The studies were few in number and were of poor quality. Clearly, there is inconclusive evidence and educational meetings require more rigorous investigations to determine their effect in nursing.

Educational meetings and local opinion leaders

One study tested the effect of interactive educational meetings combined with a local opinion leader, and found that nurses who attended both the lecture and the tutorial (led by a local opinion leader) reported increased research utilization related to urinary catheter practices [25]. It was not possible to determine whether the positive effect was due to the local opinion leader, the educational meeting, or a combination of both. The intervention consisted of a 30 minute lecture by an educator, followed one week later by a demonstration tutorial conducted by a local opinion leader (Table 6). The length of the demonstration tutorial was not reported. No data were collected during the lapse between interventions. Outcomes were assessed twice: two weeks and two months following the intervention. The authors used a practice survey at two weeks, and direct observation at two months. Longitudinally, education and local opinion leaders appeared to sustain an increase in research utilization, but this study was also of low quality and represents inconclusive evidence for educational interventions combined with a local opinion leader.

Multidisciplinary committees

One study was found in which formation of multidisciplinary committees was reported to be effective at increasing nurses' research use related to oncology pain [24]. The intervention lasted 28 weeks and was divided into six stages (Table 6). Each stage was sequential and lasted between two and nine weeks. Stages were constructed around collaboration of members of the multidisciplinary team working to operationalize an existing research utilization process (the Conduct and Utilization of Research in Nursing Project) [42]. Unlike other interventions, education was not the primary component. Outcomes were assessed at one point using a research utilization scale. The investigators did not report the duration between the intervention and outcome measurement. Multidisciplinary committees require further investigation.

Summary of Findings

In summary, the four studies reviewed were of poor quality [24–27]. The findings of this review represent a lack of evidence to support or refute the benefit of educational meetings for increasing research utilization in nursing, and further study is required.

Comparison with Existing Reviews

In a review of organizational interventions aimed at increasing evidence-based nursing practice, Foxcroft and Cole could locate no rigorous studies [11]. Grimshaw and colleagues [6–8] published comprehensive reviews of provider behavior change reviews and guideline dissemination strategies. While we were interested specifically in nurses' research utilization and Grimshaw and colleagues [6–8] examined broader outcomes (provider behavior change and guideline dissemination), these reviews were all aimed at improving understanding of how to translate research findings into practice. Grimshaw and colleagues [7] concluded that interventions with different educational strategies showed mixed effects depending upon a combination of strategies. We report inconclusive evidence compared to these results [24–27]. The educational interventions included in our review were small interactive group sessions. In medicine, these types of educational strategies showed the most promise. We did find some limited support for two interventions: multidisciplinary committees and local opinion leaders. Grimshaw and colleagues [7] also found that multidisciplinary collaboration was effective and that use of local opinion leaders showed mixed effects.

Similarities and differences between these reviews can be attributed to multiple factors. Perhaps the most obvious is in the review methods. Grimshaw and colleagues [8] had a more robust dataset and derived a single effect size for each of the 235 studies reviewed, as well as summarizing the range of effects and median effects across studies for each intervention. In contrast, we were only able to locate four studies, and were limited to a narrative analysis based on the number of positive and negative results (vote counting).

Secondly, investigators of existing reviews have focused on changing provider behavior [6, 7], implementing practice guidelines [8, 10] and conducting continuing education [9]. We located no review focusing specifically on research utilization. Instead, authors have relied upon changes in patient outcomes, provider behavior, or a combination of both. For example, in a systematic review of guidelines in professions allied to medicine, 24% of nursing studies measured patient outcomes, 24% measured provider behavior, and 53% measured both provider and patient outcomes [10]. Additionally, all four nursing studies included in a review of continuing education and workshops measured provider outcomes [9]. Findings from these reviews [6–10] are increasingly being used to guide strategies aimed at increasing research use in clinical practice [13]. Hakkennes and Green suggested that authors must choose between assessing changes in provider practice or changes in patient outcomes when evaluating interventions aimed at implementing evidence or changing practice [13]. While this is true for the latter, we argue there is a third and perhaps more accurate choice: assessing changes in research use. By not assessing research use and moving directly to changes in provider or patient outcomes, investigators are treating research utilization interventions as a 'black box' phenomenon that somehow produces a change in clinician behavior or patient outcomes. Clearly, there are additional factors that influence both clinician and patient outcomes. The assumption that research use is the only mediating factor represents a large gap in the literature, and thus our understanding of how to increase the use of research in practice.

Conceptual Challenges

A major conceptual issue we identified is related to outcome measurement. We excluded several studies due to unclear conceptualizations of research use (Table 2). Investigators have commonly aligned themselves with a model of evidence-based practice consisting of five steps: 1) converting information needs to an answerable question, 2) locating the evidence, 3) critically appraising the evidence, 4) implementing the evidence in practice, and 5) evaluating care performances [43]. Most often, evaluations of interventions designed to increase evidence-based practice (or in our case, research use) relied upon behavior change outcomes [13]. Using the model outlined above, this approach is synonymous with evaluating care performances (Step five). However, research use is only one of the factors that influence care performance or behavior change [21–23]. Reliance on behavior change outcomes has treated the implementation stage (Step four) also as a 'black box' and led to less understanding of how implementation of research in practice occurs. We and others argue that the key to increasing research use in practice lies in the implementation stage [6, 22, 44, 45].

There is a lack of clarity and uncertainty, in fact a near silence, in the research community about what constitutes an appropriate measure of research use [46–48]. This silence can be partly to blame for a lack of research examining the intervention stage of research use and can be attributed to a poor understanding of the conceptual structure of research utilization [46, 49]. Ideally, selecting outcomes to assess the effectiveness of an intervention aimed at increasing research use should be informed by an explicit conceptualization of research use [22, 50]. Only two authors in our review explicated how they conceptualized research utilization [24, 26]; both offered different conceptualizations, and it was not clear from either how their conceptualization informed outcome selection. Rich argues that misconceptions of how research-based knowledge enters the decision-making process lead to inaccurate measures of research use [51]. Estabrooks and colleagues suggested that "unresolved measurement challenges present an important and practical problem" to advancing the field of research utilization [22]. The findings of this review support these claims and suggest such issues persist.

Investigators are interested in the link between using research in practice and improving patient outcomes. The abundance of studies focusing on behavior change and patient outcomes as a result of research uptake points at this interest. However, establishing this link is best accomplished if we first develop sufficient evidence to support the relationship between specific interventions and research use. From this, we can explore the relationship between effective research use interventions and behavior change or patient outcomes. If studies aim to evaluate an intervention to increase research use, outcomes must be structured to capture changes in research use. More attention to the fit between study outcomes and the conceptual structure of research use will advance the field by peering into the 'black box' and producing more accurate results.

Common problems with the instruments used in the studies we reviewed, and elsewhere, include lack of theory (measurement or research utilization theory), lack of construct clarity, lack of psychometric assessment, a presumption of linearity, lack of longitudinal work, and influential yet unacknowledged assumptions [22]. Until more reliable and valid instruments are developed, investigators should present explicit statements outlining the conceptual and practical basis for chosen outcomes. Making use of available conceptualizations to operationalize research use would increase the validity and ability to compare results across studies [16, 17], [51–53].

Methodological Challenges

The studies in our review were published between 1990 to 2003. Methodological quality (Table 3) was low in all four [24–27]. This suggests that the field is not developing within nursing as would be expected. We present what we believe are the most urgent methodological challenges facing the field.

Identification of Primary Outcomes

A primary outcome helps determine the key endpoint signifying the effectiveness of an intervention [54]. Explicit reporting of the primary outcome enables the reader to determine whether the study results provide sufficient evidence for an intervention and to whom the study results apply. In our review, we extracted only research use outcomes. However, three investigators in this review also reported outcomes additional to research use and all three assessed attitude towards research [24, 26, 27]. The relationship between such characteristics and research use is not well-supported [55]. When authors report on multiple outcomes without discussing why particular measures were chosen or what constitutes the primary outcome, it is difficult to interpret study findings in the context of research utilization.

Use of Multiple Outcomes

The challenge in using multiple outcomes to evaluate research utilization interventions is determining the number that must be changed to indicate effectiveness [33]. We excluded many studies due to uncertainty that the investigators were actually measuring research use (Table 2). In these cases, rationale or support for multiple outcomes in the context of research utilization was not provided. It is challenging to determine whether an intervention was effective at increasing research use if there were changes in some, but not all, of the outcomes. Also challenging is determining how many recommendations from clinical practice guidelines must be met to indicate research use, and for this review we included only one study that measured all recommended practices [25]. Measuring all outcomes may not be the most accurate or feasible approach, especially if guidelines recommend large numbers of practices or procedures.

Intervention Sustainability

Two studies [24, 27] measured longitudinal outcomes; one illustrated a benefit of intervention over time (two months) [24] and the other illustrated no effect either immediately or six months following [27]. Longitudinal outcome measurements will advance our understanding of the optimal timing and frequency of outcome evaluation and are needed to establish the sustainability of research use. Titler has described two challenges in assessing sustainability of research utilization interventions: defining the boundary between the end of the intervention phase and the start of the sustainability phase, and timing the outcome measurement to differentiate between sustained improvements and residual effects [48]. Compartmentalizing these stages becomes difficult when multiple and overlapping interventions are tested. Thus far, the literature on research utilization provides little guidance on the optimal timing or length of outcome measurement for different interventions. Hong [25] and Tsai [27] did not report why they assessed outcomes at two and six months. Future investigators should clearly describe intervention characteristics such as duration and frequency, deliverer and receiver, and mode of delivery. Guidelines such as the Consolidation of Standards for Reporting of Trials (CONSORT) [56] or the CONSORT statement for cluster RCTS [57] should be followed. Future reviews would also benefit from using a common classification system for interventions. We used a classification system proposed by EPOC. However, this approach may require adaptation for use in nursing and needs to be examined and validated.

Unit of Analysis Errors

Two RCTS included in our review had unit of analysis errors (Table 3) [24, 25]. Unit of analysis errors occur when investigators assign clusters or groups of individuals to a study group (i.e., intervention or control), and then analyze as if each individual had an equal chance of being assigned to either group [58]. When this occurs, outcomes for each individual are not independent of others within the same group. This is a unit of analysis error because people within clusters share similarities (i.e., burn unit and psychiatric nurses are each familiar with different practices) that may not be accounted for during analysis. When clustering is ignored, the number of participants required (sample size) is underestimated and the level of study significance (p value) is overestimated, resulting in an over-estimate of the precision of the result [59]. Future studies should be designed and analyzed using methods that account for clustering if allocation is done by groups of individuals, and allocation procedures should be unbiased (i.e., central randomization) and explicitly outlined in study reports.

Limitations

This systematic review has some limitations. First, we did not conduct a meta-analysis because of lack of effect sizes and a small size. The method we used (vote counting) is a crude estimate of effectiveness. Second, we used the EPOC classification that was developed for broad use [38]. Its applicability specifically to nursing has yet to be established. Third, the four studies included were all of low quality. Including studies of low quality limits any conclusions. Fourth, we included only studies published in English. While researchers have reported that language restrictions do not change the results of systematic reviews on conventional medicine interventions [60, 61], the impact of language restriction on research utilization intervention systematic reviews is unknown. However, it is possible that we missed potential studies due to our language restriction. Finally, because we focused on research use as the outcome of interest specific to intervention effectiveness, comparisons between our review and those of others must be made cautiously.