How pragmatic is it? Lessons learned using PRECIS and RE-AIM for determining pragmatic characteristics of research

Over the last several years, there has been a substantial movement toward practical, pragmatic implementation research that will translate into usable health-related policies, programs and practices [1]–[4]. Pragmatic research is conducted internationally in wide ranging settings [5]–[8]. Funding to support pragmatic research and evaluation is provided by major health institutions such as the National Institutes of Health in the United States (U.S.), the U.S. Department of Veterans Health Affairs, the Canadian Institutes for Health Research, and the National Health Service’s National Institute for Health Research in the United Kingdom [9],[10]. Pragmatic research is increasingly being conducted in networks of primary care practices, health maintenance organizations, and other research networks such as the Patient Centered Outcomes Research Institute patient-powered research networks and the clinical data research networks [11].

The differentiation of pragmatic from explanatory research can be traced to a seminal paper by Schwartz and Lellouch [12] wherein they define explanatory research as conducted under optimal circumstances to determine the ‘efficacy’ of an intervention while pragmatic research tests an intervention under usual conditions. This distinction is important because trials are frequently designed as explanatory investigations, when the researchers’ intent is actually to answer the pragmatic question of effectiveness under usual or differing conditions. Inasmuch as trials are inadequately formulated for the type of research question asked, research outcomes are compromised and effort wasted [12]. The importance of pragmatic research has been given a major boost by the development of criteria and evaluation tools intended to increase transparency of research and results reporting and provide a means for practitioners and policy makers to assess local applicability of trial findings [13]–[15].

The original intent of the PRECIS framework was to inform trial designs by providing a visual display in the form of a hub and spoke diagram, where each of the 10 domains are represented by a line depicting the pragmatic-explanatory continuum. No numerical anchors were originally used. The endpoint closest to the hub represented a more explanatory study, whereas the endpoint furthest away from the hub represented a more pragmatic study [16]. However, modifications have been proposed and tested in variety of ways in an attempt to expand its utility to evaluate studies post completion, including use in systematic reviews [17]–[21]. Modifications have included quantifying the pragmatic-explanatory nature of a study by using numeric rating systems, where each domain is scored on a Likert-type scale. The original scale ranged from 0 to 4 where 0 represented an extremely pragmatic study and 4 was extremely explanatory. Over time the range most commonly used has been 1 to 5 (scales of 0 to 4 and 1 to 20 have also been used) [22]. Regardless of which scale is used, all have transposed the endpoints so the smaller number represents an extremely explanatory study and the larger number represents an extremely pragmatic study. Another modification was made to accommodate evaluating systematic reviews. Each study in a systematic review is scored individually on each of the 10 original PRECIS domains [20]. After the individual scoring, a 10-domain average for each individual trial can be calculated, as well as a single domain average across all trials included in the review and an overall combined average for the entire systematic review. Regardless of which version was used, all studies concluded that PRECIS was useful in designing trials and assessing the level of pragmatism of a trial or a body of evidence. However, PRECIS does not include domains to evaluate generalizability and applicability of a pragmatic trial to a specific context. Thus, additional domains are required.

The RE-AIM framework, which is an acronym for reach, effectiveness, adoption, implementation, and maintenance, was created out of the need for improved reporting on key issues related to robustness, translatability, and public health impact of health research [23],[24]. RE-AIM was developed as a response to trends toward research conducted under optimal efficacy conditions instead of in real-world, complex settings [25] and is intended to be used at all stages of research from planning through evaluation and reporting, and across different types of research (e.g., effectiveness, implementation, and dissemination trials) [26]. RE-AIM domains address issues focused on setting and participant representativeness, setting/site engagement with intervention, intervention adaptation during the study, program sustainability, and monetary/resource costs of an intervention. Over the past 14 years, RE-AIM has been applied to a wide range of conditions and study settings and has evolved to include additional items necessary for translation of research findings, such as use of qualitative methods and assessment of unanticipated consequences, both negative and positive (e.g. generalization effects). These domains address pragmatic and external validity issues not included within the PRECIS domains and are shown in Table 1. Originally, RE-AIM domains were not defined by a rating scale [23]. The first scale was modeled after our first use of the PRECIS rating scale that ranged from 0 to 4, where 0 represented an extremely pragmatic study and 4 was extremely explanatory. With subsequent uses of the RE-AIM domains, the scale has been changed to remain identical to the PRECIS scale where the smaller number represents an extremely explanatory study and the larger number represents an extremely pragmatic study.

The purpose of this article is to build on the work that has been done on the use and applicability of the PRECIS and RE-AIM frameworks by summarizing our experience applying these models to three studies that have combined both frameworks to provide a more robust and comprehensive assessment of issues related to translation of research. We begin by describing experiences of using both PRECIS and RE-AIM frameworks in three different studies. Thereafter, we summarize lessons learned using the combined criteria and make recommendations for future use. We conclude with a discussion on implications for the broader issue of designing and reporting results for studies intended to promote translation into policy and practice.

An increasing number of programs and studies claim to be pragmatic. Use of both PRECIS and RE-AIM frameworks can be used to demonstrate specifically where and how an individual study, or group of studies, is and is not pragmatic. Comparing ratings of domains within the same study allows for understanding the pragmatic versus explanatory design elements of the trial. Whereas comparing domain ratings across trials allows clinicians, policy makers, and study reviewers to compare across studies to make meaningful judgments about which intervention has generalizability and applicability to their population(s) of interest and the level of reasonable effectiveness that can be expected in different contextual settings versus those in explanatory trials.

Several evaluation frameworks have been developed to facilitate translation of research findings. However, many are designed solely for evaluation [35]. Combining both PRECIS and RE-AIM allows for standard reporting of both development and evaluation over the life course of a study. In the planning phase of a study, PRECIS allows for assessing the match between the trial design and the research and RE-AIM can be used to provide greater detail relative to some PRECIS domains (e.g., description of eligibility criteria and calculation of reach), and also to address other issues not in PRECIS important to potential adopting settings (e.g., costs required, representativeness of settings). RE-AIM can be used across the entire span of the study to understand the why behind success or failure of a study by describing the context in which the study occurred [36]. PRECIS can be used periodically throughout the study and at study conclusion to assess how adaptations and changes made over the course of the study impact the design and whether the end result still aligned with the original purpose of the study, respectively.

There is considerable benefit to using both frameworks to assess key components necessary for designing and reporting results for studies intended to promote translation of research into practice and policy. However, there are still many questions that need to be explored as use of both frameworks increases. First, what is the best rating scale to use? Is a 5-point Likert scale or some other scale the best way to evaluate a study or should one solely use a diagram without defined end-points? Is there value to using a scale to assess each PRECIS and RE-AIM domain or is a visual diagram sufficient? If a visual diagram is sufficient, is the PRECIS ‘spoke and hub’ diagram effective for also displaying the RE-AIM domains?

Second, use of some PRECIS domains to rate some health services studies is currently problematic. For example, in the eHealth review, there were studies that evaluated automated interventions without involvement of practitioners, with no personal or phone contact. ‘Not applicable’ ratings were applied to the PRECIS domains on practitioner expertise and practitioner compliance to study protocol (see Additional file 1). Is this the right way to apply the domains or should it be given a score? Moreover, usual care comparison conditions could be viewed as either explanatory or pragmatic depending on the lens of the evaluator. For example, how would participant compliance be rated when a health educator meets with a patient regarding self-management of diabetes and the educator encourages the individual to problem solve concerning self-monitoring their blood sugar levels and/or exercise more frequently to reach their health goals? This is usual care so could be viewed as extremely pragmatic in nature. However, it could also be viewed as being explanatory as it is encouraging patients to be more compliant. Thus, use of RE-AIM in addition to PRECIS can help complete, or at minimum, provide additional information to help understand why PRECIS domains might be viewed as not relevant or interpreted differently by two different evaluators.

Third, who should be a reviewer? There are pros and cons to including reviewers who are intimately familiar with a project versus those who are completely independent. Although not investigated in these three studies, these domains could be used by both researchers and stakeholders including patients and practitioners to help collaboratively design pragmatic studies. Additional studies are needed to determine if the finding that those familiar with a study rate it as more pragmatic is a generalizable phenomenon, as we observed in the ratings of the MOHR study. If so, this would imply that familiarity with a study should be balanced across studies to prevent potential bias.

The importance of pragmatic trials and dissemination and implementation research to improve health and health care delivery in the U.S. is gaining increased attention [38],[39]. Reporting on pragmatic rating criteria such as the PRECIS and RE-AIM scales can increase transparency and help reviewers and potential adoption settings make more informed judgments about programs and their applicability to different settings and under different conditions. However, because pragmatic research focuses on real-world applications of interventions, understanding the context in which it occurred is critical. Understanding whether a study design aligns with one’s research question in terms of being pragmatic versus explanatory should not stand alone without an understanding of how the context of participants, setting, and processes involved affected the results. We encourage those planning and evaluating health research interventions to use and report on PRECIS and RE-AIM domains, and to contribute to their refinement.