Table 2 Use of the Pragmatic Explanatory Continuum Indicator Summary (PRECIS) and the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework domains to evaluate three studies

Setting

The POWER trials Collaborative Research Group tested weight loss interventions in primary care settings.

eHealth intervention trials that used patient facing technologies addressing all phases of the cancer care continuum (i.e., prevention through end of life care).

Ongoing, delayed cluster randomized intervention trial in primary care settings.

Sample

Three individual studies (approximately 1,100 participants) aimed at reducing weight in primary care patients who were obese and had at least one other cardiovascular disease risk factor.

113 grouped studies (149 total papers) were reviewed.

One study (nine pairs of diverse primary care practices matched on practice type, practice ownership, geographic region, electronic health record infrastructure, and patient population served).

When study was conducted

2008 to 2011

Studies published between 1980 and mid-2010.

January 2013

When during the research process was the domains used (evaluation, planning, etc.)

Implementation of study protocols.

Reporting - evaluation of published, peer-reviewed literature.

Planning

Modifications made to domains

Individual composite scores were created for both sets of ratings - PRECIS and REAIM.

For studies in which eHealth intervention replaced practitioners with no personal or phone contact, ‘not applicable’ ratings were applied to relevant PRECIS domains on practitioner expertise and practitioner compliance to study protocol.

Added patient engagement items to RE-AIM domains.

For studies in which multiple interventions were compared, the most intensive intervention served as the experimental arm and least intensive intervention served as the control arm.

Lessons learned from applying domains to type of study and stage of study

The utility of creating a composite score for each of the evaluation domains is questionable.

PRECIS and RE-AIM domains could be productively and reliably applied to a variety of eHealth studies. However, for studies in which eHealth interventions replaced practitioners with no personal or phone contact, deciding the best way to rate (or not rate) these domains was a challenge.

PRECIS domains prior to the start of the trial confirmed the design decisions aligned with the trial’s stated pragmatic purpose.

Since results of the trials were not available at the time of applying the domains, one must wait to evaluate the utility of the domains in terms of the extent to which scores predict eventual success of programs and their adoption, implementation, and sustainability in real-world settings.

Details of raters (number, who they were, how/why selected)

Nine reviewers total—six involved in one of the studies and three independent reviewers not associated with any part of the POWER studies.

Three PhD level and four masters level scientists were randomly paired for study reviews from a convenience sample.

Three PhD level scientists indirectly involved in the project were selected as convenience sample.

Most had Ph.D. or M.D. degrees and at least moderate experience in clinical trials.

Three had previous experience using PRECIS.

Only one had previous experience using PRECIS.

Training (resources used, processes, etc.)

Read article on the PRECIS domains by Thorpe et al.[16] and reviewed the slide presentation on PRECIS by Sackett [33].

All reviewers read the article on PRECIS domains by Thorpe et al.[16] the RE-AIM original article by Glasgow, et al., [22] and the slides on PRECIS by Sackett [33].

Read article on PRECIS domains by Thorpe et al.[16] and reviewed the slides by Sackett [33].

RE-AIM training was integrated into PRECIS training following discussion of original PRECIS domains. No background materials provided on RE-AIM.

Training sessions also served to develop consensus on all domains.

Conducted a one one-hour meeting to go over domains as a group and instructions for scoring.

The rating form was pilot-tested by all reviewers and refined based on the ratings of a sub-sample of four papers, not included in the study. After refinements and clarification of rating process, another two papers were assigned to the entire group.

Evaluation process

Each reviewer read the centrally available protocol materials on each intervention.

Articles were identified and grouped into studies if more than one article per study. Reviewers were assigned by study grouped papers.

Study protocol was independently reviewed and scored by each reviewer and a consensus meeting was held to resolve any discrepancies

Each reviewer read a background description of each project that appeared in Obesity and Weight Management [26].

After assignments, each reviewer conducted a brief web search of publications within PubMed to identify any relevant papers to the main study paper.

Any questions the reviewer had were answered by a contact person at each site, not involved in the ratings.

Each study was independently reviewed and scored by each reviewer and a consensus meeting was held to resolve any discrepancies among any ‘not applicable’ ratings.

Reviewers then independently rated each of the three projects on the PRECIS domains followed by rating on RE-AIM domains.

Reliability procedures

PRECIS domains - The overall kappa inter-rater reliability on the composite PRECIS score was r = 0.88. The interclass correlation for individual items was 0.72 and for averages 0.96.

Each study was rated by two reviewers.

Percent agreement using both exact and within one-point measures

RE-AIM domains- Inter-rater reliability on the composite scale was r = 0.71. The interclass correlations were 0.45 for single items and 0.83 for averages.

Weighted percent agreement scores were calculated for PRECIS and RE-AIM domains.

Lessons learned from training on use of domains

Leave ample time for training. Several conference calls were held among reviewers to discuss and attempt to specify precisely what was meant by each dimension and to develop and refine the rating instruments and instructions.

Leave ample time for reviewers to become familiar with the evaluation domains. It may take multiple sessions to arrive at consensus on how to routinely apply the domains.

Domains can be used with relatively little training to evaluate a study protocol.

Applications of criteria should be completed by independent reviewers not associated with studies being rated in addition to the team members due to observed differences in rating one’s own project.