Turning evidence into recommendations: Protocol of a study guideline development groups
Implementation Science volume 2, Article number: 29 (2007)
Health care practice based on research evidence requires that evidence is synthesised, and that recommendations based on this evidence are implemented. It also requires an intermediate step: translating synthesised evidence into practice recommendations. There is considerable literature on evidence synthesis and implementation, but little on how guideline development groups (GDGs) produce recommendations. This is a complex process, with many influences on communication and decision-making, e.g., the quality of evidence, methods of presentation, practical/resource constraints, individual values, professional and scientific interests, social and psychological processes. To make this process more transparent and potentially effective, we need to understand these influences. Psychological theories of decision-making and social influence provide a framework for this understanding.
This study aims to investigate the processes by which GDGs formulate recommendations, drawing on psychological theories of decision-making and social influence. The findings will potentially inform the further evolution of GDG methods, such as choice of members and procedures for presenting evidence, conducting discussion and formulating recommendations.
Longitudinal observation of the meetings of three National Institute of Health and Clinical Excellence (NICE) GDGs, one from each of acute, mental health and public health, will be tape recorded and transcribed. Interviews with a sample of GDG members at the beginning, middle, and end of the GDG's work will be recorded and transcribed. Site documents including relevant e-mail interchanges, GDG meeting minutes, and stakeholders' responses to the drafts of the recommendations will be collected. Data will be selected for analysis if they refer to either evidence or recommendations; the focus is on "hot spots", e.g., dilemmas, conflicts, and uncertainty. Data will be analysed thematically and by content analysis, drawing on psychological theories of decision-making and social influence.
Widespread claims of evidence-based clinical practice are premised on development of recommendations based on research evidence. Using research evidence to inform practice requires three stages: synthesising evidence, translating evidence into recommendations, and implementing recommendations in practice. There is considerable literature on the processes of synthesising evidence, and of implementing recommendations, but little on how evidence is translated into recommendations. The model of guideline development adopted by the National Institute for Health and Clinical Excellence (NICE) in the United Kingdom involves a Guideline Development Group (GDG) comprising academic, professional, and lay representatives from relevant disciplines and practices. Groups meet between three and fifteen times to consider evidence and recommendations. Their face-to-face and e-mail discussions are informed by: Verbal and written presentation of research evidence by systematic reviewers and health economists as well as contextual evidence from co-opted experts; comments on the evidence and draft recommendations by stakeholders; and comments and advice by NICE internal review teams.
Despite the stated evidence base of guideline recommendations, there is little information about the relationship between research evidence and recommendations. A study of 15 clinical guidelines on Type 2 diabetes from 13 countries found that research evidence is not necessarily the most powerful influence on the content of recommendations . The investigators found little overlap between guidelines in the evidence used: only 18% of citations were shared with any other guideline, and only 1% appeared in six or more guidelines. Despite this, there was a high degree of consensus about recommendations about clinical care. Very different results were found in a study of two independent expert panels formulating appropriateness criteria for investigation of patients with angina. Given the same evidence summary and using a formal consensus process, the two groups showed only moderate agreement in their recommendations, with a two fold difference in their estimates of under-use for investigations in some subgroups of patients (Hemingway, et al., personal communication).
There is often no explicit framework for the conduct of the discussion or decision-making within GDGs. There are various possible influences on group discussion and decisions, e.g., the quality of evidence, the methods of presentation of the evidence, evaluation of different types of evidence, weight given to economic evidence, practical/resource constraints, opportunity costs, values (implications for inequalities), degree of change required by the recommendations, professional and scientific interests, potential direct or indirect financial interests [2, 3], and the knowledge and experience of the group members in evidence evaluation and synthesis. Additionally, social and psychosocial processes within small groups (e.g. status, conformity and compliance) may influence guideline development . Several tensions within this process have been reported: The extent to which recommendations should reflect contextual experience rather than just research evidence; the extent to which practical constraints should shape recommendations; and whether more evidence is needed to stop current practice than to start something new. A recent study using formal consensus methods for clinical guidelines found that half of group judgments did not agree with the research evidence. Where clinical experience and beliefs were not consistent with research evidence, the experience and beliefs seemed to take precedence .
There are calls for greater transparency about the processes of synthesising research and other evidence and of translating these into recommendations, but how best to do this is unclear. There are also calls for current practice to be changed to briefer, more structured deliberations . However, changing future practice should be informed by understanding current practice.
Communication with experts in the field and subsequent literature searches identified four studies that have analysed GDG processes. A pilot study has examined normative influences on guideline construction, although it was unclear what were "normative influences" . Pagliari and Grimshaw analysed social interaction between members and amount of communication according to professional status, and found more communication amongst higher status professionals . A sociological analysis of the structure and organisation of group processes found four domains of group reasoning: the scientific quality of evidence, the practical basis of recommendations, political implications of recommendations, and the quality of the process that evaluated evidence and formulated recommendations [9, 10].
The current study aims to build on this work by drawing on psychological theories of decision-making and social influence in describing and analysing how issues in these areas are resolved, with a goal of identifying good practice. It will do this across three health care settings (acute health, mental health and public health).
To investigate the processes by which GDGs formulate recommendations on the basis of the research evidence they receive, using psychological theory to guide data analysis.
The main questions to be addressed concern:
Who? Who has most influence, demonstrates particular approaches
How? What strategies are used in considering evidence and in formulating recommendations
Why? What beliefs about the purpose and nature of evidence and recommendations may explain these strategies
With what result? GDG members views about the quality of GDG process and outcome; objective quality indicators of the recommendations, e.g., clarity, comprehensibility.
A longitudinal observation study of the development of national guideline practice recommendations, from the first to last GDG meeting.
The first NICE GDG to start from November 2006 in each of three areas of health care, acute health, mental health and public health, will be selected. The GDGs participating in the study are not specified in order to protect anonymity of the members.
Data will be collected from several sources:
Meetings of GDGs and their topic group will be tape recorded and transcribed. Using an ethnographic approach, a researcher will attend meetings, take field-notes, and make observations. Members will be asked to introduce themselves by name and role at the beginning of each meeting to assist in their identification. The researcher will position herself in a place that is as unobtrusive as possible, but that will allow identification of speakers and a view of members' facial expressions. Not all meeting recordings will be transcribed and analysed. A pre-validated process will be used to identify the key exchanges and interactions relating to recommendations and evidence. Data from the pilot study is being used to develop criteria for this process.
Site documents include relevant e-mail interchanges, GDG meeting minutes, and stakeholders' responses to draft recommendations. Given the large number of e-mails that each GDG may generate, the research group will use the pilot study to develop a "key word" search approach to filter the site documents on the basis of their relevance to evidence and recommendations.
Following the first GDG meeting, the researcher will select a purposive sample for interviews from the different constituencies in the group (i.e., service providers, academics, practitioners, service users, systematic reviewers, NICE project team and chair). A selection of group members, representing different levels of involvement, will be interviewed at the beginning, middle, and end of the GDG process. Higher order interview questions have been developed to investigate GDG members' perceptions of the factors influencing their thinking and decisions about recommendations (additional file 1). These higher order questions will be mapped to specific incidences from each guideline meeting (identified by the researcher) in order to ensure the interviews are specific and focussed. Interviews will be audiotaped and transcribed.
The findings from the GDG discussions and interviews will be presented to GDGs for their comments to inform the face "validity" of the findings as well as any implications of the findings for future practice. The findings will be presented and responses sought first by e-mail to GDG members, followed by a discussion with the GDG as a whole. This will allow less vocal GDG members to input their views, while also enabling group interaction.
These are collected routinely and presented to the GDG.
Contextual influences may occur during the study period, e.g., high profile media or scientific publications. If these are discussed within the context of any GDG, they will be highlighted in the analysis.
The above procedures – gaining consent, the GDG observation, transcript analysis, collection and use of e-mail data – were piloted in September 2006 on a GDG that was considered by NICE members of the study steering group to be fairly representative.
Data will be selected if they refer to either evidence or recommendations; the focus will be on "hot spots", e.g., dilemmas, conflicts, and uncertainty. Selected transcript excerpts will be analysed thematically and by content analysis. The analysis of the transcripts and other data will draw upon a number of different theories, including group decision-making [11, 12] and communication/persuasion  theories. In consensual group decision-making (such as that occurring in NICE GDGs), it is assumed that there are background factors at play in the group's decision-making processes beyond the expression of mere individual preferences. These background factors include cognitive and affective components that are shared among the group during the decision-making process . Such components include majority/plurality processes (the degree to which members' preferences about information and evidence are shared socially), group polarisation (processes by which extremes in an individual's preference will be exaggerated when the group's preferences are combined), procedural manipulation (the manipulation of group preferences presenting information in a specific order or manner), and the common knowledge effect (where shared information is more likely to be discussed within a group than unshared information). Specific features of communication and the process of persuasion may influence how GDGs translate evidence into recommendations. The elaboration likelihood model of persuasion  suggests that when individual group members are subject to persuasive messages from others in their group with specific agendas, their ability and desire to cognitively elaborate on that message may influence the process and outcomes of decisions about the evidence.
In data analyses, discrepancies between reported and observed processes will be highlighted, as will any areas of tension or difficulty. Case studies will be systematically identified and analysed within the development process, e.g., uncontroversial translation of evidence into recommendations; lack of evidence for a practice but recommendation for it; clear evidence but no associated recommendation made; mixed evidence and no recommendation; and mixed evidence and recommendation made.
A variety of published GDG recommendations will be used to develop a reliable measure of recommendation clarity, e.g., readability, perceived comprehensibility, specificity and avoidance of ambiguity, and building on prior research [14, 15].
The primary deliverable for the study will be a set of recommendations, including the composition of GDGs, the presentation of evidence, and discussion of recommendations to aid transparent decision-making and reporting in GDGs.
A signed confidentiality agreement between the researchers, NICE, and the GDG chairs specify which GDG documents will be accessed, where they will be kept and reviewed, storage/disclosure, and access to data. Data will be anonymised as much as possible, and individuals will be given the opportunity to see draft reports to satisfy themselves about anonymity.
The study received ethics approval from the Research Ethics Committee of the UCL Psychology Department (ref: 0819/001). A study information sheet (additional file 2) will be provided to all GDG members before their first GDG meeting, and informed consent (additional file 3) will be sought for use of the following data: transcripts of GDG discussions, site documents (minutes arising out of the GDG meetings, e-mail exchanges about GDG business between GDG members, and draft versions of the recommendations), interviews with GDG members, and feedback focus groups of GDG members to comment on findings from the transcripts, e-mails, and interviews. Comments from non-consenting GDG members will not be transcribed.
Burgers JS, Bailey JV, Klazinga NS, Van der Bij AK, Grol R, Feder G: Inside Guidelines: Comparative analysis of recommendations and evidence in diabetes guidelines from 13 countries. Diabetes Care. 2002, 25: 1933-39. 10.2337/diacare.25.11.1933.
Choudry NK, Stelfox HT, Detsky AS: Relationships between authors of clinical practice guidelines and the pharmaceutical industry. JAMA. 2002, 287: 612-7. 10.1001/jama.287.5.612.
Black N, Murphy M, Lamping D, McKee M, Sanderson C, Askham J, Marteau TM: Consensus development methods: a review of best practice in creating clinical guidelines. J Health Serv Res Policy. 1999, 4: 236-48.
Pagliari C, Grimshaw J: The potential influence of small group processes on guideline development. Journal of Evaluation in Clinical Practice. 2001, 7: 165-173. 10.1046/j.1365-2753.2001.00272.x.
Raine R, Sanderson C, Hutchings A, Carter S, Larkin K, Black N: An experimental study of determinants of group judgments in clinical guideline development. Lancet. 2004, 364: 429-37. 10.1016/S0140-6736(04)16766-4.
Raine R, Sanderson C, Black N: Developing clinical guidelines: a challenge to current methods. BMJ. 2005, 331: 631-633. 10.1136/bmj.331.7517.631.
Berg M, Meulen RT, van den Burg M: Guidelines for appropriate care: The importance of empirical normative analysis. Health Care Anal. 2001, 9: 77-99. 10.1023/A:1011307112091.
Pagliari C, Grimshaw J: Impact of group structure and process on multidisciplinary evidence-based guideline development: an observational study. J Eval Clin Pract. 2002, 8: 145-153. 10.1046/j.1365-2753.2002.00333.x.
Moreira T: Diversity in clinical guidelines: the role of repertoires of evaluation. Soc Sci Med. 2005, 60: 1975-1985. 10.1016/j.socscimed.2004.08.062.
Moreira T, May C, Mason J, Eccles M: A new method of analysis enables a better understanding of clinical practice guideline development processes. J Clin Epidemiol. 2006, 59: 1199-1206. 10.1016/j.jclinepi.2005.08.021.
Kameda T, Tindale S, Davis J: From Cognitions, preferences and social sharedness; past, present and future directions in group decision-making. Emerging Perspectives on Judgement and Decision Research. Edited by: Schneider S, Shanteau J. 2003, USA: Cambridge University Press, 458-483.
Klein GA, Orasanu J, Calderwood R: Decision-making in Action: Models and Methods. 1993, Connecticut, USA: Ablex Publishing
Petty RE, Cacioppo JT: The Elaboration Likelihood Model of Persuasion. 1986, New York: Academic Press
Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A: Making psychological theory useful for implementing evidence-based practice: a consensus approach. Qual Saf Health Care. 2005, 14: 26-33. 10.1136/qshc.2004.011155.
Michie S, Lester K: Words matter: increasing the implementation of clinical guidelines. Qual Saf Health Care. 2005, 14: 367-370. 10.1136/qshc.2005.014100.
This study is funded by UK Medical Research Council and National Institute for Health and Clinical Excellence.
SM, JBS, PD and RR declare that they have no competing interests. GF has been paid to chair two NICE guideline development groups and one guideline development group by Weleda. FC, PA and SE advise on the methodology used for developing clinical guidelines at NICE.
SM conceived and developed the study, drafted the study protocol, and leads its implementation. SM helped to draft the manuscript. JBS coordinates the ongoing study, collected and analysed pilot data, and helped to draft the manuscript. SP, GF, PD, RR, FC, PA, and SE are members of the Study Steering Group, and have contributed to the development of the protocol. All authors read and approved the final manuscript.
Electronic supplementary material
Additional file 1: Draft Interview Questions. These are the core questions that were adapted for interviews with Guideline Development Group members. (PDF 9 KB)
Additional file 2: Study Information Sheet. This is the information sheet about the study given to eligible participants. (PDF 20 KB)
About this article
Cite this article
Michie, S., Berentson-Shaw, J., Pilling, S. et al. Turning evidence into recommendations: Protocol of a study guideline development groups. Implementation Sci 2, 29 (2007). https://doi.org/10.1186/1748-5908-2-29