Skip to main content

Impact of patient involvement on clinical practice guideline development: a parallel group study



Patient and public involvement (PPI) is recognized as a key component of clinical practice guideline development with important implications for guideline implementability. The impact of PPI on guidelines, however, has not been rigorously assessed. Better understanding of the impact of PPI must start with guideline question formation, which drives all subsequent development steps. The aim of this study was to investigate the effect of PPI on guideline question formation and validate a conceptual model of patient and public contributions to guidelines.


For development of a clinical practice guideline on the topic of using amyloid positron emission tomography in the diagnosis of dementia, we convened two parallel guideline development groups, one with and one without patient representatives. Participating physicians were randomized to group assignment. Each group developed Population, Intervention, Comparator, Outcome, Time (PICOT) questions and identified key benefits and harms to incorporate in guideline development. Analysis included a descriptive comparison of proposed PICOT questions, benefits, and harms between groups and a qualitative analysis of discussion themes from audio recordings of the question development retreats.


Proposed guideline questions, benefits, and harms were largely similar between groups, but only the experimental group proposed outcomes relating to development of cognitive impairment at specific time points and rate of progression. The qualitative analysis of the discussions occurring during guideline question development demonstrated key differences in group conduct and validated the proposed conceptual model of patient and public contributions to guidelines. PPI influenced the conduct of guideline development, scope, inclusion of patient-relevant topics, outcome selection, and planned approaches to recommendation development, implementation, and dissemination with implications for both guideline developers and the guideline development process.


Evidence of how PPI impacts guideline development underscores the importance of engaging patient stakeholders in guideline development and highlights developer- and guideline-specific outcomes of PPI, both of which have implications for guideline implementation. It also raises the question of whether guidelines developed without such input are acceptable for use. PPI should be considered an essential element of trustworthy guideline development for purposes of development and funding.

Peer Review reports


Clinical practice guidelines provide recommendations aimed at optimizing patient care and outcomes (at the individual or system level) based on a systematic literature review and assessment of benefits and harms [1, 2]. Guidelines are foundational to health care improvement efforts [3], but compliance with guidelines is variable and often poor [4,5,6]. When guidelines are implemented, they can improve outcomes and reduce resource utilization [7, 8], but important gaps remain in understanding optimal implementation strategies [9, 10]. There are many identified barriers to guideline implementation [11, 12]. These include guideline-related factors such as lack of confidence in development (credibility) [11, 13], stakeholder involvement (i.e., use of experts alone) [13], utility/applicability of the guideline in general [12, 13], applicability to individual patients [11, 13], and physician judgment about the balance of benefits and risks [11]. Research suggests that guidelines are more implementable when they address patient needs and preferences and include information to support patient involvement in decision-making [14] and that stakeholder involvement is a key domain impacting implementability [13]. Improving guideline implementation must thus start with involving key stakeholders—including patients—and developing guideline questions that result in relevant and applicable recommendations addressing patient preferences and needs.

In part for these reasons, patient and public involvement (PPI) is internationally recognized as an important component of guideline development. Numerous organizations recommend or require that guideline development panels include patients, patient representatives, or health consumers, including the Guidelines International Network [15], the United States’ Institute of Medicine (IOM, now renamed the National Academy of Medicine) [1], and the United Kingdom’s National Institute for Health and Care Excellence [16]. The Appraisal of Guidelines for Research and Evaluation II instrument requires that guideline developers seek the views of the target population [2]. PPI in guidelines is advocated because it recognizes that patients are experts, respects the rights of citizens in health policy development, empowers and informs consumers in health care decisions, and leads to the development of more patient-centered and trustworthy guidelines [17].

It is largely unknown, however, how PPI influences guidelines. Frameworks describe mechanisms for engaging patients and the public in guidelines [17, 18] and research [19, 20], but no identified framework describes contributions PPI makes to guidelines or anticipated outcomes. The Patient-Centered Outcomes Research Institute’s (PCORI’s) conceptual model of patient-centered outcomes research [21] outlines near-term, intermediate, and long-term outcomes of patient engagement:

  • Near-term: patient-centered culture and meaningful and effective partnerships

  • Intermediate: research relevant to patients and other stakeholders, questions and outcomes meaningful to end-users, use of research results in health decisions, quality health decisions, and satisfaction with health care experiences

  • Long-term: optimal health

When considered within this model, investigations of PPI in guidelines—largely guideline developers’ reflections on PPI experiences and analyses of participant feedback—suggest a framework for PPI contributions to guideline development (Table 1, Additional file 1) [16, 22,23,24,25,26,27,28,29,30,31,32,33], but one that requires further investigation and validation.

Table 1 Conceptual model of patient and public contributions to guidelines

Even if arguing for PPI on ethical and societal grounds rather than on anticipated outcomes, understanding the effect of PPI on guidelines is important in order to understand the guideline development steps at which engagement is critical, whether guideline developers are missing key questions by excluding patients/consumers, and whether PPI results in more patient-relevant outcomes and guidelines that are more acceptable to end-users and the public, all of which impact implementability. This study thus aimed to (1) investigate the effect of PPI on guideline question formation and (2) validate the conceptual model of patient and public contributions to guidelines.



A pragmatic parallel group study design was chosen to investigate the impact of PPI on guideline question formation. The question development step was selected because it is one of the critical first steps in guideline development, it is a step for which PPI is likely to have an important role, and the impact of PPI on later steps may depend on early engagement. The study compared the conduct and proposed questions for two guideline development groups, one including patient representatives (experimental group) and one involving physicians alone (control group).

The guideline chosen for the study—the use of amyloid positron emission tomography (PET) imaging in patients with or at risk for dementia—was selected from the American Academy of Neurology (AAN) guideline subcommittee’s waiting list of nominated projects [34]. The only prior amyloid PET guideline was developed by expert consensus [35], not an IOM-compliant evidence-based process, and two more recent policy statements [36, 37] are similarly consensus-based. This guideline topic is well suited to investigate the impact of PPI as there are strong patient, advocate, and public opinions regarding the diagnosis of Alzheimer’s disease (AD) and biomarker testing, amyloid PET tracers are approved by the Food and Drug Administration but not covered by the Centers for Medicare and Medicaid Services [38], and research shows that patients want amyloid testing results even when implications are uncertain [39, 40].

The University of Florida Institutional Review Board provided approval for this study (IRB201501210).


Each guideline development group included eight participants with different types of experience, balancing diverse views and manageability. Facilitators with prior experience in leading guideline development groups were invited to chair the two groups. As per usual AAN practice, an open invitation was offered at a guideline subcommittee meeting to engage subcommittee member participants. All guideline subcommittee members provided guideline development expertise; a subset also had content expertise. Additional content experts were recruited by their reputation in the field and recommendations from colleagues and/or prior AAN interactions. Patient, caregiver, and advocacy participants (“patient representatives”) were recruited through the Alzheimer’s Association’s Minnesota-North Dakota chapter. The US-based AAN takes a patient/stakeholder approach to PPI rather than seeking public/consumer representatives. All participants reviewed the IRB-approved consent document and verbally agreed to participate and have sessions audio-recorded. The study was operated under a waiver of documentation of informed consent (i.e., no signature was required). Participants were reimbursed for expenses related to meeting attendance but not otherwise compensated.

Population and randomization

Each guideline development group included one facilitator from the guideline subcommittee with experience in moderating but not content expertise, one dementia content expert without guideline experience, one content expert with amyloid PET experience, and 1–2 guideline subcommittee members with dementia experience. The four non-physician participants in the experimental group were offset by guideline subcommittee members without content expertise in the control group to maintain consistent group size with similar degrees of physician content expertise.

Professional participants were randomized to group assignment after training at the in-person guideline retreat in order to avoid bias that might result from specific physician assignments. Randomization was stratified according to role (facilitator, dementia expert, imaging expert). An AAN staff member typed group assignment on identical cards which were double-enveloped in opaque envelopes labeled with respective roles. A separate staff member confirmed that group assignment could not be determined by looking at the envelopes. Participants selected an envelope corresponding to their role for group assignment.

Data collection

Data was collected at a day-long in-person guideline question retreat at AAN headquarters in Minneapolis, MN, on July 15, 2016. Based on pilot study results [26], all participants received identical three-page pre-reading describing the topic and guideline development basics. At the retreat, participants jointly received training regarding the topic and guideline development. The pre-reading and in-person training were pilot-tested with University of Florida Citizen Scientist program participants.

After randomization, the groups split to meet in separate but identical conference rooms, one floor apart. The groups were tasked with developing the questions for the amyloid PET guideline using the PICOT (Population, Intervention, Comparator, Outcome, Time) format [30], identifying relevant benefits and harms, and crafting patient-language versions of proposed questions. The PICOT framework is the format used to develop answerable clinical question and inform search concepts and systematic review analysis; this framework is used for all AAN guideline questions [34]. Anticipated benefits and harms of amyloid PET use were collected separately as these inform recommendation development [1, 2] and may be distinct from diagnostic accuracy guideline question outcomes.

Data collected included meeting length, number of proposed questions, details of the PICOT categories for each question, and anticipated benefits and harms of testing. Session recordings were professionally transcribed verbatim using one recording and confirmed using a second. The principal investigator (MJA) added a blinded code for each speaker and confirmed removal of identifying information.


The analysis consisted of three components: (1) a descriptive analysis of meeting characteristics (e.g., length) and (2) retreat deliverables (e.g., PICOT questions) and (3) qualitative analysis of transcribed discussions. Investigators were not blind to group results. A qualitative descriptive approach [41] was used to identify, define, and organize themes from retreat discussions using NVivo 11 Pro and Microsoft Word tables. One investigator (MJA) with an interest in guideline methodology and patient engagement independently analyzed retreat discussions to create a log of codes reflecting emerging themes and sample quotes illustrating theme coding (open coding using an inductive approach). These were reviewed and discussed with a second investigator (ARG) to achieve consensus on emerging themes and to expand or merge thematic codes (axial coding). Similarities and differences in themes between groups were compared. Consolidated criteria for reporting qualitative research [42] guided the reporting of study findings (Additional file 2). Coding supporting article conclusions is included within the article and Additional file 3.

Conceptual model

Given the lack of an existing framework of PPI contributions to guidelines, themes and differences between groups were assessed using a revised version of the PCORI conceptual model of patient-centered outcomes research [21]. Prior to finalization of the current analysis, existing studies of PPI in guidelines [16, 22,23,24,25,26,27,28,29,30,31,32,33] were reviewed by the principal investigator (MJA) and results categorized within the PCORI model. The draft model was reviewed by co-investigators to achieve consensus on placement of themes within the model (Table 1, Additional file 1). Results from the current study were then framed within that model. The PCORI conceptual model, literature review, and current results were subsequently combined into a model relating to guidelines rather than research.


Nineteen individuals consented to participate: two methodologists, two facilitators, two dementia content experts, two dementia imaging content experts, seven members of the guideline subcommittee with and without dementia expertise, and four lay participants (one person with mild cognitive impairment [MCI], the spouse of the person with MCI, the spouse of a person with dementia, and an advocate from the Alzheimer’s Association) (Table 2). Two guideline subcommittee members contacted for participation declined (one had not volunteered and one was too busy), and five content experts contacted for participation declined (due to busyness, schedule conflicts, and/or conflicts of interest). No volunteer was turned down for participation. One consenting content expert was unexpectedly unable to travel as planned; guideline subcommittee members with dementia expertise were used to maintain balance in content expertise between the two groups. Eight panel members were randomized to each group, with each group supported by a methodologist and a staff person.

Table 2 Participant demographics

Meeting results

Question development was finished by both groups at the retreat. After the combined training, the experimental group met for longer than the control group (4 h 11 min versus 2 h 55 min; difference 1 h 16 min), but the control group forgot to draft plain-language question versions. The experimental group drafted eight over-arching PICOT questions versus four drafted by the control group, but the control group nested multiple populations within each PICOT question. In the control group, the methodologist, facilitator, content expert, and one of the guideline content experts accounted for 82% of the transcript word count. In the experimental group, the two content experts, facilitator, and guideline content expert accounted for 81% of the transcript word count. The patient, caregivers, and advocate in the experimental group accounted for the same percent of the transcript word count as the non-expert guideline committee members in the control group (14%).

PICOT questions

The PICOT questions proposed by each group are listed in Additional file 4. The main difference between the two groups was in the proposed outcomes.


Both groups identified three populations for guideline questions: individuals at risk for AD dementia without cognitive impairment (preclinical), people with MCI, and people with dementia. The experimental group specified that the MCI population of interest was people with MCI suspected to be prodromal AD. The control group suggested a screening question investigating the prevalence of positive amyloid PET in a general population without AD dementia.


The intervention for the guideline was, by definition, amyloid PET. Neither group specified individual PET tracers in their questions. The control group asked one question about the diagnostic accuracy of amyloid PET plus a “standard evaluation” in addition to the diagnostic accuracy of amyloid PET alone. The experimental group noted that amyloid PET performed in addition to an exam might be better than either amyloid PET or an exam alone, but did not write a question on this topic.


Both groups included two overarching comparators: (1) not getting amyloid PET and (2) some kind of other testing. For “other testing,” both groups described clinical evaluation, lumbar puncture with cerebrospinal fluid testing, volumetric magnetic resonance imaging (MRI), fluorodeoxyglucose-positron emission tomography (FDG-PET), and pathology in the setting of a clinical presentation consistent with AD dementia. The experimental group also mentioned tau imaging, functional MRI, 123I-Ioflupane SPECT scans, and genomic testing as potential tools that could contribute to reference standards.

Outcomes and time

Both groups included accurate diagnosis of AD dementia as an important outcome. For people without cognitive impairment (preclinical population) or with MCI, the experimental group was interested in the development of cognitive impairment due to AD pathology at 1, 3, 5, 10, and 15 years post-scan, whereas the control group was interested in the accurate future diagnosis of AD dementia. The experimental group also included the outcomes of accurate diagnosis of type of dementia and accurate prediction of the rate of decline/progression due to AD dementia.

The control group included a screening question for the general population without AD dementia; the outcome for that question was the prevalence of positive amyloid scans. The control group also asked one question about the outcomes of having an amyloid PET, such as costs, quality of life, benefits/harms of a “correct” diagnosis, stopping the search for other causes, treatment, and preparation for personal and social consequences of AD dementia if diagnosed. The benefits and harms of an AD dementia diagnosis and amyloid imaging were discussed by the experimental group, but the group opted to not include these as specific PICOT outcomes.

Benefits and harms

Because PICOT questions were expected to focus on diagnostic and prognostic accuracy outcomes, anticipated benefits and harms of amyloid PET use were collected separately to inform recommendation development [1, 2]. The groups described largely overlapping benefits and harms. The control group broadly grouped benefits of amyloid testing as (1) knowing the correct diagnosis, (2) stopping the search for other causes, (3) treatment, and (4) preparing for the personal and social consequences of AD dementia. Specific examples in both groups fell within these categories, including reducing uncertainty, accessing care and support, understanding prognosis to enable long-term planning, and opportunities for research enrollment. Harms described by both groups included misdiagnosis or misprognosis, coping with a correct diagnosis (e.g., depression), discrimination, and loss of job, volunteer, or insurance options.

Qualitative analysis

After the inductive qualitative descriptive analysis (Additional file 3), differences in group conduct were framed within the proposed conceptual model (Table 1, with numbers below referencing theme numbers within the table).

Culture of patient-centeredness

Shaping how discussions are conducted (1.1)

Introductions were markedly different between groups. In the control group, participants provided their names, specialties, and academic affiliations. In the experimental group, introductions started with the patient and caregivers who described their personal experiences with cognitive impairment. The advocate gave her name and affiliation, similar to control group physicians. Physician participants in the experimental group described their purpose and role in the project in addition to introducing themselves. Context experts in the experimental group described their backgrounds in depth.

In the control group, guideline subcommittee members without content expertise asked questions about amyloid PET scanning (how it is performed, tracer differences, etc.). These technical details were not discussed in the experimental group. The control group also spent more time discussing comparators/reference standards. In contrast, the experimental group spent more time discussing reasons it is important for patients and families to have a diagnosis in the setting of cognitive impairment. The patient representatives emphasized the importance of having a diagnosis in general, getting closure, validation that something is wrong, reduction in uncertainty, avoidance of unnecessary testing, understanding prognosis, linking to services and support, being able to plan one’s life, enabling formal advance planning, accessing disability and employment protections, accessing targeted treatment and care, and gaining increased control.

Setting patient-centered scope (1.2)

Patient representatives emphasized that the amyloid PET topic was important because of its real-life impact:

You know you can talk about the pathology but I have a life to live and the more information I can get about what are my likely paths for my life, that allows me more control over my life. And so, how can you translate the imaging and the other techniques to help me plan my future life? It’s kind of like a business, you know, you want to know the economic climate you’re in, what are your resources, what’s your prognosis and so I’ve got to plan my life based on what you can tell me. (Patient)

Describing personal impact of disease (1.3)

The patient and two caregivers described their personal experiences with cognitive impairment in the introduction and with examples throughout the meeting. These testimonies emphasized the guideline’s relevance:

I am caregiving for my husband of 52 years. He was diagnosed with Alzheimer’s in 2009, early stage Alzheimer’s. Subsequently he’s been diagnosed with MCI, subsequently with vascular dementia. Subsequently he has had a spinal tap, the [FDG]-PET scan... MRI, a blood test... Most recently the doctor said “I’m rather certain it’s not Alzheimer’s and I don’t know why you’re here.” And I fell apart because I’ve been caregiving for a very long period of time and my life is devoted to caring for my husband and my husband, according to him he has nothing wrong and is doing just fine and he continues to live as normal a life as possible… It’s been a very long process and actually in conversation with the doctor, he’s going to have the… [amyloid PET] imaging next week. We’ve been actually waiting for a bit to have it. So I will be very relieved to have some closure on what is happening with his brain because on a day to day basis, life is very challenging. (Caregiver-2)

Impacting how professional team members view PPI (1.4)

Participants in the experimental group noted the value of the patient voice:

It is really helpful having you here to hear…the decreased uncertainty, increased confidence in diagnosis and prognosis is so important. You know as doctors… we don’t want to do a test if it’s not going to change management, if there’s not a specific therapy that we’re going to offer based on that test… (Moderator)

Guidelines relevant to patients/stakeholders (including questions and outcomes)

Identifying issues that may be overlooked by medical professionals (3.1)

Patient representatives emphasized the importance of having a diagnosis, challenges of physicians not recognizing that something is wrong, poor communication between primary care physicians (PCPs) and neurologists, and physician reluctance to give a dementia diagnosis:

I am hoping that one of the outcomes of the amyloid testing will be that the diagnosis of the disease will be much more respected with doctors… Doctors don’t want to burden the family with the news when there’s no cure and they don’t really know what to do about it and why should I worry the patient. But I’ll tell you… having gone to 7 doctors before [my husband] was kind of diagnosed, it’s better to know than not to know than not to know. (Caregiver-2)

A content expert in the control group also noted a tendency for PCPs to underdiagnose dementia. The value of early diagnosis was also emphasized by the patient:

I think my life is extremely improved by having that early diagnosis… How can we get to that earlier diagnosis or just channeling say this is something to check up? (Patient)

Both groups mentioned caregiver issues, but this was more common in the experimental group (Additional file 3).

Helping select patient-relevant topics and outcomes (3.2)

While the control group was interested in the accurate future diagnosis of AD dementia, the experimental group specified an interest in the development of cognitive impairment due to AD pathology at 1, 3, 5, 10, and 15 years post-scan, as noted above. This emphasis on time-based prognostication was also present in transcripts:

The question really is, Ok can you answer questions that will help me live a better life or you know help me reduce the impact of the disease? So I mean part of it is just tell me I…you know, I have six months to live or six years. And the other one is, you know, what is the progression going to be and is there anything I can do to intervene in that? But it…and I just sense I’m going to repeat myself. I’m going to demonstrate I have this disease, but I mean how do I, you know, both treat it and live with it? (Patient)

Inclusion of time parameters was also based on input from physician panel members. Within the experimental group, there were different views on whether the specific type of future dementia was important. The benefits of a specific diagnosis were emphasized, but for prognostication, the patient felt that the dementia type was less critical:

I mean as a patient I just want to know am I going to get dementia, memory loss… The type depends on how you treat it, but the response to me is the same, I’m losing my mental ability. (Patient, with a mis-speak; treatment depends on dementia type)

Influencing guideline structure/development (3.3)

While this study focused on question development, groups identified issues relevant to later recommendation development. Both groups discussed potential harms resulting from PCPs/non-specialists ordering amyloid PET, restricting amyloid PET orders to specialists, establishing minimum standards for ordering, and pre-test counseling (Additional file 3). The experimental group also emphasized that there should be guidance regarding giving a diagnosis of dementia:

Also another very, a really important aspect of it... when a person receives the diagnosis, is how is it presented. The “how” is very, very important because if the neurologist is supportive and can say it in a way that it can be a safe diagnosis, if there’s such a thing for a patient and the caregiver, that makes all the difference… That is going to make a whole, a whole difference in terms of how a patient is going to take this diagnosis. (Caregiver-2)

In the experimental group, patient representatives played an important role in crafting the plain-language versions of the guideline questions, though there was banter regarding whether some suggestions reflected advanced educational backgrounds.

Use of results in health decisions/facilitating guideline dissemination and implementation

While the meeting focused on question development, topics pertaining to dissemination and implementation were addressed in both groups. When a guideline committee member in the experimental group described how AAN guidelines typically target general neurologists, the patient and advocate both raised the importance of disseminating the guideline to PCPs:

Can I just say, though, that patients generally interact with their general practice and how do we get this back to that stage, because that’s where I think… You know I was fortunate that my general practice doctor listened to us and you know asked the questions and then got us into looking at this seriously long before I would ever have thought about it and I think my life is extremely improved by having that early diagnosis. (Patient)

If we could get the family doctor to more readily refer to the neurologists; you know if the family doctor is not the one that’s actually doing this, but we need to connect them to neurology; I think sometimes they are in a little bit of a box even if they are in the same building. We’ve had experiences with clinics where the neurology department doesn’t talk to primary care and they are in the same hall. You know so I think… if this starts to establish some communication in that process that will improve the diagnostic process and it will improve outcomes. (Advocate)

Only the control group specifically mentioned creation of a shared decision-making tool. Patient representatives (experimental group) mentioned the importance of recognizing that patients will have differing views on whether they want the test or the results:

Everyone’s coming from a different place with regard to this and the passion that some people will have for wanting this information is …it differs on many different levels. (Caregiver-2)

In some ways from the Alzheimer’s Association’s perspective these [guideline group participants] are some unique individuals. Not everybody takes that position of wanting to know. (Advocate)

As noted above, patient representatives participated in crafting plain-language question versions for the protocol. Only the advocate provided e-mail feedback on the guideline protocol prior to public comment posting.

Conceptual model

Based on these study results and the preceding literature review [16, 22,23,24,25,26,27,28,29,30,31,32,33], the PCORI conceptual model [21] was revised to reflect a conceptual model of outcomes of PPI specific to guidelines (Fig. 1). PCORI’s “near term” and “intermediate” outcomes were reframed as organizational (developer) outcomes and guideline outcomes, with organizational outcomes also influencing guideline development, dissemination, and implementation. The “long-term” PCORI goal of optimal health was reframed as the goal of clinical practice guidelines: optimized patient care and health outcomes [1, 2].

Fig. 1
figure 1

Conceptual model of outcomes of patient involvement in guideline development. PPI patient and public involvement


Proposed PICOT questions were largely similar between groups, but the experimental group proposed looking at the future development of cognitive impairment at certain time points (rather than considering development of dementia as a binary outcome) and proposed rate of progression as an outcome. These outcomes relate to the importance of being able to plan for the future—a theme raised by patient representatives—suggesting that this difference relates to the presence of patient representatives and not to other group differences alone. This is also consistent with a systematic review finding that 85% (95% CI 76–94%) of individuals with memory concerns favor disclosure of their diagnosis for reasons relating to autonomy and the ability to plan one’s future [43].

Themes from the experimental and control groups validate the proposed conceptual model of patient and public contributions to guidelines. Patient representatives shaped how retreat discussions were conducted, contributed to a patient-centered scope, described the personal impact of disease, and impacted how physicians in the experimental group viewed the topic and patient involvement, all contributing to a culture of patient-centeredness at the organizational level and within the guideline development process. Patient representatives described issues not raised by participating physicians, identified patient-relevant outcomes, and contributed to discussions of how future recommendations should be framed. Finally, patient representatives participated in crafting of plain-language guideline questions, suggested a broad target audience for the guideline, and identified that patient preferences regarding this topic will vary, all issues with dissemination and implementation implications. Apart from forgetting to develop plain-language guideline questions—something only recently added to the AAN approach—the control group discussion was also patient-centered (e.g., identified benefits and harms largely overlapping with the experimental group, proposal of a decision aid as an important implementation tool), but patient representative participation influenced selected outcomes and group conduct/discussion. These identified PPI contributions to guideline development address barriers to guideline implementation including credibility, stakeholder involvement, and utility/applicability [11,12,13] and enhance guideline implementability though incorporation of patient needs and preferences [13, 14].

In contrast to prior work describing guideline developer experiences with PPI [16, 23, 24, 28, 31], the results of separate engagement of patient representatives and professionals [22, 27], or feedback from potential or actual guideline group participants [16, 24,25,26, 33], this study compared the conduct and products of guideline development groups with and without patient representatives, confirming an important role for this population even when physician participants have patient-centered views. This finding is critical, particularly as only 8% of US-based guideline developers require PPI on guideline development groups and only an additional 15% sometimes require it or describe it as optional [44]. A 2008 survey of international guideline developers reported that 39% of responding guideline developers reported PPI on guideline development groups, but only 29% of guideline developers always involved patients or the public and 39% involved them “only if necessary” [45].

Guideline developers describe barriers to PPI in guideline development groups including insufficient resources [29, 46], recruitment difficulties [29], the need for training and support (which is often described as inadequate) [16, 46, 47], uncertainty of how to incorporate patient experiences [33, 46], uncertainty of the patient role [33], patient representatives’ feelings of isolation [29], representativeness of selected participants [29], and patient representatives’ difficulty understanding medical terminology and participating in the systematic review [15, 16, 29, 46, 47]. With increasing research on engagement strategies, however, approaches for successful PPI in guidelines are also identified, including initiation strategies (e.g., diverse recruitment, establishing purpose), building reciprocal relationships (e.g., engaging multiple stakeholders in small panels, equal treatment), co-learning (e.g., training, skilled moderation), practical support (e.g., meeting logistics), and reassessment and feedback [20, 26]. Patient participation in this study followed best practices including involvement of multiple stakeholders with different backgrounds, identifying the purpose of engagement, practical support for meeting logistics, provision of pre-reading and in-person training, equal treatment, and use of skilled moderators. Feedback has not occurred to date as guideline development is ongoing.

The conceptual framework provides a novel model for the impact of PPI on guideline development. It highlights that PPI impacts organizational/developer outcomes as well as individual guidelines. Both outcomes have implications for successful guideline development and implementation. This underscores the need for guideline developers to prioritize PPI in guideline planning (including resource allocation) and further research on optimal methods for overcoming barriers. Research is also needed on the impact of consultation PPI strategies (e.g., reviews of published preferences, surveys, focus groups, public comment) and whether these approaches can meaningfully substitute for active patient participation or only supplement it. Consultation strategies can assist developers in setting patient-centered scope and selecting topics and outcomes but lack the ability to influence outcomes such as shaping discussion conduct, identifying overlooked topics, influencing guideline development, contributing to patient guideline versions, and active dissemination.


This study used a parallel-group design with randomized physician assignment to investigate the impact of engaging patient representatives on guideline development, but results reflect experience from a single guideline, limiting generalizability to other developers/methodologies and topics. The use of retreat group recordings did not permit specific questioning regarding why groups felt certain PICOT elements were of particular importance, but much of this can be gauged from the conversations informing question development. Guideline development groups included only neurologist and patient stakeholders, not other stakeholders (e.g., PCPs, psychiatrists) as is recommended [1, 2, 15]. While content expertise was balanced between groups, observed differences could reflect composition differences other than the presence of patient representatives. Patient representatives accounted for the same proportion of the transcripts as non-experts in the control group, but the “ideal” contribution from patient representatives in guideline development is unknown. Data were acquired solely during question development. This limits conclusions for the impact of PPI at other guideline steps, though retreat discussions touched on these subjects in a manner consistent with the conceptual model. The model outcome of meaningful and effective partnerships could not be fully assessed within the study, though patient representatives were recruited through the local Alzheimer’s Association chapter and this is hoped to promote additional collaborations. It is possible that other researchers would identify additional transcript themes or that researchers’ backgrounds in patient engagement influenced interpretation of results, attributing differences to PPI where others would point to group differences alone. Differences in proposed questions were supported by identified themes, however, and results fit within the referenced model. Finally, research was conducted in partnership with a guideline developer already largely compliant with the IOM standards and with participants aware of the study aims; it is possible that the impact of PPI would be different in other organizations and circumstances.


This study shows that engaging patient representatives on guideline development groups at the step of question development influences the conduct of guideline development, scope, inclusion of patient-relevant topics, outcome selection, and planned approaches to recommendation development, implementation, and dissemination. The conceptual model showing the impact of PPI on organizations/developers and the guideline development process underscores the importance of engaging patient stakeholders and raises the question of whether guidelines developed without such input are relevant to patients’ needs and implementable. PPI should be considered an essential element of trustworthy guideline development for purposes of development and funding. Further research is needed to identify optimal strategies for PPI involvement in guideline development to address recruitment, panel composition, and mechanisms of engagement.



American Academy of Neurology


Alzheimer’s disease


Fluorodeoxyglucose-positron emission tomography


Institute of Medicine


Mild cognitive impairment


Magnetic resonance imaging


Patient-Centered Outcomes Research Institute


Primary care physician


Positron emission tomography


Patient and public involvement


  1. Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. In: Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E, editors. Clinical practice guidelines we can trust. Washington, DC: The National Academies Press; 2011. Accessed 18 Aug 2017.

    Google Scholar 

  2. Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: advancing guideline development, reporting and evaluation in healthcare. CMAJ. 2010;182:E839–42.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Eccles MP, Grimshaw JM, Shekelle P, Schünemann HJ, Woolf S. Developing clinical practice guidelines: target audiences, identifying topics for guidelines, guideline group composition and functioning and conflicts of interest. Implement Sci. 2012;7:60.

    Article  PubMed  PubMed Central  Google Scholar 

  4. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348:2635–45.

    Article  PubMed  Google Scholar 

  5. Sheldon TA, Cullum N, Dawson D, Lankshear A, Lowson K, Watt I, et al. What’s the evidence that NICE guidance has been implemented? Results from a national evaluation using time series analysis, audit of patients’ notes, and interviews. BMJ. 2004;239:999.

    Article  Google Scholar 

  6. Runciman WB, Hunt TD, Hannaford NA, Hibbert PD, Westbrook JI, Coiera EW, et al. CareTrack: assessing the appropriateness of health care delivery in Australia. Med J Aust. 2012;197:100–5.

    Article  PubMed  Google Scholar 

  7. Mittal V, Darnell C, Walsh B, Mehta A, Badaway M, Morse R, et al. Inpatient bronchiolitis guideline implementation and resource utilization. Pediatrics. 2014;133:e730–7.

    Article  PubMed  Google Scholar 

  8. Hinchey PR, Myers JB, Lewis R, De Maio VJ, Reyer E, Licatese D, et al. Improved out-of-hospital cardiac arrest survival after the sequential implementation of 2005 AHA guidelines for compressions, ventilations, and induced hypothermia: the Wake County experience. Ann Emerg Med. 2010;56:348–57.

    Article  PubMed  Google Scholar 

  9. Chan WV, Pearson TA, Bennett GC, Cushman WC, Gaziano TA, Gorman PN, et al. ACC/AHA special report: clinical practice guideline implementation strategies: a summary of systematic reviews by the NHLBI Implementation Science Work Group: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. Circulation. 2017;135:e122–37.

    Article  PubMed  Google Scholar 

  10. Gagliardi AR, Alhabib S. Members of Guidelines International Network Implementation Working Group. Trends in guideline implementation: a scoping systematic review. Implement Sci. 2015;10:54.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, et al. Why don’t physicians follow clinical practice guidelines? A framework for improvement. JAMA 1999;282:1458–1465.

  12. Cochrane LJ, Olson CA, Murray S, Dupuis M, Tooman T, Hayes S. Gaps between knowing and doing: understanding and assessing the barriers to optimal health care. J Contin Educ Heal Prof. 2007;27:94–102.

    Article  Google Scholar 

  13. Kastner M, Bhattacharyya O, Hayden L, Makarski J, Estey E, Durocher L, et al. Guideline update is influenced by six implementability domains for creating and communicating guidelines: a realist view. J Clin Epidemiol. 2015;68:498–509.

    Article  PubMed  Google Scholar 

  14. Gagliardi AR, Brouwers MC, Plada VA, Lemieux-Charles L, Grimshaw JM. How can we improve guideline use? A conceptual framework of implementability. Implement Sci. 2011;6:26.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Qaseem A, Forland F, Macbeth F, Ollenschläger G, Phillips S, van der Wees P. Board of Trustees of the Guidelines International Network. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156:525–31.

    Article  PubMed  Google Scholar 

  16. Jarrett L, Patient Involvement Unit (PIU). A report on a study to evaluate patient/carer membership of the first NICE Guideline Development Groups. London: National Institute for Clinical Excellence (NICE); 2004. Accessed 30 Jan 2017.

  17. G-I-N PUBLIC Toolkit: Patient and Public Involvement in Guidelines. Accessed 4 Jan 2018.

  18. Armstrong MJ, Rueda J-D, Gronseth GS, Mullins CD. Framework for enhancing clinical practice guidelines through continuous patient engagement. Health Expect. 2017;20:3–10.

    Article  PubMed  Google Scholar 

  19. Carman KL, Dardess P, Maurer M, Sofaer S, Adams K, Bechtel C, et al. Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Aff (Millwood). 2013;32:223–31.

    Article  Google Scholar 

  20. Shippee ND, Domecq Garces JP, Prutsky Lopez GJ, Wang Z, Elraiyah TA, Nabhan M, et al. Patient and service user engagement in research: a systematic review and synthesized framework. Health Expect. 2015;18:1151–66.

    Article  PubMed  Google Scholar 

  21. Frank L, Forsythe L, Ellis L, Schrandt S, Sheridan S, Gerson J, et al. Conceptual and practical foundations of patient engagement in research at the patient-centered outcomes research institute. Qual Life Res. 2015;24:1033–41.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Tong A, Lopez-Vargas P, Howell M, Phoon R, Johnson D, Campbell D, et al. Consumer involvement in topic and outcome selection in the development of clinical practice guidelines. Health Expect. 2012;15:410–23.

    Article  PubMed  Google Scholar 

  23. Diaz Del Campo P, Gracia J, Blasco JA, Andradas E. A strategy for patient involvement in clinical practice guidelines: methodological approaches. BMJ Qual Saf. 2011;20:779–84.

    Article  CAS  PubMed  Google Scholar 

  24. Coon JT, Gwernan-Jones R, Moore D, Richardson M, Shotton C, Pritchard W, et al. End-user involvement in a systematic review of quantitative and qualitative research of non-pharmacological interventions for attention deficit hyperactivity disorder delivered in school settings: reflections on the impacts and challenges. Health Expect. 2016;19:1084–97.

    Article  PubMed  Google Scholar 

  25. Brouwers MC, Vukmirovic M, Spithoff K, Makarski J. Understanding optimal approaches to patient and caregiver engagement in the development of cancer practice guidelines: a mixed methods study. BMC Health Serv Res. 2017;17:186.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Armstrong MJ, Mullins CD, Gronseth GS, Gagliardi AR. Recommendations for patient engagement in guideline development panels: a qualitative focus group study of guideline-naive patients. PLoS One. 2017;12:e0174329.

    Article  PubMed  PubMed Central  Google Scholar 

  27. den Breejen EM, Hermens RP, Galama WH, Willemsen WN, Kremer JA, Nelen WL. Added value of involving patients in the first step of multidisciplinary guideline development: a qualitative interview study among infertile patients. Int J Qual Health Care. 2016;28:299–305.

    Article  PubMed  Google Scholar 

  28. Graham K. Patient/carer involvement in clinical guidelines: the SIGN experience. Presented at the conference of the Spanish HTA group (Seville, Spain). 2006. Accessed 30 Nov 2017.

    Google Scholar 

  29. Légaré F, Boivin A, van der Weijden T, Pakenham C, Burgers J, Légaré J, et al. Patient and public involvement in clinical practice guidelines: a knowledge synthesis of existing programs. Med Decis Mak. 2011;31:E45–74.

    Article  Google Scholar 

  30. Guyatt GH, Oxman AD, Kunz R, Atkins D, Brozek J, Vist G, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol. 2011;64:395–400.

    Article  PubMed  Google Scholar 

  31. Cowl J, Tyrrell H, Sakala C, Gracia J, Huang N. Chapter 1: How to conduct public and targeted consultation (2012). In: G-I-N Public Toolkit: Patient and Public Involvement in Guidelines (2015 ed). Pitlochry: Guidelines International Network; 2015. p. 17–27. Accessed 4 Jan 2018.

    Google Scholar 

  32. Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. BMJ. 2016;i2089:353.

    Google Scholar 

  33. van der Ham AJ, van Erp N, Broerse JE. Monitoring and evaluation of patient involvement in clinical practice guideline development: lessons from the Multidisciplinary Guideline for Employment and Severe Mental Illness, the Netherlands. Health Expect. 2016;19:471–82.

    Article  PubMed  Google Scholar 

  34. Gronseth GS, Cox J, Gloss D, Merillat S, Dittman J, Armstrong MJ, et al. Clinical Practice Guideline Process Manual, 2017 ed. Minneapolis: American Academy of Neurology; 2017.

    Google Scholar 

  35. Johnson KA, Minoshima S, Bohnen NI, Donohoe KJ, Foster NL, Herscovitch P, et al. Appropriate use criteria for amyloid PET: a report of the Amyloid Imaging Task Force, the Society of Nuclear Medicine and Molecular Imaging, and the Alzheimer's Association. Alzheimers Dement. 2013;9:1–16.

    Article  Google Scholar 

  36. Frey KA, Lodge MA, Meltzer CC, Peller PJ, Wong TZ, Hess CP, et al. ACR-ASNR practice parameter for brain PET/CT imaging dementia. Clin Nucl Med. 2016;41:118–25.

    Article  PubMed  Google Scholar 

  37. Minoshima S, Drzezga AE, Barthel H, Bohnen N, Djekidel M, Lewis DH, et al. SNMMI procedure standard/EANM practice guideline for amyloid PET imaging of the brain 1.0. J Nucl Med. 2016;57:1316–22.

    Article  CAS  PubMed  Google Scholar 

  38. Jacques L, Jensen TS, Rollins J, Coachman BB, Caplan S, Hakim R, et al. Final decision memorandum for CAG-00431N beta amyloid positron emission tomography in dementia and neurodegenerative disease. Center for Medicare and Medicaid Services; 2013. Available at: Accessed 5 Jan 2018.

  39. Shulman MB, Harkins K, Green RC, Karlawish J. Using AD biomarker research results for clinical care: a survey of ADNI investigators. Neurology. 2013;81:1114–21.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  40. Lerner AJ. Amyloid imaging: the court of public opinion. Neurology. 2013;81:1108–9.

    Article  PubMed  Google Scholar 

  41. Colorafi KJ, Evans B. Qualitative descriptive methods in health science research. HERD. 2016;9:16–25.

    Article  PubMed  Google Scholar 

  42. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007;19:349–57.

  43. van den Dungen P, van Kuijk L, van Marwijk H, van der Wouden J, Moll van Charante E, van der Horst H, et al. Preferences regarding disclosure of a diagnosis of dementia: a systematic review. Int Psychogeriatr. 2014;26:1603–18.

    Article  PubMed  Google Scholar 

  44. Armstrong MJ, Bloom JA. Patient involvement in guidelines is poor five years after institute of medicine standards: review of guideline methodologies. Res Involv Engagem. 2017;3:19.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Lavis JN, Paulsen EJ, Oxman AD, Moynihan R. Evidence-informed health policy 2—survey of organizations that support the use of research evidence. Implement Sci. 2008;3:54.

    Article  PubMed  PubMed Central  Google Scholar 

  46. van de Bovenkamp HM, Trappenburg MJ. Reconsidering patient participation in guideline development. Health Care Anal. 2009;17:198–216.

    Article  PubMed  Google Scholar 

  47. van Wersch A, Eccles M. Involvement of consumers in the development of evidence based clinical guidelines: practical experiences from the north of England evidence based guideline development programme. Qual Health Care. 2001;10:10–6.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references


The authors thank the University of Florida Citizen Scientists who participated in pilot-testing the guideline retreat materials.


MJA is supported by an Agency for Healthcare Research and Quality (AHRQ) K08 career development award (K08HS24159) for research on patient engagement in clinical practice guidelines, through which this work was conducted. CDM is supported by AHRQ grant number R24HS022135. AHRQ played no role in the study design; the collection, analysis, or interpretation of data; or writing the manuscript. The content is solely the responsibility of the authors and does not necessarily represent the official views of ARHQ.

Availability of data and materials

All data generated or analyzed during this study are included in this published article and its supplementary information files.

Author information

Authors and Affiliations



MJA contributed to the study concept and design, acquisition of the data, analysis and interpretation of the data, drafting of the manuscript, and critical revision of the manuscript. CDM contributed to the study concept and design, analysis and interpretation of the data, and critical revision of the manuscript. GSG contributed to the study concept and design, acquisition of the data, analysis and interpretation of the data, and critical revision of the manuscript. ARG contributed to the analysis and interpretation of the data and critical revision of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Melissa J. Armstrong.

Ethics declarations

Ethics approval and consent to participate

The University of Florida Institutional Review Board provided the approval for this study (IRB201501210). All subjects provided verbal informed consent after reviewing the IRB-approved consent form. The study was performed under a waiver of documentation of informed consent.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

Conceptual model of patient and public contributions to guidelines (Complete). Conceptual model table with all examples extracted from literature (not simply sample quotes as provided in table in the text). (DOCX 22 kb)

Additional file 2:

COREQ 32-item checklist for manuscript “Impact of Patient Involvement on Clinical Practice Guideline Question Formation: A Randomized Controlled Study”. COREQ checklist. (DOCX 16 kb)

Additional file 3:

Qualitative coding from transcripts. Microsoft Word tables showing qualitative coding. (DOCX 183 kb)

Additional file 4:

PICOT questions from experimental (Gp1) and control (Gp2) groups. Microsoft Word table of proposed guideline PICOT questions. (DOCX 17 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Armstrong, M.J., Mullins, C.D., Gronseth, G.S. et al. Impact of patient involvement on clinical practice guideline development: a parallel group study. Implementation Sci 13, 55 (2018).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: