Standardising evidence strength grading for recommendations from multiple clinical practice guidelines: a South African case study
Implementation Science volume 13, Article number: 117 (2018)
Significant resources are required to write de novo clinical practice guidelines (CPGs). There are many freely-available CPGs internationally, for many health conditions. Developing countries rarely have the resources for de novo CPGs, and there could be efficiencies in using CPGs developed elsewhere. This paper outlines a novel process developed and tested in a resource-constrained country (South Africa) to synthesise findings from multiple international CPGs on allied health (AH) stroke rehabilitation.
Methodologists, policy-makers, content experts and consumers collaborated to describe the pathway of an ‘average’ stroke patient through the South African public healthcare system and pose questions about best-practice stroke rehabilitation along this pathway. A comprehensive search identified international guidance documents published since January 2010. These were scanned for relevance to the South African AH stroke rehabilitation questions and critically appraised for methodological quality. Recommendations were extracted from guidance documents for each question. Strength of the body of evidence (SoBE) gradings underpinning recommendations were standardised, and composite recommendations were developed using qualitative synthesis. An algorithm was developed to guide assignment of overall SoBE gradings to composite recommendations.
Sixteen CPGs were identified, and all were included, as they answered different project questions differently. Methodological quality varied and was unrelated to currency. Seven clusters, outlining 20 composite recommendations were proposed (organise for best practice rehabilitation, operationalise strategies for best practice communication throughout the patient journey, admit to an acute hospital, refer to inpatient rehabilitation, action inpatient rehabilitation, discharge from inpatient rehabilitation and longer-term community-based rehabilitation).
The methodological development process, tested by writing a South African AH stroke rehabilitation guideline from existing evidence sources, took 9 months. The process was efficient, collaborative, effective, rewarding and positive. Using the proposed methods, similar synthesis of existing evidence could be conducted in shorter time periods, in other resource-constrained countries, avoiding the need for expensive and time-consuming de novo CPG development.
Over the past decade, the South African burden of disease has swung towards chronic conditions, trauma and disability, as more lives are saved from communicable diseases [1, 2]. The shift from communicable disease mortality, to communicable and non-communicable disease morbidity, puts the spotlight firmly on the need for evidence-based rehabilitation, to ensure that resources are wisely allocated to achieve best health and cost outcomes for people living with sequelae of illness and injury . South Africa (SA) has been described as an anomaly among developing countries, having features of high-income country economies (such as good infrastructure) but also features of low-middle-income country economies, with its social and economic problems and continuing need for development aid .
Effectively implementing evidence-based practice, particularly using comprehensive high quality clinical practice guidelines (CPGs), has been the subject of considerable research in high-income countries over the past three decades . Much of this research has focused on why so much difficulty is experienced by policy-makers, managers and clinicians in implementing evidence into practice . Across health disciplines, there are generally positive attitudes to using evidence in practice, and despite discipline-differences in competencies and scope of practice, similar reasons emerge for not actually doing so [7, 8]. Commonly, these are lack of time, ready access to CPGs, understanding about different forms of evidence recommendations (e.g. CPGs, protocols, guidance) and knowledge about evaluating evidence quality. Barriers also include disagreement with recommendations, unwillingness to change practices, peer-pressure, lack of managerial and organisational support and differences between recommendations and clinical realities [7,8,9].
Research into evidence implementation and uptake in low and lower-middle-income countries has largely been in knowledge translation into policy, which has concurrently identified gaps between research and end-user/stakeholder needs for guidance . The challenges of evidence implementation into clinical practice in these countries are yet to be fully identified or addressed .
This paper outlines a novel process of using existing best evidence sources to develop new, composite, implementation-ready, evidence-based recommendations. The process was tested on allied health (AH) rehabilitation for a high-burden health condition in a resource-constrained environment (stroke). AH disciplines such as physiotherapy, occupational therapy, speech and language therapy, clinical nutrition and social work provide rehabilitation services to stroke survivors, with the aim of optimising function and quality of life. A 2015 summary of systematic reviews of the impact of AH care on stroke rehabilitation outcomes showed significant economic benefits . Key barriers to implementing evidence-based AH rehabilitation in SA are similar to those reported in high-income countries (for instance lack of workforce, training, support, resources and recognition of effort) [8, 9, 13]. However, there are also unique SA-context facilitators which mediate some of these barriers to improve AH rehabilitation, including the innovative use of resources and informal AH rehabilitation networks with the common goal of improving functioning and quality of life .
The World Health Organization (WHO) notes that AH rehabilitation services are generally poorly accessed and/or suboptimal in many low and low-middle economy countries . The World Health Assembly resolution on disability, including prevention, management and rehabilitation, identified that effective AH rehabilitation could contribute to reducing poverty through improving functioning, activity levels and participation. Inefficient and ineffective rehabilitation can cause health deterioration, which is associated with an increased rate of complications and healthcare utilisation .
Stroke is a leading cause of disability worldwide . Over the past 40 years, stroke rates in poor economies such as southern India and rural SA have approximately doubled, whereas rates in upper-middle- and high-income economy countries have decreased. The most striking problem is that disability- and mortality-rates from stroke are at least tenfold greater in medically under-served countries versus medically well-served countries . The causes of these disparities are explained by reduced access to services such as early stroke screening, effective early medical management, post-stroke rehabilitation and secondary stroke prevention. The WHO promotes evidence-based public health programs for stroke prevention, management and rehabilitation worldwide, however, the success of such programs depends on government commitment to evidence availability, uptake and implementation .
In SA, stroke is a high health burden [1, 2]. It is estimated that 240 people have a stroke each day, which translates into ten strokes each hour . Stroke now affects many young South Africans in their 20s and 30s due to co-morbidities such as HIV/AIDS. In SA, stroke is a leading cause of disability among adults of all ages , contributing significantly to healthcare-costs with long-term implications, particularly if rehabilitation is sub-optimal.
South African AH stroke rehabilitation services are currently not supported by local CPGs that summarise the AH literature . Moreover, there is no nationally-recommended CPG from another country that is routinely used by South African AH stroke rehabilitation providers. The lack of local guidance perhaps underpins variable stroke-rehabilitation outcomes across SA . This concurs with the WHO report, which suggests that only 26–55% of people in poorer countries receive the rehabilitation services they need . The World Health Survey analysis indicates that people with disabilities were more than twice as likely to find healthcare provider skills or equipment inadequate and nearly three times more likely to be denied care . South African AH providers require access to good quality, locally-relevant guidance to support them to deliver best-practice stroke rehabilitation to the growing number of South Africans who require it [1, 2]. They also require a process by which guidance can be readily updated to ensure that the practice remains current.
The aims of the research were to develop a template for producing defensible, locally-implementable recommendations from existing evidence sources, and test this by writing an implementation-ready SA-contextualised AH stroke rehabilitation guideline relevant to any South African public healthcare setting.
Stellenbosch University Africa Human Research Ethics Committee (#0642).
Seed funding was provided from Stellenbosch University and the WHO Alliance for Health Policy and Systems Research in February 2017. The funders had no influence on research design, conduct or reporting.
This comprised 15 expert clinicians, academics, policy-makers and consumer representatives and five researchers. Other than the consumer representative (a stroke survivor), each team member had an AH discipline background. Each team member brought different understandings of stroke rehabilitation, such as how, when and why stroke rehabilitation is offered, what care is provided, who provides it, how it is received and in what rehabilitation settings. Project funding was sufficient for travel and meeting costs and to support one researcher 2 days/week. All other team members participated voluntarily or as part of usual work commitments.
The timeframe was short (9 months) (Feb–Nov 2017), reflecting the urgent need for effective, efficient, equitable and safe AH rehabilitation for patients with acute and chronic stroke across South African healthcare settings. There was no time or funding to develop a de novo CPG for South African AH stroke rehabilitation, which outlined ‘what’ to do. The focus needed to be on implementing best evidence into local practice, by addressing service delivery questions of ‘who’, ‘how’, ‘when’, ‘where’, ‘why’ and ‘how much’. The project team agreed that existing CPGs would form the evidence base.
The research adopted the WHO characteristics for good quality service delivery . This separates current best-practice information (largely derived from non-biased comparative or intervention studies) from service operationalisation issues. These speak to inputs such as workforce, service comprehensiveness, resources, continuity, coordination, accountability and outputs (quality care processes and health outcomes). Outputs can be variably measured by person-centredness, efficiency, equality (individual rights to care), equity (coverage), access, timeliness and effectiveness.
Any document published from 2010, which provided freely- and publicly-available guidance to inform best practice AH rehabilitation for any adult stroke sufferer was eligible for consideration. The 7-year search window encompassed the usual five-yearly update period for CPGs . For inclusivity, guidance documents did not need to be called CPGs to be included. This is because different nomenclature has been reported internationally to describe similar types of guidance document . Thus an inclusive, overarching term ‘guidance documents’ was used.
Essential preliminary steps
To ensure the common understanding of purpose and an efficient focus for discussions, agreement was reached at the first meeting on the underpinning project premises (Additional file 1) and on the assumptions underpinning the decision to employ existing guidance documents as the evidence base (Additional file 1). Moreover, based on the project team’s familiarity with international stroke rehabilitation CPGs, it was assumed that no one guidance document would be identified that would answer all project questions.
Scope, target group and purpose
The scope was the best-practice guidance for AH rehabilitation for acute or chronic stroke, in any public healthcare setting in SA. Over 90% South African stroke patients receive treatment in the South African public system [1, 2]. The target group was AH policy-makers, clinicians, managers, educators and researchers. Not in scope were the areas of stroke care in which AH do not play a role in SA (specific pre-hospital emergency care, specific hospital-based medical care to manage and stabilise acute stroke, pharmaceutical management (except where it may be relevant to AH rehabilitation)). Whilst AH care specific to the South African private sector was not directly in scope, guidance documents discussing private sector care may be included, if relevant.
The project purposes were to
Produce a methodology for efficiently finding and combining existing evidence sources into new, comprehensive, evidence-based, implementation-ready, locally-relevant guidance for resource-constrained healthcare settings, and
Test the methods by producing best-practice guidance for AH rehabilitation for South African stroke survivors from current international guidance documents.
The research built on the implementation framework proposed in the South African Guidelines Excellence Project (Project SAGE 2013–2017) [20, 21]. Project SAGE framework produced a three-tier model of CPG activity particularly relevant to resource-constrained environments. Tier 1 was a summary of the current best-available evidence; tier 2 reflected local stakeholder input on implementation of tier 1 evidence and tier 3 comprised documentation, collated from existing resources, or developed specifically for local contexts, to underpin efficient local implementation of tier 1 evidence .
Identifying relevant guidance documents
A comprehensive search of internet repositories was undertaken to identify guidance documents for AH stroke rehabilitation published since January 2010, by any organisation, in any country. The search strategy is provided in Additional file 1.
Guidance document quality
Irrespective of their nomenclature (CPG, guidelines, protocol etc) , methodological quality of the included guidance documents was assessed using the AGREE II instrument [22, 23]. The AGREE II domains are scope and purpose, stakeholder involvement, rigour of development, clarity of presentation, applicability, editorial independence. Because of its sensitivity, AGREE II would discriminate between the construction quality of different types of guidance documents . For instance, it would detect differences in the rigour of development domain between an opinion-based protocol and a CPG based on systematic evidence searching . The project team also assumed that the use of the AGREE II instrument overrode the need to interrogate included guidance documents for search strategies, literature inclusion, evidence synthesis methods, evidence tables or included studies.
Independent scorer dyads, experienced in using AGREE II, scored randomly-allocated guidance documents, and the scoring rubric was calculated independently using a MSExcel macro [22, 23]. Whilst AGREE II instrument metrics are not usually reported as total scores, these were calculated as an overall quality indicator. Arbitrary total overall quality score classifications were determined as
high quality (HQ) being 80% + of the total possible AGREE II score,
moderate quality (MQ) being 60–79% of the total possible AGREE II score and
poor quality (PQ) being < 60% of the total AGREE II score.
The project team identified, discussed and agreed on key questions which, if answered by current best-evidence, could improve the quality of South African AH rehabilitation. A five-step process was taken.
‘Usual’ patient care pathway(s) were established during the first project team meeting . These outlined how, where and when stroke patients usually accessed care in the South African public sector [1, 2, 13]. South African stroke patients could enter and exit South African public sector healthcare at different points, as outlined in the top layer of Fig. 1. This pathway assisted in framing the project questions.
The project team then discussed AH rehabilitation activities related to the patient care pathways and raised issues for which there was variable practice and/or uncertainty about what to do. This process highlighted the complexities of delivering best-practice SA stroke rehabilitation, particularly the need for guidance on the workforce, training, organisation of services and communication. These issues are reported in the second layer of Fig. 1.
The team constructed 47 questions for which evidence-based answers were required. Questions were classified relevant to second tier of the patient pathway in Fig. 1. By this process, seven broad patient care activity clusters were identified, within which answers (recommendations) could be provided (see the third tier of Fig. 1). These clusters comprised organise for best practice rehabilitation, operationalise strategies for best practice communication throughout the patient journey, admit to acute hospital, refer to inpatient rehabilitation, action inpatient rehabilitation, discharge from inpatient rehabilitation and longer term community-based rehabilitation. This approach highlighted that some activities were relevant across the entire pathway (e.g. organisational and risk-minimisation activities), whilst activities such as referral to rehabilitation, actioning rehabilitation and discharge planning were specific to sections of the patient journey.
The project questions were condensed to 38 by team discussion (Table 1). These were organised by intent and relevance to the activities described in the second and third tier of the patient pathway (see Table 2). Eighteen questions related to communication, seven to service delivery, 15 to organisational issues, nine to clinical questions and five to training. Of note was the small number of questions related to clinical care. This validated the adoption of the WHO implementation focus, as it was evident that there was greater local need to know about ‘who’, ‘how’, ‘when’, ‘where’, ‘why’, ‘how much’, rather than about ‘what to do’ .
The project team anticipated that no one guidance document might answer all project questions. This was because of the specific nature of the South African stroke rehabilitation project but also because of the different purposes for which current international AH stroke rehabilitation guidelines might be written. Thus, if a guidance document identified in the search did not answer a project question, it was assumed that this question was not relevant to its scope or purpose.
Excluding guidance documents
The scope, purpose and questions addressed in the guidance documents identified in the search were mapped to the South African project questions. To be included, a guidance document should answer at least one South African project question, provided in the form of a recommendation (this is defined in a later section). Poor methodological quality was not a reason for exclusion, as quality was taken account in the calculation of the overall strength of the body of evidence (OSoBE) (see later section).
Strength of the body of evidence (SoBE)
A summary of the relevant evidence usually underpins recommendations in good quality guidance documents evidence, reported as SoBE [25, 26]. Berkman et al.  define SoBE as a method ‘to help clinicians, policymakers, and patients make well-considered decisions about health care. The goal of strength of evidence assessments is to provide clearly explained, well-reasoned judgments about reviewers’ confidence in their systematic review conclusions so that decision-makers can use them effectively’ (p 1314). However, there is no standard approach to formulating or reporting SBE, which presents a challenge when comparing recommendations from different guidance documents.
wording in guidance documents that was clearly labelled ‘recommendation’ (appearing in designated recommendation boxes, specific fonts or tables) and accompanied by a SoBE grading, or
wording that appeared in the text, not necessarily labelled ‘recommendation’ but which had the intent of being a recommendation, in terms of intention words such as ‘should’, ‘could’, ‘might consider’, accompanied with an SoBE grading.
Not considered as recommendations were words which appeared in the body of guidance documents which were not labelled ‘recommendation’, did not have the intent of a recommendation or had no SoBE grading.
Extracting data for project questions
Separate purpose-built data extraction sheets were developed for each project question. These recorded the details of the guidance documents which answered it (year of construction, methodological quality, country of origin), recommendations relevant to the question, and the SoBE gradings for recommendations (in whichever way they were reported). Moreover, any guidance document which provided ‘how to do it’ information was identified as potentially providing useful tier 3 documents. These could include, but were not limited to, protocols, patient management or service decision-making tools, organisational flowcharts, stroke team construction, workforce issues, assessment criteria, specific assessment tools, outcome measures, minimal clinically significant changes from interventions, discharge planning checklists and patient information material.
A common, readily interpreted set of ‘faces’, was developed for this project to standardise the different ways in which SoBE was presented in included guidance documents. A smiley face was used for positive evidence, a neutral face for insufficient or conflicting evidence and a frowning face for negative evidence. Details are provided in the ‘Results’ section.
Compiling the evidence
A summary table was developed from the individual data extraction files, reporting the guidance documents which provided answers to each project question (volume of evidence), year of production of guidance documents (currency of evidence), their methodological quality and the consistency of recommendations (‘do all recommendations point in the same direction’?). Project questions were then classified as
those which were not answered by any guidance document (absence of evidence),
those answered by only one–two guidance documents (scant evidence) and
those with inconsistent SoBE (inconsistent evidence).
Writing composite recommendations
There is currently no methodology about how to develop a composite recommendation which summarises the intent of recommendations extracted from two or more guidance documents. The project team developed a new approach using inductive thematic content analyses within a social phenomenology paradigm to do this [30, 31]. Schutz  proposed social phenomenology as a descriptive, interpretive theory of social action using ‘subjective experience within the “taken-for-granted, commonsense” world of the daily life of individuals’ and takes the view ‘that people living in the world of daily life are able to ascribe meaning to a situation and then make judgments. It is the subjective meaning of experience that was the topic for interpretation in this study’ (cited by Fereday & Muir-Cochrane  (p.81)). By taking this approach, project team members’ broad understanding of aspects of SA AH stroke rehabilitation could be aligned with the intent of international stroke rehabilitation recommendations. These discussions also provided evidence credibility trails [27,28,29,30,31, 33].
For each project question, researcher dyads used an inductive analysis approach and discussed and compared wording, content, intent and meaning of recommendations extracted verbatim from guidance documents. Draft composite recommendations for each question were proposed to the other members of the project team, which discussed and ratified them for relevance to South African contexts.
Determining the OSoBE for composite recommendations
There is an emerging body of methodological guidance about how to develop OSoBE gradings for composite answers distilled from multiple CPG recommendations. To assign OSoBE gradings, the project team drew from
the determination methods of Gonzalez-Suarez et al.  (consistency of thought, volume of evidence, strength of evidence),
the evidence grade elements of the NHMRC strength of the body of evidence matrix  (the evidence base, consistency and impact of recommendations),
the decision-making flowchart proposed by Alper et al.  to reconcile recommendations from a small number of CPGS which address the same question, based on consistency of findings, and reported SoBE gradings  and
A decision-making framework to determine OSoBE was developed and tested (Fig. 2). This included the SoBE grades from the component guidance documents for each question and the number, consistency, quality and currency of these documents (See steps outlined in Additional file 1).
A conceptual framework outlining the project methods is summarised in Fig. 3.
Endorsement of process and recommendations
Regular feedback and workshops supported the development and validation of the evidence-synthesis processes and endorsement of composite recommendations and OSoBE.
Included guidance documents
The search identified 16 guidance documents [37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52] (See Table 3). Each document had been named as a ‘Clinical (Practice) Guideline’, although purpose, scope, questions, methodology, presentation, layout and content differed. All 16 guidance documents were included because they answered at least one project question (See Table 1). The broad interest in AH stroke rehabilitation in upper-middle and high-economic countries was reflected by the volume of documents and countries producing them. Four documents each came from Australia, USA and UK, and one each came from Canada, Malaysia, South Africa and New Zealand. None came from low or lower-middle-income countries.
Questions and composite recommendations
No guidance document answered all project questions, thus, we found no comprehensive existing source of evidence-based guidance for AH rehabilitation that could be immediately applied to guide South African AH stroke rehabilitation in any SA healthcare setting. Overall, the guidance documents addressed all but five project questions.
Methodological quality varied (See Table 3 for overall AGREE II scores). Four guidance documents were of high quality [42, 44, 46, 48], eight were moderate quality [37,38,39, 43, 45, 47, 49,50,51] and three were poor quality [40, 41, 52]. The age of the guidance was not correlated with methodological quality scores (p > 0.05). The AGREE II domain scores are reported in Additional file 1. Considering the domains which provided most relevant information on transferability of guidance documents to the South African AH stroke rehabilitation questions, for scope and purpose, all but one guidance document scored ≥80% (New Zealand  which scored 77.8%). For rigour of development, only three guidance documents did not score highly. These comprised two with poor (≤ 30%) scores (ACSQH , South Africa ), whilst the third (AHA/ASA  scored moderately (72.9%).
Standardising SoBE gradings
As anticipated, the included guidance documents reported different ways of determining and reporting SoBE gradings. Table 4 outlines these and the standardised SoBE approach developed for this project.
An example of a SoBE summary for three project questions (2, 3 and 4) is provided in Fig. 4. This figure includes the initial SoBE grading for recommendations relevant to these questions, the standardised SoBE, and the consistency of the evidence (number and type of ‘faces’). Additional file 1 outlines the steps taken to answer the three project questions (2, 3 and 4).
Most of the composite recommendations were underpinned by moderate to good OSoBE, which provided believable tier 1 evidence to support contextualised implementation of recommendations into SA healthcare settings. Additional file 1 reports the composite recommendations clustered into the patient care decision-making categories (third tier Fig. 1) and the relevant OSoBE.
This is the first research of which we are aware that has developed and tested processes to transparently summarise recommendations from multiple guidance documents, to answer questions relevant to a local context problem. Our project builds on the research reported by Alper  and Shiffman  in terms of combining recommendations from multiple evidence sources, and using strength of wording in recommendations to underpin implementation decisions.
The new methods address the lack of guidance in the literature about how to
standardise differently reported SoBE gradings,
synthesis intent and wording from multiple recommendations that reported the evidence for one question in different ways and
determine an OSoBE grading for a composite recommendation which summarised recommendations from multiple guidance documents, for one question.
The test vehicle for this work was AH stroke rehabilitation delivered in a country (South Africa) that cannot afford the time or finances to develop its own de novo CPG. Moreover, because of the burden that stroke imposes on the SA economy, on communities, families and individuals, AH stroke rehabilitation in SA needs best evidence recommendations to ensure that scant resources are used wisely and equitably [1, 2, 13, 15]. Thus, it was an ideal vehicle to test the new methods.
These new methods add to discussions on methods of CPG repurposing and transferability from one context to another. The methods should assist other South African CPG teams and teams in other resource-constrained organisations or countries, to efficiently contextualise currently available guidance documents for other health conditions. Our work provides a simple, new step-by-step approach that focuses efforts on evidence implementation, rather than on development of de novo CPGs , when there was no urgent need to do so. We believe that the key to the project was the development of a clear and comprehensive patient pathway (pioneered in the Philippines ). This pathway provided a visual overview of the public healthcare options for stroke rehabilitation in SA and provided an important prompt for discussions regarding what actually happened prior to, during and after AH stroke rehabilitation for average and atypical patients with stroke. This assisted the project team to formulate questions. The best practice recommendations now linked to this pathway have the potential to impact on policy, since the National Health Insurance (NHI) of South Africa white paper  strongly supports the use of local CPGs to guide the delivery of evidence-informed and cost-effective health services in SA.
Project team members with little background in methodology found the step-by-step process simple to understand, and they required little further explanation than was provided in the figures reported in this paper. The simplicity of the process meant that future work should be able to be undertaken with minimal training by non-academics (clinicians, managers, policy-makers) or even students. Those managers and clinicians on the project team who were less focused on methodology, and more interested in recommendation relevance and implementation in local settings, validated the usefulness of the wording and intent of the composite recommendations, as well as the way the OSoBE had been developed. They indicated that the recommendations aligned with their understanding of current best practice undertaken elsewhere in the world. The policy-makers expressed confidence that the composite SA–relevant recommendations positioned them to progress evidence-informed discussions on future funding and policy priorities. Consumers considered that the process encouraged engagement of consumers on future guidance teams. The recommendations would also improve SA patient understanding of rehabilitation, as well as shared decision-making.
Integrating existing recommendations was far more efficient than starting ‘from scratch’ with a de novo CPG oriented to South African contexts [25, 26]. This project was conducted with minimum of external funding or reliance on individual ‘out-of-hours’ commitment. Project team members’ employers largely agreed to commitment to this project within work hours. The team members came from different parts of South Africa, thus, it took time and resources to organise face-to-face meetings. Much contact occurred electronically, when versions of recommendations were being discussed. Development of the process and constructing the new evidence base from 16 guidance documents took approximately 3 months intensive work by the research team, compared to 12–18 months to produce a de novo CPG. There were four team meetings (1 day each), and the remainder of the project time was spent in gaining agreement and endorsement of recommendations using electronic means. It is anticipated that future applications of the new methods to produce composite guidance for other health conditions in other settings would be quicker, particularly if there are fewer relevant guidance documents to summarise.
The need for this project was highlighted as there was no one international CPG which answered all the project questions. Moreover, the project questions contained few clinical ones (‘what to do’) and mostly focused on implementation of evidence locally (‘who’, ‘how’, ‘when’, ‘where’, ‘why’ and ‘how much’). This validated the theoretical model proposed by Project SAGE , of using three CPG tiers when constructing local guidance. The use of existing ‘what to do’ recommendations extracted from existing guidance documents provided the current best evidence base (tier 1) for AH stroke rehabilitation potentially relevant to South African public healthcare sectors. What is more, the extracted information supported effective and efficient tier 2 discussions on local service delivery implementation issues. The majority of included guidance documents were well constructed (moderate to good AGREE II scores). Their scope and purpose correlated with at least some of the project questions, making these guidance documents relevant evidence sources. Useful tier 3 documents were also identified from the included guidance documents, obviating the need to reproduce these locally. Whilst the SoBE gradings varied between guidance documents for the recommendations pertinent to the project questions, the evidence was mostly consistent in terms of consistency of direction, intent and wording.
The importance of including the multiple components in the OSBoE decision-making framework was highlighted by the variability in the number and currency of included guidance documents providing answers to project questions. The differing SoBE gradings for extracted recommendations relevant to a project question mostly related to the underpinning research design(s) which formed the evidence base. This reflects a fundamental issue with determining appropriate SoBE gradings related to AH stroke rehabilitation. The notion of SoBE has its genesis in intervention studies, largely delivered using a medical model [25, 26, 35, 36, 54, 55]. High SoBE gradings generally come from prospective comparative studies with control arms and blinding (experimental trials, diagnostic studies etc.). High quality intervention studies are often difficult to design for AH stroke rehabilitation because lack of homogeneity in stroke aetiology, affected site and stroke density, access to subjects, individuals’ responses to brain insult, attitudes to therapy, service delivery differences, family and community support, potential for quality life after stroke etc. [12, 15, 55]. Moreover, service quality research questions are mostly answered by cross-sectional or retrospective designs which usually graded as having moderate to high risk of bias and, therefore, lower SoBE grades [35, 36, 55]. In SA, where stroke service delivery can differ markedly across public health service sectors, and in urban, regional and remote areas, there is no one clear patient care pathway for many stroke sufferers from acute care to long-term rehabilitation and community integration. This was why the discussions on the patient pathway were so valuable at project commencement (See Fig. 1).
The applicability of our novel method requires further testing by others, in other healthcare settings, for stroke and other health conditions. The proposed methods to combine recommendations and SoBE gradings need validation in different types of research evidence. However, critical to update of this method is the issue of transferability of recommendations from existing CPGs to new environments. This must be guided by tier 2 (local expert input). Evidence that has been developed on one population may not be immediately transferrable to other populations, even if patients come from similar socioeconomic environments [24, 28, 29]. Implementation-focused questions need to be asked such as ‘Can this recommendation be put into place for our patients, in our contexts’, and if the answer is no, then the reasons for this need to be explored in terms of local barriers and facilitators. Getting the right tier 2 representatives together and engaging them in discussion relevance of recommendations appears to be key to moving from evidence-based recommendations to implementation-ready recommendations.
The limitations were that a new process was developed and tested on one set of questions (allied health stroke rehabilitation). This condition provided a vehicle. The project team was learning as it went, and errors and mis-interpretations were sure to have been made, particularly in the formulation of composite recommendations from the included guidance documents. The process of combining evidence sources into one composite set of recommendations needs to be tested by other teams for other conditions.
Our practical innovative methods add to the scarce body of evidence in repurposing and transferring CPGs from one etting to another. Given the urgent need to implement evidence in resource-constrained settings to reduce alarming disease burdens, the costs and time to develop de novo CPGs, and the untenable burden the lack of appropriate evidence places on many low- and lower-middle economy countries, our methods provide a feasible, efficient process to repurpose and transfer CPGs, rather than produce new ones. The SA AH stroke rehabilitation recommendations developed using this new method will ensure that appropriate treatments are offered at appropriate times, in appropriate ways, relevant to local contexts, to improve stroke survivors’ access to ongoing care and quality post-stroke lives.
Clinical practice guidelines
Overall strength of the body of evidence
South African contextualised stroke rehabilitation guidelines
Strength of the body of evidence
World Health Organization
Coovadia H, Jewkes R, Barron P, Sanders D, McIntyre D. The health and health system of South Africa: historical roots of current public health challenges. Lancet. 2009;374(9692):817–34.
Mash B, Fairall L, Adejayan O, Ikpefan O, Kumari J, Mathee S, Yogolelo W. A morbidity survey of south African primary care. PLoS One. 2012;7(3):e32358.
WHO. Global Disability Action Plan 2014–2021: Better health for all people with disability. Geneva: World Health Organization; 2015.
Sharma R. Breakout nations: in pursuit of the Next Economic Miracles. New York: WW Norton & Company; 2012.
Taba P, Rosenthal M, Habicht J, Tarien H, Mathiesen M, Hill S, Bero L. Barriers and facilitators to the implementation of clinical practice guidelines: a cross-sectional survey among physicians in Estonia. BMC Health Serv Res. 2012;12:455. https://doi.org/10.1186/1472-6963-12-455.
Spencer LM, Schooley MW, Anderson LA, Kochtitzky CS, DeGroff AS, Devlin HM, Mercer SL. Seeking best practices: a conceptual framework for planning and improving evidence-based practices. Prev Chronic Dis. 2013;10:130186.
Weng Y-H, Kuo KN, Yang C-Y, Lo H-L, Chen C, Chiu Y-W. Implementation of evidence-based practice across medical, nursing, pharmacological and allied healthcare professionals: a questionnaire survey in nationwide hospital settings. Implement Sci. 2013;8:112.
Abrahamson K, Fox R, Doebbeling B. Facilitators and barriers to clinical practice guideline use among nurses. Am J Nurs. 2012;112(7):26–35.
Ploeg J, Davies B, Edwards N, Gifford W, Miller PE. Factors influencing best-practice guideline implementation: lessons learned from administrators, nursing staff, and project leaders. Worldviews Evid Based Nurs. 2007;4:210–9.
Orem JN, Mafigiri DK, Marchal B, Ssengooba F, Macq J, Criel B. Research, evidence and policymaking: the perspectives of policy actors on improving uptake of evidence in health policy development and implementation in Uganda. BMC Public Health. 2012;12:109.
Goyet S, Touch S, Ir P, Saman S, Fassier T, Frutos R, Tarantola A, Barennes H. Gaps between research and public health priorities in low income countries: evidence from a systematic literature review focused on Cambodia. Implement Sci. 2015;10:32.
The impact of allied health professionals in improving outcomes and reducing the cost of treating diabetes, osteoarthritis and stroke. Services for Australian Rural and Remote Allied Health (SARAH). Canberra, Australia. 2015. https://sarrah.org.au/publication/report-economic-impact-allied-health-interventions. Accessed 11 Aug 2018.
Dizon J, Grimmer K, Louw Q, Machingaidze S, Parker H, Pillen H. South African allied health clinical practice guidelines: barriers and facilitators. Health Res Policy Systems. 2017;15(1):79.
Norrving B, Kissela B. The global burden of stroke and need for a continuum of care. Neurology. 2013;80(3 Suppl 2):S5–12. https://doi.org/10.1212/WNL.0b013e3182762397.
Health24. Salt is killing South Africans. The Heart and Stroke Foundation. 2015. https://www.health24.com/Diet-and-nutrition/News/Salt-is-killing-South-Africans-20130312. Accessed 11 Aug 2018.
Machingaidze S, Zani B, Abrams A, Durao S, Louw Q, Kredo T, Grimmer K, Young T. Quality and reporting standards of south African primary care clinical practice guidelines. J Clin Epi. 2017;83:31–6. https://doi.org/10.1016/j.jclinepi.2016.09.015.
Yaolong C, Yang K, Marušić A, Qaseem A, Meerpohl JJ, Flottorp S, Akl EA, Schünemann HJ, ESY C, Falck-Ytter Y, Ahmed F, Barber S, Chen C, Zhang M, Xu B, Tian J, Song F, Shang H, Tang K, Wang Q, Norris SL, for the RIGHT (Reporting Items for Practice Guidelines in Healthcare) Working Group. A reporting tool for practice guidelines in health care: the RIGHT Statement. Ann Intern Med. 2017;166:128–32. https://doi.org/10.7326/M16-1565.
WHO Health Service Delivery. 2010. http://www.who.int/healthinfo/systems/WHO_MBHSS_2010_section1_web.pdf. Accessed 12 Aug 2018.
Kumar S, Young A, Lizarondo LM. What’s in a name? Current case of nomenclature confusion. In: Grimmer-Somers K, Worley A, editors. Practical tips for using and developing guidelines. Manila: UST Publishing House; 2010. ISBN 978-971-506-559-7.
Machingaidze S, Grimmer K, Louw Q, Kredo T, Volmink J, Young T. Next generation clinical guidance for primary care in South Africa – credible, consistent and reliable. PLOS One. 2017; in press
Dizon JMR, Machingaidze SG, Grimmer KA. To adopt, adapt or contextualise? That is the question. BMC Res Notes. 2016;9:442.
AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003;12:18–23.
Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau F, et al. AGREE II: advancing guideline development, reporting and evaluation in healthcare. Prev Med. 2010;51(5):421–4.
Gonzalez-Suarez C, Grimmer-Somers K, Dizon J, King E, Lorenzo S, Valdecanas C, Fidel B. Contextualising western guidelines for stroke and low back pain to a developing country (Philippines): an innovative approach to putting evidence into practice efficiently. J Healthcare Leadership. 2012;4:141–56.
Hillier S, Grimmer-Somers K, Merlin T, Middleton P, Salisbury J, Tooher R, Weston A. FORM: an Australian method for formulating and grading recommendations in evidence-based clinical guidelines. BMC Med Res Methodol 2011; 11: 23. doi: https://doi.org/10.1186/1471-2288-11-23.
Schünemann HJ, Wiercioch W, Etxeandia I, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CAMJ. 2014;186(3):E123–42.
Berkman ND, Lohr KN, Ansari MT, Balk EM, Kane R, McDonagh M, Morton SC, Viswanathan M, Bass EB, Butler M, Gartlehner G, Hartling L, McPheeters M, Morgan LC, Reston J, Sista P, Whitlock E, Chang S. Grading the strength of a body of evidence when assessing health care interventions: an EPC update. J Clin Epidemiol. 2015;68(11):1312–24. https://doi.org/10.1016/j.jclinepi.2014.11.023. Epub 2014 Dec 20
Alper B, Price A, Kunnamo I, Qaseem A, Vandvik P, Elwyn G, Oettgen P. Classification of consistency across guidelines: a model for informing patients about global guidance. Session 7. Global Evidence Summit. Cape Town; 2017. https://www.globalevidencesummit.org/special-sessions. Accessed 12 Aug 2018.
Shiffman RN, Michel G, Rosenfeld RM, Davidson C. Building better guidelines with BRIDGE-wiz: development and evaluation of a software assistant to promote clarity, transparency, and implementability. J Am Med Inform Assoc. 2012;19(1):94–101. https://doi.org/10.1136/amiajnl-2011-000172.
Schutz A. The phenomenology of the social world (Walsh G., Lehnert F., trans.). Evanston: North Western University Press; 1967. (Original German work published 1932)
Patton MQ. Qualitative research and evaluation methods. 3rd ed. Thousand Oaks: Sage Publications; 2002.
Fereday J, Muir-Cochrane E. Demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development. Int J Qual Methods. 2006;5(1):80.
Horsfall D, Byrne-Armstrong H, Higgs J. Researching critical moments. In: Byrne-Armstrong H, Higgs J, Horsfall D, editors. Critical moments in qualitative research. Oxford: Butterworth Heinemann; 2001. p. 3–16.
Healthcare GPS. Recommendation Classification. https://www.healthcaregps.org/recommendationclassification. Accessed 12 Aug 2018.
Guyatt GH, Oxman AD, Kunz R, Falck-Ytter Y, Vist GE, Liberati A, Schünemann HJ. GRADE: going from evidence to recommendations. BMJ. 2008a;336:1049–51.
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Schünemann HJ. GRADE: what is “quality of evidence” and why is it important to clinicians? BMJ. 2008b;336:995–8.
Winstein CJ, Stein J, Arena R, Bates B, Cherney LR, Cramer SC, Deruyter F et al on behalf of the American Heart Association stroke council, council on cardiovascular and stroke nursing, council on clinical cardiology, and council on quality of care and outcomes research (AHA/ ASA CPG). Stroke 2016;47:31–371. DOI: https://doi.org/10.1161/STR.0000000000000098.
American Occupational Therapy Association. Occupational therapy practice guidelines for adults with stroke, 2008 (revised 2015). https://www.guidelinecentral.com/share/summary/56531e59a35ad#sectionsociety. Accessed 12 Aug 2018.
Agency for Healthcare Research and Quality. Stroke CPG. https://www.ahrq.gov/gam/summaries/index.html. Accessed 12 Aug 2018.
Australian Commission on Safety and Quality in Health Care (ACSQHC) Stroke CPG. Evidence Sources: Acute Stroke Clinical Care Standard. Sydney: ACSQHC; 2015. ISBN 978–1–925224-16-0.
Australian Stroke Foundation. Clinical Guidelines for Stroke Management 2017. https://informme.org.au/Guidelines/Clinical-Guidelines-for-Stroke-Management-2017. Accessed 12 Aug 2018.
Hebert D, Lindsay MP, McIntyre A, Kirton A, Rumney PG, Bagg S, et al. Canadian stroke best practice recommendations: stroke rehabilitation practice guidelines, update 2015. Int J Stroke. 2016;11(4):459–84.
Veterans Association, Department of Defence (VA/DoD). VA/DoD Clinical Practice Guideline for the of Stroke Rehabilitation Version 3.0. Washington, DC: Department of Defense; 2010. https://www.rehab.research.va.gov/jour/10/479/pdf/vadodcliniaclguidlines479.pdf. Accessed 12 Aug 2018.
Malaysian Society of Nerosciences. Clinical Practice Guidelines: Management of Ischaemic Stroke (2nd Edition), 2012. Academy of Medicine Malaysia. http://www.acadmed.org.my/index.cfm?&menuid=67. Accessed 12 Aug 2018.
Stroke Foundation of New Zealand (SFNZ) and New Zealand Guidelines Group (NZGG). Clinical Guidelines for Stroke Management 2010. Wellington: Stroke Foundation of New Zealand; 2010. ISBN (Electronic): 978–0–9582619-6-8.
NICE. Stroke rehabilitation in adults. Clinical Guideline 162. London, UK: National Institute for Health and Care Excellence; 2013. https://www.nice.org.uk/guidance/cg162. Accessed 12 Aug 2018.
New South Wales Agency for Clinical Innovation (ACI). NSW Clinical Care Guide for the management of acute stroke 2016. https://www.aci.health.nsw.gov.au/networks/stroke/resources. Accessed 12 Aug 2018.
Intercollegiate Stroke Working Party, Royal College of Physicians (RCP). National Clinical Guideline for Stroke, 2016. https://www.rcplondon.ac.uk/guidelines-policy/stroke-guidelines. Accessed 12 Aug 2018.
South Australian Statewide Stroke Clinical Network. Clinical Guideline - Stroke Management Procedures & Protocols. Adelaide: South Australian Dept Health; 2014.
SIGN. Management of patients with stroke: identification and management of dysphagia. A national clinical guideline. Edinburgh, Scotland: Scottish Intercollegiate Guidelines Network; 2010. http://www.sign.ac.uk/assets/sign119.pdf. Accessed 12 Aug 2018.
SIGN. Management of patients with stroke: rehabilitation, prevention and management of complications, and discharge planning. A national clinical guideline. Edinburgh, Scotland: Scottish Intercollegiate Guidelines Network; 2010. http://www.sign.ac.uk/assets/sign118.pdf. Accessed 12 Aug 2018.
Bryer A, Connor MD, Haug P, Cheyip B, Staub H, Tipping B, Duim W, Pinkney-Atkinson V. South African guideline for management of ischaemic stroke and transient ischaemic attack 2010: A guideline from the South African Stroke Society (SASS) and the SASS Writing Committee. S Afr Med J. 2010;100(11):750–78.
Department of Health. National Health Insurance for South Africa: Towards universal coverage. Pretoria: Department of Health, Republic of South Africa; 2017. http://www.health.gov.za/index.php/component/phocadownload/category/383. Accessed 12 Aug 2018.
Graham ID, Harrison MB, Brouwers M, Davies BL, Dunn S. Facilitating the use of evidence in practice: evaluating and adapting clinical practice guidelines for local use by healthcare organizations. JOGNN. 2002;3:599–611.
Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Savović J, Schulz KF, Weeks L, Sterne JAC. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. https://doi.org/10.1136/bmj.d5928.
Key contributors: Maluta Tshivhase, Elma Burger; Tina Ointo, Sameera Haffejee, Juliana Freeme, George Scola, Bhavika Chhania, Bianca Knoetze Dietition, Witness Mudzi, Veronica Ntsiea, Pauline Ramushu, Frida Kotsokoane, Kganetso Sekome, Caitlin von Berg, Marlie Enright, Caroline De Wet, Jamie de Grass-Clementser, Anthea Rhoda, Daleen Campher, Metilda Lewis, Jinnae Kleinsmit, Loreta Krige, Carolyn Davids, Maatje Kloppers, Rochelle Felix, Ivy Kekana and Maryke Bezuidenhout.
The authors were funded partially by Stellenbosch University with support from the WHO Alliance for Health Policy and Systems Research.
Availability of data and materials
The dataset(s) supporting the conclusions of this article is (are) available at the Stellenbosch University.
Ethics approval and consent to participate
Ethical approval was provided by Stellenbosch University (#0642). Written consent forms were obtained from the participants in this study.
Consent for publication
The authors declare they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Grimmer, K., Louw, Q., Dizon, J.M. et al. Standardising evidence strength grading for recommendations from multiple clinical practice guidelines: a South African case study. Implementation Sci 13, 117 (2018). https://doi.org/10.1186/s13012-018-0803-0
- Clinical practice guidelines
- Guidance documents
- Allied health stroke rehabilitation
- Resource poor environments
- South Africa