Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Expanding the evidence base for global recommendations on health systems: strengths and challenges of the OptimizeMNH guidance process

Implementation Science201611:98

https://doi.org/10.1186/s13012-016-0470-y

Received: 3 February 2016

Accepted: 9 July 2016

Published: 18 July 2016

Abstract

Background

In 2012, the World Health Organization (WHO) published recommendations on the use of optimization or “task-shifting” strategies for key, effective maternal and newborn interventions (the OptimizeMNH guidance). When making recommendations about complex health system interventions such as task-shifting, information about the feasibility and acceptability of interventions can be as important as information about their effectiveness. However, these issues are usually not addressed with the same rigour. This paper describes our use of several innovative strategies to broaden the range of evidence used to develop the OptimizeMNH guidance. In this guidance, we systematically included evidence regarding the acceptability and feasibility of relevant task-shifting interventions, primarily using qualitative evidence syntheses and multi-country case study syntheses; we used an approach to assess confidence in findings from qualitative evidence syntheses (the Grading of Recommendations, Assessment, Development and Evaluation-Confidence in Evidence from Reviews of Qualitative Research (GRADE-CERQual) approach); we used a structured evidence-to-decision framework for health systems (the DECIDE framework) to help the guidance panel members move from the different types of evidence to recommendations.

Results

The systematic inclusion of a broader range of evidence, and the use of new guideline development tools, had a number of impacts. Firstly, this broader range of evidence provided relevant information about the feasibility and acceptability of interventions considered in the guidance as well as information about key implementation considerations. However, inclusion of this evidence required more time, resources and skills. Secondly, the GRADE-CERQual approach provided a method for indicating to panel members how much confidence they should place in the findings from the qualitative evidence syntheses and so helped panel members to use this qualitative evidence appropriately. Thirdly, the DECIDE framework gave us a structured format in which we could present a large and complex body of evidence to panel members and end users. The framework also prompted the panel to justify their recommendations, giving end users a record of how these decisions were made.

Conclusions

By expanding the range of evidence assessed in a guideline process, we increase the amount of time and resources required. Nevertheless, the WHO has assessed the outputs of this process to be valuable and is currently repeating the approach used in OptimizeMNH in other guidance processes.

Keywords

Health systems Guidelines Guidance Qualitative research Systematic reviews GRADE GRADE-CERQual Decision-making

Background

In 2012, the World Health Organization (WHO) published evidence-based recommendations regarding the use of optimization or “task-shifting” strategies for key, effective maternal and newborn interventions (OptimizeMNH) [1]. Typically, task-shifting involves training lower level health worker cadres to deliver tasks that are normally delivered by higher level health worker cadres. The main aim of this strategy is to increase access to care for people living in areas with a shortage of health workers.

While task-shifting strategies have the potential to increase access to care in settings with critical shortages of specialised health workers, there is also concern that the delivery of health care by health workers with less training could cause harm to people already in vulnerable situations. The effectiveness and safety of such interventions therefore needs to be rigorously evaluated. However, task-shifting strategies that have been assessed as effective and safe in the context of experimental studies may be challenging to implement at scale within a health system. The shifting of tasks from one group of health workers to another can involve social, behavioural and organisational change and, like other complex interventions, its success may depend not only on characteristics of the intervention itself but also on characteristics of the systems in which the intervention is implemented or on interactions between properties of the intervention and systems [24]. The success of task-shifting strategies is also likely to be influenced by factors such as the training, supervision and support provided to health workers as well as access to supplies and referral systems. The culture and organisation of healthcare delivery in a particular setting, as well as views and expectations among stakeholders such as service users, the providers involved in task-shifting and managers, may also play important roles in the success of these types of strategies [57]. Furthermore, the shifting of tasks can have serious implications for health worker motivation, for instance because of changes in working conditions and salaries, and for health service users’ experience of care [8, 9].

As these issues can represent important challenges to successful implementation, information about an intervention’s feasibility and acceptability can therefore be as important as information regarding intervention effectiveness and safety when recommendations are made. Initiatives that aim to improve the quality of guidelines, such as the AGREE II instrument, also emphasise the importance of seeking stakeholders’ views and preferences, and of discussing potential organisational barriers to the application of recommendations [10]. But despite their potential importance, questions about acceptability and feasibility are usually not addressed with the same rigour as questions about intervention effectiveness. Instead, it appears that these issues are often left to the opinions and experiences of guideline panel members. While this input is useful as part of a dialogue [11], it is limited by the composition of the group. For global guidelines such as OptimizeMNH, the input of panel members from specific geographic or clinical settings and with experience of particular population groups may not give sufficient insight into the values and experiences of service users and other stakeholders more broadly. In addition, the perspectives of some panel members may take precedence over other members because of perceived differences in social status within the guideline panel [12, 13].

As the technical team responsible for developing the OptimizeMNH guidance (Table 1), we saw a need to address these issues in the evidence-to-decision process. In this paper, we describe our use of innovative strategies to expand the evidence used as a basis for WHO recommendations and discuss our experiences with these strategies.
Table 1

Defining guidance

Guidance in the context of health systems can be defined as “systematically developed statements produced at global or national levels to assist decisions about appropriate options for addressing a health systems challenge in a range of settings and to assist with the implementation of these options and their monitoring and evaluation” [49].

The term “guidance” can be seen as preferable to the more prescriptive term “guidelines” as health systems, public health and other complex interventions, and the evidence on these issues are often very context sensitive. Health systems guidance statements (which are also sometimes referred to as recommendations) would usually include policy options accompanied by assessments of the quality of evidence supporting them, including evidence of unintended consequences and of acceptability and feasibility issues and resource use. Guidance statements may also be accompanied by discussions of implementation and contextual issues.

Adapted from [49]

Methods

In developing the OptimizeMNH guidance, we followed the standard procedures set out in the WHO’s Handbook for Guideline Development [14]. These procedures (which include the formulation of questions and outcomes; the retrieval, assessment and synthesis of evidence; the development and formulation of recommendations; and finally, the dissemination and implementation of these recommendations) are intended to ensure that WHO recommendations address key needs and are evidence-based [15]. However, we also elaborated on these standard procedures by adopting innovative approaches that could help us address our goal of producing evidence-informed recommendations that are informed by a broader set of evidence than data on effectiveness alone. These approaches were used at different stages of guidance development: when gathering the evidence to support the guidance, when assessing and synthesising this evidence, and when developing the recommendations (Table 2). The approaches included the following:
Table 2

Innovative strategies and resources available to guidance developers

Stage of the guidance

Innovative approaches used in the OptimizeMNH guidance

Areas of use

Resources available to guideline organisations interested in using similar approaches

Gathering the evidence

Systematic reviews of qualitative research (also referred to as qualitative evidence syntheses)

Can be used to:

 - Gather evidence about stakeholders’ views and preferences, for instance, on which outcomes that stakeholders value the most in relation to the guidance question/s

 - Gather evidence about the acceptability and feasibility of interventions

 - Offer information about implementation considerations

- WHO Handbook for guideline development, chapter 15 on using evidence from qualitative research to develop WHO guidelines: http://www.who.int/kms/guidelines_review_committee/en/

- Developing NICE guidelines: the manual: https://www.nice.org.uk/article/pmg20/chapter/1-Introduction-and-overview

- Website of the Cochrane Qualitative and Implementation Methods Group, which includes guidance on conducting qualitative evidence syntheses: http://methods.cochrane.org/qi/supplemental-handbook-guidance

- Texts on conducting qualitative evidence syntheses: [33, 35]

Multi-country case study syntheses

Can be used to gather evidence about the issues mentioned above and may be particularly useful where reviews of qualitative research do not cover sufficiently macro-level issues

- Texts on conducting case study syntheses: [22, 42, 50]

Assessing and synthesising the evidence

GRADE-CERQual (“Confidence in Evidence from Reviews of Qualitative Research”) approach

Used to transparently assess and describe how much confidence to place in findings from qualitative evidence syntheses

- Website with information about the GRADE-CERQual approach: www.cerqual.org

- Text describing the GRADE-CERQual approach: [26]

Developing the recommendations

DECIDE evidence-to-decision framework

A structured health system framework to help guidance panel members move from evidence to recommendations. For each guidance question, the framework presented, in a structured format, a summary of the evidence regarding:

 • The benefits and harms of the intervention

 • Anticipated resource use

 • Acceptability of the intervention, i.e. the extent to which that intervention is considered to be reasonable among those receiving, delivering or affected by the intervention

 • Feasibility of the intervention, i.e. the likelihood that the intervention can be properly carried out or implemented in a given context

- Website: https://ietd.epistemonikos.org/#/login

- Texts describing the DECIDE framework: [30, 31]

  • Broadening the range of evidence used in the guidance by including systematic reviews of qualitative research, multi-country case study syntheses and results from a primary qualitative study

  • Using the Grading of Recommendations, Assessment, Development and Evaluation-Confidence in Evidence from Reviews of Qualitative Research (GRADE-CERQual) approach to assess our confidence in the evidence from systematic reviews of qualitative research

  • Using a structured health systems framework (the DECIDE framework) to help the guidance panel move from this wider range of evidence to recommendations

Results

Gathering the evidence—broadening the range of evidence included in the guidance

The WHO’s Handbook for Guideline Development describes not only several types of questions that can legitimately be addressed in a WHO guideline, including the effectiveness or efficacy of an intervention, but also the negative consequences of the intervention, its social acceptability and its cost-effectiveness. The Handbook describes how qualitative evidence synthesis can be used to explore contextual barriers as well as values and preferences [14]. The main focus of the Handbook is, however, on the use of evidence about effectiveness and, in practice, most WHO guidelines have focused almost exclusively on this question.

For OptimizeMNH, we systematically assessed the effectiveness and safety of the relevant task-shifting interventions through systematic reviews and the use of GRADE. In addition, we used an innovative approach in this stage of guidance development by systematically including other types of evidence. Specifically, we compiled evidence on the acceptability of the interventions among health workers, recipients of care, policy makers and other stakeholders and on the feasibility of these interventions, taking into account the organisational changes to health systems that each task-shifting intervention would entail.

While systematic reviews of randomised trials are seen as the most robust source of evidence when assessing intervention effectiveness, this approach is not well-suited for exploring issues surrounding intervention acceptability and feasibility, particularly for more complex health and social interventions [2, 16]. To answer these questions, we included systematic reviews of qualitative studies (sometimes called qualitative evidence syntheses) as our primary source of evidence [17]. Together with colleagues, we co-authored four qualitative evidence syntheses focusing on factors affecting the implementation of task-shifting among lay health workers, midwives, doctors and nurses [1821].

The qualitative evidence syntheses provided us with information that was clearly relevant to the questions we had asked. However, most of the studies were of relatively small-scale or pilot programmes, often implemented in the context of research, and focused on factors unfolding at the level of programme delivery in communities and primary care facilities. We decided therefore to complement this evidence with two multi-country case study syntheses focusing on factors affecting the implementation of large-scale programmes. When selecting these programmes, we aimed for broad geographical representation and looked for countries in Africa, Asia and South America that had implemented large-scale programmes. To make the synthesis manageable, we limited ourselves to seven programmes and only selected programmes where a reasonable amount of English-language documentation was available. The multi-country case study syntheses aimed to identify “upstream”, system-level factors associated with programme policies, governance, financing, planning, management and organisation [1, 22]. These case studies synthesised evidence from a variety of sources, including peer-reviewed qualitative and quantitative studies, programme reports, web sites and information from personal communication with individuals familiar with the selected task-shifting programmes.

Another limitation of the evidence we identified through the qualitative evidence syntheses was that it focused primarily on the views and experiences of service users and healthcare providers but paid less attention to the viewpoints of programme managers, policy makers and other stakeholders. We attempted to address this imbalance by carrying out our own primary study. We undertook qualitative analyses of the views and experiences of those contributing to the “Health Information For All By 2015” electronic discussion list, which covers a diverse membership of programme managers, healthcare providers, policy makers, academics and others from 170 countries [23]. These analyses evaluated opinions about how the roles of healthcare providers could be optimised to improve maternal and newborn health in LMICs and the implications of such role optimization [1, 24].

When presenting the evidence from the qualitative evidence syntheses, the country case studies and the primary qualitative studies, we attempted to extract findings that were relevant to specific recommendations being considered by the guidance panel and that concerned specific tasks and groups of health workers. These findings were used to (a) support the guidance panel when reaching recommendations and (b) develop information for end users of the guidance regarding implementation considerations for each recommendation. Tables 3 and 4 provide examples of how these findings informed the development of recommendations in the OptimizeMNH guidance.
Table 3

Example of how the qualitative evidence informed the final recommendations in the OptimizeMNH guidance (1): provision of continuous support during labour by lay health workers, in the presence of a skilled birth attendant

The guidance panel was asked to consider whether lay health workers could provide continuous support, such as emotional and practical support, during labour, while in the presence of a skilled birth attendant providing the necessary clinical care.

Information regarding benefits and harms came from a systematic review of trials [51]. This evidence suggested that this support, when provided by lay health workers or other birth supporters, may have important health benefits (low to moderate certainty evidence). Based on discussion in the technical team, we also concluded that the intervention would require little additional training, supervision and supplies.

Information regarding acceptability and feasibility came from two syntheses of qualitative evidence [19, 20]. The evidence suggested that mothers appreciated this support from lay health workers and that health professionals working alongside lay health workers often appreciated their contribution to their busy workload and their skills in communicating with mothers (moderate confidence in the evidence). However, the relationships between lay health workers or other birth supporters and professional midwives was sometimes ambivalent and, at times, conflictual, possibly because the midwife role was shifted in a more medical direction (moderate confidence in the evidence). Having to be present during labour and birth could also lead to irregular and unpredictable working conditions for the lay health worker, which might have implications for their expectations regarding incentives (low confidence in the evidence) and could lead to concerns about personal safety when working in the community or travelling at night (low confidence in the evidence).

This information was presented to the guidance panel in a summarised form using the DECIDE evidence to decision framework. More detailed versions were presented in appendices using summary of findings tables and full versions of each review were also made available to the panel.

Based on this evidence, the panel decided to recommend the intervention. Potential challenges regarding the acceptability of the intervention to lay health workers and other healthcare providers were highlighted under “Implementation considerations”.

Adapted from [1]

Table 4

Example of how the qualitative evidence informed the final recommendations in the OptimizeMNH guidance (2): provision of vasectomy by trained midwives

The guidance panel was asked to consider whether midwives could perform vasectomies.

We were unable to identify any eligible studies that assessed the benefits or harms of midwives performing vasectomies. We did have indirect evidence from one systematic review of trials [52] that there may be little or no difference between midwives and doctors with regard to complications during surgery or postoperative morbidity for tubal ligation (low quality evidence).

Based on discussion in the technical team, we concluded that the intervention would require additional training, supervision and supplies and a functioning referral system for failed vasectomies or complications and might also require changes to norms and regulations.

Information regarding acceptability and feasibility came from one synthesis of qualitative evidence [20]. The synthesis did not identify any studies that evaluated the acceptability of vasectomy when performed by midwives. For other midwife-delivered interventions, the synthesis suggested that midwives and their trainers generally felt that midwives had no problem learning new clinical techniques (moderate confidence in the evidence) and might be motivated by being “upskilled” as this could lead to increased status, promotion opportunities and increased job satisfaction (moderate confidence in the evidence). However, midwives might be unwilling to take on tasks that moved beyond obstetric care, such as tasks related to family planning, possibly because this was not viewed as part of their role and might entail an increased workload (moderate confidence in the evidence). In addition, a lack of clarity in roles and responsibilities between midwives and other healthcare providers, as well as status and power differences, might also lead to poor working relationships and “turf battles” (moderate confidence in the evidence). Finally, the synthesis suggested that additional training and supervision were often insufficient in midwife task-shifting programmes.

This information was presented to the guidance panel in a summarised form using the DECIDE framework. More detailed versions were presented in appendices using summary of findings tables and full versions of each review were also made available to the panel.

Based on this evidence, the panel decided to recommend the intervention only in the context of rigorous research. The panel further specified that implementation in the context of research should only be done where a well-functioning midwife programme already exists and a well-functioning referral system is in place or can be put in place.

Adapted from [1]

We also wanted to offer those using the OptimizeMNH guidance more general information about the acceptability and feasibility of task-shifting strategies. We therefore carried out a “cross-cutting” analysis of findings from all of the sources described above. This cross-cutting analysis provided evidence regarding task-shifting in general where no direct evidence was available for specific task-shifting interventions. The cross-cutting analysis also contributed to a chapter in the WHO OptimizeMNH guidance on overarching implementation considerations.

Assessing and synthesising the evidence—introducing an approach to assess our confidence in findings from qualitative evidence syntheses

The WHO Guideline Handbook notes that guideline technical teams need to provide an assessment of the quality of the evidence included in a WHO guideline [14]. For evidence of effectiveness, the WHO uses the GRADE approach to assess quality [25]. GRADE is not, however, designed to be applied to qualitative research, and we were not aware of other systems for assessing quality or confidence in findings from qualitative evidence syntheses or for indicating these assessments to end users. Having made the decision to include qualitative evidence syntheses, we therefore needed an approach in transparently assessing and describing how much confidence to place in findings from these types of syntheses.

Work to develop this approach was carried out in collaboration with other qualitative researchers [26], informed by the principles of qualitative research. We were also influenced by the processes used by the GRADE Working Group, which has given considerable thought to how to assess confidence in evidence from systematic reviews. This work resulted in an approach which we named GRADE-CERQual [26]. Since its initial use in the OptimizeMNH guidance, CERQual is being developed further as part of the GRADE Working Group and is now being used in several other WHO guidance development processes [27].

Developing the recommendations: using a structured health systems framework

When assessing the different factors that influence recommendations, including effectiveness, acceptability, feasibility, and cost, the Guideline Handbook [14] suggests that the guideline panel makes use of evidence-to-recommendation tables. These tables can be used to lay out what we know about these different factors and can also be used to record the guideline panel’s judgements about these factors and how they contributed to the development of the recommendation [14]. For OptimizeMNH, we piloted an early paper version of a decision table developed by the DECIDE project [28] and the Grade Working Group [29]: the DECIDE Evidence to Decision framework [30, 31].

The DECIDE framework aims to help guidance panel members move from evidence to health system recommendations by informing judgements about the balance of consequences of each option. The framework is based on a review of relevant literature, brainstorming, feedback from stakeholders [32], and the application of the framework to examples.

For each guidance question, the framework presented, in a structured format, a summary of the evidence regarding the following considerations:
  • The benefits and harms of the intervention/s (sometimes referred to as “effectiveness and safety”)

  • Anticipated resource use

  • Acceptability of the intervention/s

  • Feasibility of the intervention/s

For each of these elements, the framework also included an assessment of the certainty of the evidence, using GRADE (for evidence of effectiveness) and CERQual (for evidence on acceptability and feasibility from qualitative evidence syntheses). The following were also included for each guidance question:
  • A judgement regarding the balance of desirable and undesirable consequences for the intervention/s

  • A recommendation and a justification for this

  • Implementation considerations for the intervention/s

  • Relevant monitoring and evaluation/research priorities in relation to the intervention/s

In addition to the evidence collated in the DECIDE frameworks, full evidence profiles for the reviews of the effectiveness of interventions, as well as summaries of findings for the qualitative evidence syntheses on the acceptability, feasibility and implementation of these interventions, were made available to the guidance panel. These summaries were, in turn, linked to full systematic reviews. The DECIDE Team also prepared a video to help guidance panel members understand the purpose of the framework and how to use it. This was made available prior to the guidance panel meeting and also shown at the start of the meeting.

Discussion

Broadening the range of evidence included in guidance: lessons learnt

The first innovative approach that we used in the development of the OptimizeMNH guidance was the systematic inclusion of additional types of evidence to complement the evidence of effectiveness (Table 5). This approach had a number of implications. On a practical level, broadening the scope of evidence to be considered in relation to each guidance question required more time and resources. It also required a wider set of synthesis, assessment and interpretation skills within the technical team preparing the documents for the guidance panel. In our case, these skills were represented within the technical team. Other technical teams using a similar guidance development approach may not have the skills to undertake qualitative evidence syntheses or country case study syntheses but will at least need skills in commissioning and critically appraising such evidence. Creating technical teams with these skills may be challenging as there are few groups with extensive experience in implementing them. However, the number of researchers with the appropriate skills is increasing rapidly, and guidance is now well-developed [3335].
Table 5

Overview of the different types of evidence used in the OptimizeMNH guidance development process

What type of evidence did we use?

Which part of the DECIDE evidence-to-decision framework was it used to address?

What type of evidence did it consider?

Systematic reviews of effectiveness

- What are the benefits and harms of the different task-shifting options?

The reviews primarily included randomised trials of task-shifting interventions. In some reviews, non-randomised study designs were also included.

Qualitative evidence syntheses

- Is the task-shifting option acceptable to most stakeholders?

- Is the task-shifting option feasible to implement?

The qualitative evidence syntheses included primary studies of task-shifting that had used qualitative methods for data collection and for data analysis

Multi-country case study syntheses

- Is the task-shifting option acceptable to most stakeholders?

- Is the task-shifting option feasible to implement?

The multi-country case study syntheses reviewed evaluations and studies of large-scale programmes designed to optimise the health workforce in LMICs. Evaluation reports, programme guidelines and published studies were gathered for each selected country programme

Primary research

- Is the task-shifting option acceptable to most stakeholders?

- Is the task-shifting option feasible to implement?

The primary research involved qualitative thematic analysis of messages submitted to two email discussion forums.

The forums focus on the healthcare information needs of frontline health workers and citizens in LMICs and how these needs can be met and also include discussion of diverse aspects of health systems.

We had originally planned to gather evidence about stakeholder acceptability and feasibility through qualitative evidence syntheses only. As described above, these syntheses, which only included published qualitative studies, primarily offered evidence surrounding factors and stakeholders at the community or primary healthcare facility level. This led us to carry out the multi-country case studies, where we included a broader set of information sources, and also led us to carry out our own primary study among stakeholders poorly represented in the qualitative evidence syntheses. In retrospect, we think that the findings of this primary study should have been included in the qualitative evidence syntheses rather than being treated as a stand-alone product.

Although this innovation expanded the preparatory work needed for the guidance, we would argue that the inclusion of evidence from qualitative evidence syntheses and country case study syntheses offered relevant information for the guidance panel on the feasibility and acceptability of the interventions. When such evidence is not systematically assessed and included, acceptability and feasibility issues may either be overlooked altogether or incorporated into guideline decisions based only on anecdotal evidence. Our experience from the development of the OptimizeMNH guidance suggests that the inclusion of evidence from qualitative evidence syntheses and country case study syntheses reduced the use of anecdotal evidence by guidance panel members when assessing the balance of consequences for each guidance question. In addition, our use of such evidence appeared to address guideline panel members’ perceptions that the WHO often over-emphasises evidence from randomised trials at the expense of evidence from programme experience. One additional benefit of synthesising relevant qualitative evidence was that we were able to use this to develop implementation considerations for task-shifting in maternal and newborn health [1].

Introducing an approach to assess confidence in findings from qualitative evidence syntheses: lessons learnt

Our second innovation was the use of the GRADE-CERQual approach in order to systematically and transparently assess our confidence in findings from the qualitative evidence syntheses conducted to inform the guidance.

Our use of CERQual was received well by the guidance panel, in part because it addressed the need to systematically incorporate a wider range of evidence into the guidance process. The approach also had a number of advantages. When making recommendations, all guidance panels need to take into account how confident they can be in the underlying evidence. Had we not provided the OptimizeMNH panels with the CERQual assessments of our confidence in the findings from the qualitative evidence syntheses, they would have had to make their own judgments and may not have done this in a systematic way. The use of CERQual also allowed us to identify and highlight gaps in the evidence. For instance, where we assessed the confidence in the evidence for a qualitative finding to be low or very low, this was an indication that further research was needed in this area. These research gaps were also reflected in the information presented to the guidance panel and carried through to the recommendations.

We were initially concerned that using the CERQual approach would confuse guidance panel members as they would not have encountered this approach before. In practice, this did not appear to be a problem, probably because members were already familiar with GRADE for assessing findings regarding the effectiveness of interventions, which uses the same principles, and because the panel was briefed on the approach at the start of the meetings.

Some of the findings contributing to OptimizeMNH were based on a very wide range of evidence types. In particular, the country case studies included both qualitative and quantitative studies as well as programme reports, which often did not provide detailed descriptions of methods used. Using CERQual to assess our confidence in the findings from the country case study syntheses was not feasible as the CERQual approach is not designed, at present, to accommodate this range of evidence [26].

Using a structured health systems framework: lessons learnt

Our final innovation was the use of DECIDE evidence-to-decision frameworks [36] to present evidence to the guidance panel. There were several advantages in using this structured framework. Firstly, the framework helped panel members to think through each of the considerations that might be important in making a recommendation and ensured that key considerations were taken into account. The concise format also appeared to help focus and structure the discussions and may have improved the use of the limited time that the guidance panel had to weigh the balance of consequences and make judgements about complex questions regarding task-shifting. Because the framework prompted the guidance panel to justify all of the recommendations made, we believe that the transparency of the decision-making process was improved. Furthermore, end users of the guidance have access to a clear record of recommendation decisions and how these were made. The framework also contributes to closing the knowledge-to-action cycle by capturing implementation considerations and highlighting evidence gaps where more research is needed [37, 38].

The use of the evidence-to-decision frameworks also led to a number of challenges. First of all, preparation of the frameworks took considerable time in advance of the panel meeting, although this may have saved time later on when preparing the final guidance document. Secondly, because the DECIDE framework is a new tool, it required some explanation in advance of the panel meetings. However, panel members appeared to grasp the function and content quickly, and the informal feedback on the framework was positive.

Because the OptimizeMNH guidance included a very large number of recommendations (128 in total), the amount of time available in the panel meetings to discuss each evidence-to-decision framework was limited. The inclusion of additional types of evidence also added to the amount of information that panel members were expected to read. Nonetheless, our experience suggests that the structured format of the DECIDE framework allowed the technical team to present a large and complex body of evidence to panel members in a fairly straightforward and easy-to-assimilate way. Further work is now needed to explore guidance panels’ views of such evidence-to-decision frameworks and to examine how the presentation of evidence in these formats impacts on the deliberations and decisions of guidance panels and on the transparency to users of these decisions.

Conclusion

In this paper, we have described our use of a number of innovative strategies to expand the range of evidence used to develop WHO global recommendations and discussed our experiences with these strategies. Expanding the range of evidence contributing to a guidance process increases the amount of time and resources needed and the range of skills required within the technical team developing the guidance. However, our experience with the OptimizeMNH guidance suggests that such efforts may be valuable, particularly for recommendations on more complex health and social interventions. This view is supported by efforts within WHO to replicate the OptimizeMNH approach in other guidance processes [27]. When considering whether to adopt a similar approach, producers of guidance should assess the extent to which acceptability and feasibility issues are likely to be important enough to influence their recommendations. So far, the WHO is using this approach in recommendations in guidance on more complex health systems and behavioural interventions, but acceptability and feasibility issues may be equally relevant for clinical interventions. In fact, we would suggest that there are few situations where these issues are not likely to be relevant, and even in situations where guidance developers feel confident that they already have a sufficient overview of these issues and additional evidence is not required, a framework approach where this is made transparent can still be useful. We have developed a list of questions (Table 6) to help groups involved in developing guidance decide whether additional evidence syntheses are needed. Further research is needed to refine these questions and to develop an understanding of the typical time and resources required to expand the range of evidence sources used to develop guidance. Table 2 also provides links to resources available to guidance developers who are considering expanding the range of syntheses that will be used to inform the guidance.
Table 6

When to consider expanding the range of evidence syntheses undertaken to inform the development of a guideline or guidance

Principles:

• For most interventions for which guidance is developed, including clinical, health systems and public health interventions, guidance panels should consider how different stakeholders value different outcomes; the effectiveness, acceptability and feasibility of the intervention; implications for resource use; equity impacts; and implementation considerations

• A structured evidence-to-decision framework, such as the DECIDE framework [30, 31], should be used to guide these considerations, even if it is likely that evidence syntheses will not be needed for some of the considerations included in these frameworks (see below)

When to consider expanding the range of evidence syntheses informing guidance development:

• Expanding the range of evidence syntheses beyond evidence of effectiveness should be considered under the following circumstances:

 ◦ If the guidance is considering a new intervention, an intervention for which guidance has not previously been developed, or an intervention for which the implementation mechanism/s have changed

 ◦ If there are reasons to anticipate that feasibility, acceptability, resource use and/or equity are likely to be important considerations for the interventions included in this guidance. These considerations may be particularly important for interventions directed at health systems or other systems; interventions that focus on changing people’s views or behaviours; and interventions that have multiple components and long causal pathways or involve multiple actors or systems. Note that stakeholders should be involved in discussions on this issue

• If a decision has been made that the range of evidence syntheses needed to support the development of guidance should be expanded, consider the following:

 ◦ Do evidence syntheses already exist that could be used to address the considerations identified?

 ◦ Will existing evidence syntheses require additional work, for example, to assess how much confidence to place in the synthesis findings and/or to undertake specific subgroup analyses?

 ◦ If existing evidence syntheses are not available for some or all of the considerations to be explored by the guidance:

  ▪ Do the timeline and resources available for the guidance development allow for new syntheses to be undertaken?

  ▪ If so, which are the highest priority considerations for new syntheses and what should be the scope of these syntheses?

  ▪ What skills are likely to be required to undertake these syntheses? Are the skills to undertake these syntheses available within the lead organisation for the guidance or do they need to be commissioned externally?

  ▪ What financial and other resources are needed to conduct these syntheses within the timeframe for development of the guidance?

  ▪ Could the process of conducting multiple evidence syntheses be made more efficient by combining database searching and screening for several of the syntheses, such as those syntheses focusing on intervention effectiveness, acceptability, feasibility and resource use considerations?

• Once the evidence syntheses are underway or available, consider the following:

 ◦ What tools are needed to assess how much confidence to place in the synthesis findings (for instance, the GRADE approach)? Will these assessments be conducted by the review teams, by the guidance commissioners or by a third party?

  Who will be responsible for summarising the available evidence in order to populate an evidence-to-decision framework for each guidance question?

As more qualitative evidence syntheses are produced, methods for synthesising this type of evidence are becoming more sophisticated and the number of people able to produce these types of syntheses is growing. These developments will make the process of including qualitative evidence in evidence-to-decision processes easier in the future. However, a number of challenges remain, as discussed above. Elsewhere, we have described how evidence from qualitative research can be used in developing guidelines [39], including to help shape a guideline’s key questions by informing the populations, interventions, comparators and outcomes on which each key question should focus and to understand how different stakeholders value different outcomes. We did not use these additional strategies in OptimizeMNH, but they have been part of the development for a forthcoming WHO guideline on antenatal care [40].

Our experience from OptimizeMNH suggests that relevant programmatic experience regarding “upstream” systems-level factors is often unexplored or undocumented using traditional research approaches. Researchers should explore these issues to a greater extent [22]. Alongside this, we need to develop better methods for identifying and synthesising information from programme reports and other grey literature as well as methods for assessing our confidence in syntheses based on this type of data [41, 42].

The OptimizeMNH experience suggests that the DECIDE evidence-to-decision frameworks are a useful way of guiding panels through a wide range of evidence and towards the development of transparent recommendations. We have learnt a great deal about how to populate these frameworks with evidence of the effectiveness, feasibility and acceptability of health interventions. However, more research is needed, in particular on how these kinds of evidence are used by guidance panels in their decision-making and how different kinds of evidence influence the final recommendations made. In addition, we need further worked examples of approaches for including evidence regarding resource use and cost-effectiveness. This is one area that those involved in the DECIDE work are exploring [36].

Finally, we need to explore how this type of health system guidance can best be disseminated to and utilised by end users, including national and regional policy makers and programme planners. For most health systems questions, evidence regarding effectiveness, acceptability, feasibility, and resource use is all likely to be context-specific to some degree. End users therefore need help in adapting the recommendations to their own contexts. The OptimizeMNH guidance includes a workbook that specifically aims to help end users contextualise the recommendations [43]. However, more work is needed on tools to contextualise global and national research and guidance for implementation in specific settings [4447] and on ways of appraising such guidance [48].

Declarations

Acknowledgements

Thanks to Heather Munthe-Kaas, Benedicte Carlsen, Chris Colvin, Jane Noyes, Andrew Booth, Ruth Garside, Arash Rashidian, Meghan Bohren, Özge Tuncalp and Megan Wainwright for their contribution to the development of the GRADE-CERQual approach and to Jenny Moberg, Sarah Rosenbaum and Andy Oxman for their support in the use of the DECIDE frameworks. Thanks also to all those who contributed to the reviews and other materials produced for OptimizeMNH and those who peer reviewed those documents and to Susan Norris from WHO’s Guideline Review Committee Secretariat for her support and encouragement throughout this work.

CG and SL were funded by the Norwegian Agency for Development Corporation (Norad) and by the Norwegian Knowledge Centre for the Health Services (NOKC). SL is also supported by funding from the South African Medical Research Council. None of these funders had any involvement in the design, collection, analysis or interpretation of data, in manuscript preparation or submission. This work was also supported by funding from the UNDP/UNFPA/ UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction, Department of Reproductive Health and Research, WHO, Geneva, Switzerland.

Authors’ contributions

All authors conceived of the article, participated in its design and helped to draft the manuscript. All authors read and approved the final manuscript.

Authors’ information

CG, SL and MG are all active contributors to the work of the Cochrane Collaboration.

Competing interests

The authors have all participated in the development of the DECIDE evidence-to-decision frameworks and the GRADE-CERQual approach.

Ethics approval and consent to participate

Not applicable. This study did not involve any humans or animals.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Global Health Unit, Knowledge Centre for the Health Services, Norwegian Institute of Public Health
(2)
Health Systems Research Unit, South African Medical Research Council
(3)
Department of Reproductive Health and Research, including the UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP), World Health Organization

References

  1. WHO. Optimizing health worker roles to improve access to key maternal and newborn health interventions through task shifting (OptimizeMNH). Geneva: World Health Organization; 2012.Google Scholar
  2. Noyes J, Gough D, Lewin S, Mayhew A, Michie S, Pantoja T, et al. A research and development agenda for systematic reviews that ask complex questions about complex interventions. J Clin Epidemiol. 2013;66(11):1262–70. doi:https://doi.org/10.1016/j.jclinepi.2013.07.003.View ArticlePubMedGoogle Scholar
  3. Johnson MJ, May CR. Promoting professional behaviour change in healthcare: what interventions work, and why? A theory-led overview of systematic reviews. BMJ Open. 2015;5(9):e008592. doi:https://doi.org/10.1136/bmjopen-2015-008592.View ArticlePubMedPubMed CentralGoogle Scholar
  4. Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, et al. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med. 2010;8:63. doi:https://doi.org/10.1186/1741-7015-8-63.View ArticlePubMedPubMed CentralGoogle Scholar
  5. Dambisya YM, Matinhure S. Policy and programmatic implications of task shifting in Uganda: a case study. BMC Health Serv Res. 2012;12:61. doi:https://doi.org/10.1186/1472-6963-12-61.View ArticlePubMedPubMed CentralGoogle Scholar
  6. Georgeu D, Colvin CJ, Lewin S, Fairall L, Bachmann MO, Uebel K, et al. Implementing nurse-initiated and managed antiretroviral treatment (NIMART) in South Africa: a qualitative process evaluation of the STRETCH trial. Implement Sci. 2012;7:66. doi:https://doi.org/10.1186/1748-5908-7-66.View ArticlePubMedPubMed CentralGoogle Scholar
  7. Ledikwe JH, Kejelepula M, Maupo K, Sebetso S, Thekiso M, Smith M, et al. Evaluation of a well-established task-shifting initiative: the lay counselor cadre in Botswana. PLoS One. 2013;8(4):e61601. doi:https://doi.org/10.1371/journal.pone.0061601.View ArticlePubMedPubMed CentralGoogle Scholar
  8. Cataldo F, Kielmann K, Kielmann T, Mburu G, Musheke M. ‘Deep down in their heart, they wish they could be given some incentives’: a qualitative study on the changing roles and relations of care among home-based caregivers in Zambia. BMC Health Serv Res. 2015;15:36. doi:https://doi.org/10.1186/s12913-015-0685-7.View ArticlePubMedPubMed CentralGoogle Scholar
  9. Glenton C, Scheel IB, Pradhan S, Lewin S, Hodgins S, Shrestha V. The female community health volunteer programme in Nepal: decision makers’ perceptions of volunteerism, payment and other incentives. Soc Sci Med (1982). 2010;70(12):1920–7. doi:https://doi.org/10.1016/j.socscimed.2010.02.034.View ArticleGoogle Scholar
  10. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;182(18):E839–42. doi:https://doi.org/10.1503/cmaj.090449.View ArticlePubMedPubMed CentralGoogle Scholar
  11. Lavis JN, Boyko JA, Oxman AD, Lewin S, Fretheim A. SUPPORT tools for evidence-informed health policymaking (STP) 14: organising and using policy dialogues to support evidence-informed policymaking. Health Res Policy Syst. 2009;7 Suppl 1:S14. doi:https://doi.org/10.1186/1478-4505-7-s1-s14.View ArticlePubMedPubMed CentralGoogle Scholar
  12. Atkins L, Smith JA, Kelly MP, Michie S. The process of developing evidence-based guidance in medicine and public health: a qualitative study of views from the inside. Implement Sci. 2013;8:101. doi:https://doi.org/10.1186/1748-5908-8-101.View ArticlePubMedPubMed CentralGoogle Scholar
  13. Pagliari C, Grimshaw J. Impact of group structure and process on multidisciplinary evidence-based guideline development: an observational study. J Eval Clin Pract. 2002;8(2):145–53.View ArticlePubMedGoogle Scholar
  14. WHO. World Health Organization handbook for guideline development. 2nd ed. Geneva: World Health Organization; 2014.Google Scholar
  15. Oxman AD, Lavis JN, Fretheim A. Use of evidence in WHO recommendations. Lancet. 2007;369(9576):1883–9. doi:https://doi.org/10.1016/s0140-6736(07)60675-8.View ArticlePubMedGoogle Scholar
  16. Petticrew M, Rehfuess E, Noyes J, Higgins JP, Mayhew A, Pantoja T, et al. Synthesizing evidence on complex interventions: how meta-analytical, qualitative, and mixed-method approaches can contribute. J Clin Epidemiol. 2013;66(11):1230–43. doi:https://doi.org/10.1016/j.jclinepi.2013.06.005.View ArticlePubMedGoogle Scholar
  17. Colvin CJ. Anthropologies in and of evidence making in global health research and policy. Med Anthropol. 2015;34(2):99–105. doi:https://doi.org/10.1080/01459740.2014.963196.View ArticlePubMedGoogle Scholar
  18. Glenton C, Khanna R, Morgan C, Nilsen ES. The effects, safety and acceptability of compact, pre-filled, autodisable injection devices when delivered by lay health workers. Tropical Med Int Health. 2013;18(8):1002–16. doi:https://doi.org/10.1111/tmi.12126.View ArticleGoogle Scholar
  19. Glenton C, Colvin CJ, Carlsen B, Swartz A, Lewin S, Noyes J, et al. Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and child health: qualitative evidence synthesis. Cochrane Database Syst Rev (Online). 2013;10:Cd010414. doi:https://doi.org/10.1002/14651858.CD010414.pub2.Google Scholar
  20. Colvin CJ, de Heer J, Winterton L, Mellenkamp M, Glenton C, Noyes J, et al. A systematic review of qualitative evidence on barriers and facilitators to the implementation of task-shifting in midwifery services. Midwifery. 2013. doi:https://doi.org/10.1016/j.midw.2013.05.001.PubMedGoogle Scholar
  21. Rashidian A, Shakibazadeh E, Karimi- Shahanjarini A, Glenton C, Noyes J, Lewin S, Colvin C, Laurant M. Barriers and facilitators to the implementation of doctor-nurse substitution strategies in primary care: qualitative evidence synthesis (Protocol). Cochrane Database Syst Rev. 2013(2). doi: https://doi.org/10.1002/14651858.CD010412.
  22. Gopinathan U, Lewin S, Glenton C. Implementing large-scale programmes to optimise the health workforce in low- and middle-income settings: a multicountry case study synthesis. Tropical Med Int Health. 2014;19(12):1437–56. doi:https://doi.org/10.1111/tmi.12381.View ArticleGoogle Scholar
  23. (HIFA2015) HIFAB. Health information for all by 2015 electronic discussion list. Accessed 2nd February 2016. http://www.hifa2015.org/.
  24. Owolabi OO, Glenton C, Lewin S, Pakenham-Walsh N. Stakeholder views on the incorporation of traditional birth attendants into the formal health systems of low-and middle-income countries: a qualitative analysis of the HIFA2015 and CHILD2015 email discussion forums. BMC Pregnancy Childbirth. 2014;14:118. doi:https://doi.org/10.1186/1471-2393-14-118.View ArticlePubMedPubMed CentralGoogle Scholar
  25. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94. doi:https://doi.org/10.1016/j.jclinepi.2010.04.026.View ArticlePubMedGoogle Scholar
  26. Lewin S, Glenton C, Munthe-Kaas H, Carlsen B, Colvin CJ, Gulmezoglu M, et al. Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med. 2015;12(10):e1001895. doi:https://doi.org/10.1371/journal.pmed.1001895.View ArticlePubMedPubMed CentralGoogle Scholar
  27. WHO. Health worker roles in providing safe abortion care and post abortion contraception. Geneva: World Health Organization; 2015.Google Scholar
  28. DECIDE. Developing and evaluating communication strategies to support informed decisions and practice based on evidence. Accessed 2nd February 2016. www.decide.collaboration.eu.
  29. GRADE. The GRADE working group. Accessed 2nd February 2016. www.gradeworkinggroup.org.
  30. Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl E, Davoli M, GRADE Working Group. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: introduction. BMJ (Clin Res Ed). 2016;353:i2016. doi:https://doi.org/10.1136/bmj.i2016.
  31. Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl E, Davoli M, Grade Working Group. GRADE Evidence to Decision (EtD) frameworks: 2. clinical practice guidelines. BMJ (Clin Res Ed). 2016;353:i2089. doi:https://doi.org/10.1136/bmj.i2089.
  32. Vogel JP, Oxman AD, Glenton C, Rosenbaum S, Lewin S, Gulmezoglu AM, et al. Policymakers’ and other stakeholders’ perceptions of key considerations for health system decisions and the presentation of evidence to inform those considerations: an international survey. Health Res Policy Syst. 2013;11:19. doi:https://doi.org/10.1186/1478-4505-11-19.View ArticlePubMedPubMed CentralGoogle Scholar
  33. Gough D, Oliver S, Thomas J. An introduction to systematic reviews. Thousand Oaks: Sage Publications; 2012.Google Scholar
  34. Noyes J, Popay J, Pearson A, Hannes K, Booth A, on behalf of the Cochrane Qualitative Research Methods Group. Chapter 20: qualitative research and Cochrane reviews. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration; 2011.www.cochrane.org.
  35. Saini M, Shlonsky A. Systematic synthesis of qualitative research. Oxford: University Press; 2012.Google Scholar
  36. Treweek S, Oxman AD, Alderson P, Bossuyt PM, Brandt L, Brozek J, et al. Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6. doi:https://doi.org/10.1186/1748-5908-8-6.View ArticlePubMedPubMed CentralGoogle Scholar
  37. Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26(1):13–24. doi:https://doi.org/10.1002/chp.47.View ArticlePubMedGoogle Scholar
  38. Kitson A, Straus SE. The knowledge-to-action cycle: identifying the gaps. CMAJ. 2010;182(2):E73–7. doi:https://doi.org/10.1503/cmaj.081231.View ArticlePubMedPubMed CentralGoogle Scholar
  39. Glenton C, Lewin S, Norris S. Using evidence from qualitative research to develop WHO guidelines. In: Norris S, editor. Handbook for Guideline Development. 2nd ed. Geneva: WHO: World Health Organization; 2016.Google Scholar
  40. Downe S, Finlayson K, Tuncalp Ӧ, Metin Gulmezoglu A. What matters to women: a systematic scoping review to identify the processes and outcomes of antenatal care provision that are important to healthy pregnant women. BJOG. 2016;123(4):529–39. doi:https://doi.org/10.1111/1471-0528.13819.View ArticlePubMedGoogle Scholar
  41. Lewin S, Bosch-Capblanch X, Oliver S, Akl EA, Vist GE, Lavis JN, et al. Guidance for evidence-informed policies about health systems: assessing how much confidence to place in the research evidence. PLoS Med. 2012;9(3):e1001187. doi:https://doi.org/10.1371/journal.pmed.1001187.View ArticlePubMedPubMed CentralGoogle Scholar
  42. Yin R. Case study research: design and methods. Los Angeles: Sage Publications; 2009.Google Scholar
  43. WHO. Annex 8: contextualizing the guidelines—workbook. Optimizing health worker roles to improve access to key maternal and newborn health interventions through task shifting (OptimizeMNH). Geneva: World Health Organization; 2012.Google Scholar
  44. Bergstrom A, Skeen S, Duc DM, Blandon EZ, Estabrooks C, Gustavsson P, et al. Health system context and implementation of evidence-based practices-development and validation of the Context Assessment for Community Health (COACH) tool for low- and middle-income settings. Implement Sci. 2015;10:120. doi:https://doi.org/10.1186/s13012-015-0305-2.View ArticlePubMedPubMed CentralGoogle Scholar
  45. Maar M, Yeates K, Barron M, Hua D, Liu P, Moy Lum-Kwong M, et al. I-RREACH: an engagement and assessment tool for improving implementation readiness of researchers, organizations and communities in complex interventions. Implement Sci. 2015;10:64. doi:https://doi.org/10.1186/s13012-015-0257-6.View ArticlePubMedPubMed CentralGoogle Scholar
  46. Burford B, Lewin S, Welch V, Rehfuess E, Waters E. Assessing the applicability of findings in systematic reviews of complex interventions can enhance the utility of reviews for decision making. J Clin Epidemiol. 2013;66(11):1251–61. doi:https://doi.org/10.1016/j.jclinepi.2013.06.017.View ArticlePubMedGoogle Scholar
  47. Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, et al. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:35. doi:https://doi.org/10.1186/1748-5908-8-35.View ArticlePubMedPubMed CentralGoogle Scholar
  48. Ako-Arrey DE, Brouwers MC, Lavis JN, Giacomini MK. Health systems guidance appraisal—a critical interpretive synthesis. Implement Sci. 2016;11(1):9. doi:https://doi.org/10.1186/s13012-016-0373-y.View ArticlePubMedPubMed CentralGoogle Scholar
  49. Bosch-Capblanch X, Lavis JN, Lewin S, Atun R, Rottingen JA, Droschel D, et al. Guidance for evidence-informed policies about health systems: rationale for and challenges of guidance development. PLoS Med. 2012;9(3):e1001185. doi:https://doi.org/10.1371/journal.pmed.1001185.View ArticlePubMedPubMed CentralGoogle Scholar
  50. Gilson L. Qualitative research synthesis for health policy analysis: what does it entail and what does it offer? Health Policy Plan. 2014;29 Suppl 3:iii1–5. doi:https://doi.org/10.1093/heapol/czu121.View ArticlePubMedGoogle Scholar
  51. Hodnett ED, Gates S, Hofmeyr GJ, Sakala C, Weston J. Continuous support for women during childbirth. Cochrane Database Syst Rev (Online). 2011(2):Cd003766. doi:https://doi.org/10.1002/14651858.CD003766.pub3.
  52. Polus S, Lewin S, Glenton C, Lerberg PM, Rehfuess E, Gulmezoglu AM. Optimizing the delivery of contraceptives in low- and middle-income countries through task shifting: a systematic review of effectiveness and safety. Reprod Health. 2015;12:27. doi:https://doi.org/10.1186/s12978-015-0002-2.View ArticlePubMedPubMed CentralGoogle Scholar

Copyright

© Glenton et al. 2016

Advertisement