- Systematic review
- Open Access
- Open Peer Review
Using theory of change to design and evaluate public health interventions: a systematic review
Implementation Sciencevolume 11, Article number: 63 (2016)
Despite the increasing popularity of the theory of change (ToC) approach, little is known about the extent to which ToC has been used in the design and evaluation of public health interventions. This review aims to determine how ToCs have been developed and used in the development and evaluation of public health interventions globally.
We searched for papers reporting the use of “theory of change” in the development or evaluation of public health interventions in databases of peer-reviewed journal articles such as Scopus, Pubmed, PsychInfo, grey literature databases, Google and websites of development funders. We included papers of any date, language or study design. Both abstracts and full text papers were double screened. Data were extracted and narratively and quantitatively summarised.
A total of 62 papers were included in the review. Forty-nine (79 %) described the development of ToC, 18 (29 %) described the use of ToC in the development of the intervention and 49 (79 %) described the use of ToC in the evaluation of the intervention. Although a large number of papers were included in the review, their descriptions of the ToC development and use in intervention design and evaluation lacked detail.
The use of the ToC approach is widespread in the public health literature. Clear reporting of the ToC process and outputs is important to strengthen the body of literature on practical application of ToC in order to develop our understanding of the benefits and advantages of using ToC. We also propose a checklist for reporting on the use of ToC to ensure transparent reporting and recommend that our checklist is used and refined by authors reporting the ToC approach.
Most public health interventions are inherently complex, with multiple interacting components, delivered at multiple levels. This complexity makes them difficult to evaluate using traditional experimental designs. Public health interventions often rely on ongoing quality improvement based on the implementation experience. Therefore, they may not reach the level of stability required to conduct evaluations such as randomised controlled trials (RCTs) . Some researchers propose that understanding the public health intervention’s underlying theory of change (ToC) and its related uncertainties may improve the evaluation of complex health interventions [1–3].
Theory-driven evaluation is a collection of evaluation methods which emphasise the importance of understanding how and why a programme works in order to evaluate it [4, 5]. By programme, we mean a set of organised activities or interventions supported by resources designed to achieve a specific result . The theories are first made explicit and then used to see how the programme theory results in the intended outcomes . There are several overlapping types of theory-driven evaluation approaches including logic models, logical frameworks, outcomes hierarchies, realist evaluation, and ToC [4, 5].
ToC was developed by Weiss and others  within the tradition of theory-driven evaluation. Although definitions of ToC vary, we define it as an approach which describes how a programme brings about specific long-term outcomes through a logical sequence of intermediate outcomes . The ToC is often developed using a backward mapping approach which starts with the long-term outcome and then maps the required process of change and the short- and medium-term outcomes required to achieve this . During this process, the assumptions about what needs to be in place for the ToC to occur are made explicit as well as the contextual factors which influence the ToC. Additional elements of a ToC can include beneficiaries, research evidence supporting the ToC, actors in the context, sphere of influence, strategic choices and interventions, timelines and indicators . These elements are usually presented in a diagram and/or narrative summary .
The ToC is usually developed in consultation with stakeholders through workshops or interviews although the participation of stakeholders can vary substantially in practice . For example, some ToCs are developed through a series of workshops and meetings with a wide range of stakeholders including service users [11, 12] whereas others are developed by evaluators and funders using programme documentation [13, 14]. The resulting ToCs can be used as a framework for programme development and evaluation . The ToC approach is method neutral and as such does not prescribe specific types of evaluation methods such as qualitative interviews or RCTs .
ToC is distinct from sociological or psychological theories which describe why change occurs although these may be used to inform the ToC . For example, Bauer used an ecological model of community organising to inform a ToC for a capacity and advocacy initiative for residents to impact on public health policy and training of public health professionals .
ToC differs from other theory-driven approaches to evaluation despite similar origins. For example, although logic models outline the inputs, processes, outputs and outcomes of a programme in a similar manner to ToC, they can be rigid and do not make explicit the causal pathways through which change happens in the way that ToC does . Similarly, although logframes were initially developed to summarise discussions with stakeholders, funder-driven formats have largely reduced logframes to a results-based management tool . Realist evaluation, on the other hand, comes from a perspective of scientific realism and focuses predominantly on the interaction between the context, mechanisms and outcomes of the programme. Usually used post hoc, evaluators seek to uncover the underlying programme theories. These theories are often more abstract than the theories developed through ToC or logic models . The development of ToC has been influenced by Freirean thinking on how to create social change by empowering individuals . Despite some fundamental differences in their theoretical underpinnings, many of these approaches are used interchangeably or together [18, 20].
ToC has been used widely in the development sector for programme development and evaluation by funders such as the UK’s Department for International Development, Comic Relief, Grand Challenges Canada and the Gates Foundation [3, 19]. However, there has been no global systematic review to our knowledge on the use of ToC for the design and evaluation of public health interventions. Coryn et al.  conducted a review of theory-driven evaluation more broadly. They found 45 examples of theory-driven evaluation in the peer-reviewed literature between 1990 and 2009. These evaluations included education, crime and safety and transportation interventions. Roughly half (21/45) were evaluations of health interventions . A rapid analysis of the included papers in preparation for this review indicated that only three of these used ToC.
The lack of a systematic review means that there is no clear idea of how the ToC has been used and reported in the peer-reviewed and grey literature in relation to public health interventions. Given the increasing popularity of the ToC approach, understanding how it is has been used and described previously allows future users of the approach to learn from the work of others and build upon it. It also helps to move towards a more consistent way of using the ToC approach.
In this review, we sought to review both peer-reviewed and grey literature to determine how ToCs have been developed and used in the development and evaluation of public health interventions globally. Specifically, we sought to answer the following questions:
How are ToCs for public health interventions developed and refined?
How is the ToC approach used in the
development of an intervention;
implementation of the intervention;
development of indicators for measurement;
evaluation of the intervention, including statistical approaches; and
conceptualisation/evaluation of the influence of context.
The authors developed a protocol for this review which was agreed prior to the commencement of the study. This is available in Web Additional file 1.
Inclusion and exclusion criteria
The inclusion and exclusion criteria are listed in Table 1. In summary, we included studies of public health interventions which were defined as interventions which are intended to protect health or prevent or treat ill health in communities or populations . We included papers describing interventions addressing any health issue in all populations which (a) described how a ToC approach was used to design, implement or evaluate a public health intervention or (b) described the development of a ToC for a public health intervention. Evaluation study designs included longitudinal studies, quantitative surveys, case study research  and qualitative studies.
We required papers to specifically mention that they used “theory of change” and excluded those who did not for the following reasons. Firstly, as described above, there are a range of overlapping definitions for ToC and other programme evaluation methods. Given the often minimal amount of detail provided about the programme theory in papers, and especially in abstracts, it would be difficult to enforce a standard criteria for ToC against which papers could be evaluated for inclusion. Secondly, piloting the initial broad search strategy (including all synonyms for ToC and programme logic) returned more than 20,000 hits in only three databases. By refining the criteria to specify ToC by name, we were able to thoroughly explore literature which explicitly self-identified using ToC.
As the focus of this review was on public health interventions, we excluded papers in which the long-term outcome of the ToC was a change within an individual rather than change in the population. For example, a ToC describing how cognitive behavioural therapy may impact on an individual’s cognitive processes and behaviour would be a change within the individual. However, if the focus of the ToC was on how a cognitive behavioural therapy intervention impacted the prevalence of depression would be change in the population. We excluded reviews and methodological or advocacy papers unless they included an example of how a ToC was developed or how ToC was used in the design, evaluation and/or implementation of a public health intervention. We did not limit the inclusion by date, language, study design or type of publication.
The database searches were conducted between the 16th November and the 4th December 2013 by EB. The main search term used was “theory of change”. Where the database allowed, we limited this to health or healthcare and to humans. We searched databases of peer-reviewed journal articles (Scopus, PubMed, PsychInfo, Science Citation Index, Social Science Citation Index, Academic Search Premier, Africa-Wide Information, CINAHL and BIOSIS). An example of a search string used for PyschInfo was “theory of change” AND (“health” OR “healthcare” OR “health services” OR “medicine”). We also searched grey literature databases (The Directory of Published Proceedings OpenGrey, Disability Archive UK, Eldis, Popline, DFID Research for Development, SciDevNet and World Bank Documents and Reports) and the first 50 pages of a Google search. The websites of Comic Relief, DFID, Grand Challenges Canada, The Bill and Melinda Gates Foundation, HIVOS, World Vision, the Robert Wood Johnson foundation, Actknowledge and the Theory of Change Community were also searched. In addition, we contacted experts in the field and sent requests for papers to two existing global mailing lists for evaluators: MandENEWS and Pelican.
Screening and eligibility
Following the search of databases of peer-reviewed journal articles, the titles and abstracts of the search results from peer-reviewed papers were exported into Endnote  where duplicates and irrelevant titles were removed. The peer-reviewed journal articles found through contact with experts were added to this. The titles and abstracts were double screened by EB and LL against the inclusion and exclusion criteria. Once the abstracts were screened, the full papers or reports of the included abstracts were obtained and assessed for eligibility by both reviewers.
Following the grey literature search as described above, all potentially relevant results were saved into Evernote . These were double screened by both reviewers against the inclusion and exclusion criteria.
Any differences between authors’ opinions were resolved via discussion throughout the review process.
Data extraction and analysis
The data from the papers were extracted by the first author (EB) onto a data extraction form. This included information on authors, publication dates, the type of interventions and outcomes, the development of ToC, the use of ToC in the design, implementation and evaluation of the intervention and the influence of context. The data collection form also included key principles of theory-driven evaluation proposed by Coryn et al. . These included how the programme theory was (a) formulated, (b) used to formulate and prioritise evaluation questions, (c) plan and conduct evaluations, (d) inform the measurement of constructs in the programme theory and (e) provide a causal explanation. Where a paper described or showed a ToC, we assessed what elements of ToC they presented. The list of ToC elements was adapted from Vogel and included context, long-term change, process/sequence of change and assumptions . However, as there is no agreed upon assessment of quality for papers reporting ToC, we did not asses the quality of the included papers. We did not contact authors for additional information.
Descriptive statistics were calculated using STATA 13 . The papers were compared, evaluated and summarised narratively in relation to review questions. Due to the heterogeneity of the study designs, interventions and outcomes included in this review, a meta-analysis was not conducted.
In total, 566 abstracts were screened, resulting in 200 full text peer-reviewed articles which were assessed for eligibility. An additional 65 records were identified from the grey literature search and screened for eligibility. A total of 62 papers were included [1, 12–14, 16, 26–82]. Figure 1 is adapted from the PRISMA guidelines  and summarises the search process and results.
The publication dates of the papers range between 1999 and 2013, with a steady increase in papers over time (Fig. 2). The majority were published in English in peer-reviewed journals, but we also included PhD theses, presentations and NGO reports from the grey literature. Most of the research was conducted in the USA or the UK. More details are provided in Table 2. Four pairs of papers are reported on the same public health interventions [1, 13, 42, 43, 54, 60, 81, 82]. However, as the primary interest of this paper is how the use of ToC is described in reports and peer-reviewed journal articles, we have included them as separate papers.
A variety of types of public health interventions reported using ToC in the design, development and evaluation of public health interventions (Table 3). These included systems of care for adolescents with behavioural and emotional difficulties [12, 26, 50, 53, 54, 56, 61, 65, 80–82], substance use interventions [27, 49], domestic violence interventions , comprehensive community initiatives [13, 16, 35, 62, 81], medication supply among community health workers  and integrated district level mental healthcare plans in low- and middle-income countries .
Development of ToCs
Forty-nine papers (79 %) included some information on the ToC development process. Forty-three percent (n = 27) of the papers developed their ToCs prospectively and 19.4 % (n = 12) retrospectively. The remainder either developed their ToC during project replanning (n = 3.5 %) or did not specify when they developed their ToC (n = 20, 32 %).
The ToCs were developed using workshops [28, 34, 47, 48, 55, 63, 64, 72, 76] and working groups [12, 53, 54, 61, 68, 69, 82], document reviews [16, 35, 44, 56, 67, 71], interviews and discussions [16, 27, 29, 35, 40, 44, 47, 56, 57, 62, 65, 66, 73, 80], surveys [31, 67], programme observation [16, 44, 45, 56, 67], literature reviews [33, 40, 68, 69, 80] and existing conceptual frameworks or theory [33, 40, 42–44, 51, 64, 68, 69]. The ToC development included consultations or interviews with the following stakeholders: programme staff [27, 38, 40, 44, 45, 52, 54, 57, 63, 65–67, 72, 73, 82], management [12, 57, 61, 66, 70, 77, 82], families [12, 26, 54, 65, 77, 82], service users [39, 47, 50, 61, 65], experts [40, 64] and evaluators [13, 14, 38, 44, 52, 58, 61, 70, 75, 77, 81]. Many used multiple methods, for example, Mookheriji and Lafond used immunisation programme theory and discussion with programme stakeholders, including immunisation experts, to develop a ToC of routine immunisation performance . They used a case study approach to evaluate immunisation performance and then refined the ToC based on the results of this evaluation and a stakeholder workshop.
The resultant ToCs were described using narrative summaries (n = 15, 34.1 %), diagrams (n = 22, 50 %) or both (n = 6, 13.6 %). In one case, a table was used. Table 4 outlines the components of the ToCs that were described. Almost all of the ToCs outlined the long-term outcome required, and the majority described the process or sequence of change. However, assumptions and indicators were displayed or described infrequently.
Using ToCs to design public health interventions
Eighteen papers (29 %) described the use of ToC in the development of a public health intervention. The majority of these reported that they used the ToC as a framework for the intervention [12, 31, 42, 43, 70] or as a basis for a strategic plan [61, 68, 69, 76, 82]. Some examples of how ToCs were used to design public health interventions follow. Basson et al. used formative research to develop a ToC for a workplace wellness intervention for HIV-affected nurses and presented this programme theory as a framework for future research. Lund et al. used stakeholder workshops to develop their ToC and then used this to refine the substance and delivery of integrated district mental healthcare plans in five low- and middle-income countries . A few presentations and papers reporting the development of systems of care for children with behavioural difficulties used the ToC as an outline of their public health intervention and as a basis for their strategic plan [12, 61, 65]. Chandani et al.  used the ToC to frame the results of their formative work and used the ToC to identify interventions to address the bottlenecks to the availability of essential medicines among community health workers in Ethiopia, Malawi and Rwanda.
Using ToCs to evaluate public health interventions
Forty-nine papers (79 %) describe the use of ToC in the evaluation of the intervention. This includes the development of indicators, the overall evaluation design and data analysis.
The development of indicators used in the ToC was described in 28 papers. The indicators were often developed from the short-, medium- or long-term outcomes described in the ToC [27, 35, 38, 58, 65, 74, 81, 84]. Thirty-two (51.6 %) measured process constructs, 28 (45.2 %) measured outcome constructs and 9 (14.5 %) measured contextual constructs described in the ToC. Only two papers [12, 82] explicitly described the use of ToC to identify indicators for ongoing monitoring of the implementation of the intervention.
The majority of papers (62.9 %) reported formulating their evaluation questions around the ToC. However, the papers varied in the amount of detail they provided on this process. A common description was that the ToC was used to provide a framework for the evaluation [27, 32, 33, 48, 64, 72, 74, 78, 79, 81, 82]. Others reported that they used the evaluation to develop , refine  or validate the ToC . Two papers reported that their evaluation was guided by testing the assumptions in the ToC [29, 34].
The data collection and analysis methods used varied greatly across papers. Data collected for the evaluation included routinely collected data [33, 44], custom-designed surveys [13, 16, 32, 72, 76, 79] and qualitative data. Qualitative data collection methods included interviews [13, 27, 35, 47, 71, 73, 75, 79], programme observation [13, 27, 35], programme documentation [13, 35, 71, 75, 79] and visual evidence . The quantitative data analysis methods were strongly linked to the types of data collected and included descriptive statistics , inferential statistics [27, 40, 42, 43, 74, 78], multilevel modelling  and path analysis . Other methods included case study approaches [16, 33, 36, 64] and iterative thematic analysis  whereas others did not explicitly state their specific data analysis approach [14, 77].
Few papers explicitly explored the influence of context of the intervention in relation to ToC. Although some ToCs mentioned context, particularly those with a realist evaluation focus, there was little description of how context affected the interpretation of the evaluation. There were some exceptions [40, 56, 64, 72]. Mookherji and LaFond used a case study approach to explore what worked within and between immunisation programme contexts to identify common factors influencing immunisation performance in Ghana, Ethiopia and Cameroon . For example, political and social commitment to routine immunisation was seen as a key factor in influencing immunisation performance although it was described slightly differently for each context. Similarly, Chandani et al. developed a cross-country ToC of community health worker supplied medication in Ethiopia, Malawi and Rwanda. They compared whether each of the preconditions and the outcome was achieved in each setting . These differences were then explained based on the contextual factors in each setting such as types of medication provided by the health workers, standard operating procedures and data availability and means of transport and travel times. Secker et al.  explored the influence of socioeconomic and demographic characteristics as well as infrastructure and organisational processes and systems between eight pilot sites in the evaluation of a preretirement health initiative.
Using ToC to provide causal explanations
Few papers reported on the identification of breakdowns and side effects, effectiveness or efficacy and causal explanation as described by Coryn et al. . Only four (6.5 %) identified breakdowns of programme theory, three (4.8 %) identified unexpected consequences of the intervention, ten (16.1 %) made cause-and-effect associations between theoretical constructs explicit, two (3.2 %) described differences in direction and/or strength of relationship between programme and outcomes and two (3.2 %) described the extent to which one construct accounted for/mediated the relationship between other constructs.
In this systematic review, we provide an overview of how ToCs have been developed and used to develop and evaluate public health interventions. As expected, there is variation in how ToCs are developed and used in evaluation although the papers report very little detail about the ToC process.
We have shown that the ToC approach has been in use since at least 1999 with 62 papers found in peer-reviewed journals and grey literature. This was significantly more than expected, given that Coryn et al.  found only three papers describing theory-driven evaluation of health interventions using ToC. However, Coryn et al. only included papers that reported the use of ToC for evaluation (rather than also describing the development of ToC or the use in the design of an intervention) and excluded those that did not provide enough detail .
In this review, many papers provided little detail in relation to the process of ToC development and how the ToC was used to design the intervention or conduct the evaluation. For example, Bonner  describes the ToC approach in detail but provides only a short example of the Health Action Zones experience of using ToC to evaluate an intervention to reduce drug taking. Brown et al.  reports using a ToC approach to evaluate a health promotion intervention for adolescents. The only description of ToC was found in the abstract and then mentioned briefly in the discussion. There was no clarity on how the ToC was developed or any explicit mention of how it was used to inform the analysis.
In contrast, other papers provided extensive detail on the ToC development process. For example, Hernandez and Hodges  describe the 12 step process used to develop a ToC for interagency delivery of mental health services for children with serious emotional disturbances and their families. They describe each step in detail including the purpose of the stage of the process, the types of stakeholders participating in the step, the substance of the discussions and the decisions reached. The ToC was then displayed as a logic model for readers to gain a better understanding of the output of the process. Similarly, Mookherji and LaFond  described in detail their approach to developing their initial ToC and how the ToC was used to determine case selection for a comparative case study. They then described how they used the results of the comparative case study and the ToC workshops to refine their ToC.
A range of methods were used to develop ToCs. The methods ranged from participatory methods which encourage stakeholder participation and ownership of the ToC such as workshops and working groups, to more evaluator focused approaches such as programme observation and review of programme documentation. Although the reason for the choice of methods was rarely made explicit by the authors, these methods were presumably chosen based on the purpose, depth and level of stakeholder buy-in the ToC required. For example, the examples of the development of systems of care for children and adolescents with mental and behavioural disorders viewed stakeholder participation as very important and therefore held a series of workshops with multiple stakeholders from different government departments, service providers, families and service users [12, 50, 61]. In some cases, although stakeholders were interviewed or participated in surveys, they did not contribute explicitly to the development of the ToC [16, 57]. Sullivan and Stewart  argue that although participation of all stakeholders in the development of ToC is the ideal presented by Weiss and colleagues , this is not always practical or feasible. They argue that different types of ToC development and resulting ownership may have advantages and therefore it is important to be explicit about the development process.
The lack of detail in most of the examples in this review makes it difficult to assess the thoroughness of ToC development. In many cases, the ToC seems to have been developed superficially and then used in a cursory way during evaluation. Similarly, where diagrams or narrative summaries of ToCs are presented in the papers, very little detail is included. Most authors present the long-term outcomes, sequence of change, beneficiaries and context. However, very few make their assumptions explicit although Vogel identifies these as a core part of ToC . Where ToC was used to develop the interventions, it was often not clear how this was done apart from providing an overarching framework or strategic plan for the intervention.
A surprising finding of the review was the paucity of papers that describe the use of ToC for use during the implementation of the intervention (n = 2). Given the popularity of ToC as a monitoring and evaluation tool by international development agencies such as the Department for International Development, UK, , we had expected that more papers would use ToC during the implementation phase to assess progress towards the outcomes as well as modify implementation where necessary.
ToC theorists such as Connell and Kubisch  emphasise that the ToC approach to evaluation is method neutral and, as such, does not prescribe a specific type of study design or evaluation method. This was reflected in the papers included in this review which used a variety of qualitative and quantitative data collection and analysis methods. This flexibility in methods can be an advantage if researchers can design evaluations which seek to understand and evaluate both the outcomes and causal mechanisms which are made explicit in the ToC. However, flexibility in methods may also result in evaluations being poorly formulated in terms of the appropriateness of the methods, the rigor of data analysis or the results not interpreted in light of the ToC. In this review, evaluations were often described in detail but it was not clear how they linked to the ToC or how the ToC was used to interpret the results. However, some authors clearly develop or refine their ToCs as the results of the evaluation emerge. For example, Carroll et al.  sought to describe a theory of change for health promotion activities for hard to reach groups which was developed through the evaluation.
Most papers failed to explicitly discuss the results of the ToC in relation to unexpected outcomes, direction of causation and mediation of effects. This is similar to the conclusions drawn by Coryn et al. who report that programme theory was not used in any meaningful way to develop evaluation questions or plan and conduct and interpret the analysis .
It is interesting to note that no studies used ToC alongside RCTs as a method to unpack the programme theory underpinning the intervention. As we have noted previously, ToC holds much potential for this as RCTs alone are no longer considered adequate for the evaluation of complex health interventions .
Detailed reporting of the ToC process is particularly important as definitions of ToC differ considerably . Many papers did not define ToC. However, there were clear overlaps with other theory-driven evaluation approaches, in particular, realist approaches [32, 36, 38, 56] and logic models [12, 26, 31, 52, 54, 61, 65, 66, 68, 69, 76, 82]. Realist approaches have a different theoretical basis to ToC and differ in several ways including how they articulate and generate theory, the degree to which stakeholders are involved and the types of knowledge they seek to generate . Marchal et al. , in a systematic review on realist evaluation in health systems research, also noted that ToC and realist evaluation were often used together or interchangeably. Logic models are conceptually similar to ToC but are usually presented in a linear form with boxes for inputs, activities, outputs and outcomes with little explanation of the causal pathways linking them . Reducing a ToC to a logic model may conceal some of the explanatory power of the causal pathways.
Two limitations to this review are the lack of double data extraction and the inability to effectively measure the quality of the included papers. We did extract data on a checklist of ToC components proposed by Vogel  and principles of theory-driven evaluation by Coryn et al. , but it was difficult to make an assessment of quality. This is primarily because there is no agreed upon quality criteria for ToC. This is compounded by the flexibility of the ToC approach, both in the development of ToCs and how they can be used for evaluation. Because evaluations using ToC vary in study design and method, existing methodological checklists are of little use for comparative purposes.
We suggest that authors planning to report on ToC to guide the development or evaluation of public health interventions provide more detail on the ToC process to readers. In particular, it is important to make the ToC used explicit and this is usually easier in diagrammatic form. Complex ToCs can be simplified in a summary diagram with detailed ToCs provided as web appendices. This will help the reader to understand the authors’ expected pathways of change and judge their validity. In addition, it is imperative that authors describe in detail how the ToC was developed and used. This is particularly important as there is no single way to develop or use a ToC. Making the process explicit helps readers judge the credibility of the ToC and strengthen the literature in this field.
We have therefore developed a checklist based on this review and the work of Coryn et al.  and Vogel  which can assist with the clearer reporting of the ToC approach. The checklist gives guidance as to which aspects of the ToC should be made explicit (Table 5). It covers five domains, namely the (1) definition of ToC; (2) description of the ToC development process; (3) ToC diagram; (4) process of intervention development and (5) use of ToC in evaluation. The checklist would benefit from expert review and piloting in the real world. However, it provides a starting point for authors reporting a ToC approach. As ToC is method neutral, this checklist could also be used together with other existing checklists such as the CONSORT statement for RCTs , the STROBE guidelines for observational research  or CReDECI2 for complex intervention development and evaluation .
The ToC approach is widespread in the public health literature. Clear reporting of the ToC process and outputs is important to improve to allow the readers a thorough understanding of the work and allows them to judge the validity of the approach. We recommend that our proposed checklist is used and refined by authors reporting the ToC approach.
randomised controlled trial
theory of change
Mackenzie M, O’Donnell C, Halliday E, Sridharan S, Platt S. Do health improvement programmes fit with MRC guidance on evaluating complex interventions? BMJ. 2010;340:c185. doi:10.1136/bmj.c185.
Ling T. Evaluating complex and unfolding interventions in real time. Evaluation. 2012;18:79–91. doi:10.1177/1356389011429629.
De Silva MJ, Breuer E, Lee L, Asher L, Chowdhary N, Lund C, et al. Theory of Change: a theory-driven approach to enhance the Medical Research Council’s framework for complex interventions. Trials. 2014;15:267. doi:10.1186/1745-6215-15-267.
Coryn CLS, Noakes LA, Westine CD, Schroter DC. A systematic review of theory-driven evaluation practice from 1990 to 2009. Am J Eval. 2011;32:199–226. doi:10.1177/1098214010389321.
Funnell SC, Rogers PJ. Purposeful program theory: effective use of theories of change and logic models. 1st ed. San Francisco, CA: Jossey-Bass; 2011.
US Department of Health and Human Services, Centres for Disease Conrol and Prevention, Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: a self study guide. Atlanta, GA: Centres for Disease Control and Prevention; 2011.
Weiss C. Nothing as practical as good theory: exploring theory-based evaluation for comprehensive community initiatives for children and families. In: Connell JP, editor. New approaches to evaluating community initiatives: concepts, methods, and contexts. Washington, DC: Aspen Institute; 1995. p. 65–92.
Vogel I. Review of the use of ‘Theory of Change’ in international development. UK: Department for International Development (DFID); 2012.
Andersen A. A community builder’s approach to theory of change: a practical guide to theory development. New York: The Aspen Insitute; 2004.
Sullivan H, Stewart M. Who owns the theory of change? Evaluation. 2006;12(2):179–99. doi:10.1177/1356389006066971.
Breuer E, De Silva M, Fekadu A, Luitel N, Murhar V, Nakku J, et al. Using workshops to develop theories of change in five low and middle income countries: lessons from the programme for improving mental health care (PRIME). Int J Ment Health Syst. 2014;8(1):15.
Hernandez M, Hodges S. Applying a theory of change approach to interagency planning in child mental health. Am J Community Psychol. 2006;38(3-4):165–73. doi:10.1007/s10464-006-9079-7.
Weitzman BC, Silver D, Dillman KN. Integrating a comparison group design into a theory of change evaluation: the case of the urban health initiative. Am J Eval. 2002;23(4):371–85. doi:10.1177/109821400202300402.
Knowlton LW, Phillips C. Corporate giving gets smarter: ConAgra Foods Foundation fights childhood hunger. Found Rev. 2012;4:2. doi:10.4087/FOUNDATiONREVIEW-D-11-00034.1.
Connell JP, Kubisch AC. Applying a theory of change approach to the evaluation of comprehensive community initiatives: progress, prospects, and problems. In: Fulbright-Andersen K, Kubisch A, Connell JP, editors. New Approaches to Evaluating Community Initiatives, Vol. 2: Theory, Measurement, and Analysis United States of America: The Aspen Institute; 1998. p. 15-44.
Bauer GF. Developing community health indicators to support comprehensive community building initiatives: a case study of a participatory action research project. University of California, Berkely, US: ProQuest Information & Learning; 1999
Prinsen G, Nijhof S. Between logframes and theory of change: reviewing debates and a practical experience. Dev Pract. 2015;25(2):234–46. doi:10.1080/09614524.2015.1003532.
Marchal B, van Belle S, van Olmen J, Hoeree T, Kegels G. Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Evaluation. 2012;18:192–212. doi:10.1177/1356389012442444.
James C. Theory of change review: a report comissioned by Comic Relief; 2011. http://mande.co.uk/2012/uncategorized/comic-relief-theory-of-change-review/.
Blamey A, Mackenzie M. Theories of change and realistic evaluation. Evaluation. 2007;13(4):439–55. doi:10.1177/1356389007082129.
Rychetnik L, Frommer M, Hawe P, Shiell A. Criteria for evaluating evidence on public health interventions. J Epidemiol Community Health. 2002;56:119–27.
Yin RK. Case study research: design and methods. Fourth ed. Applied Social Research Methods Series. Sage Publications; 2009
Endnote. Endnote. X7 ed. San Fransisco: Thomson Reuters; 2015.
Evernote Cooperation. California: Evernote Corporation; 2015.
Statcorp. STATA. 13th ed. College Station: Statcorp; 2013.
Andersen R, Nesman T, Hodges S. Expressing your ideas: using a logic model to create system change USF logic model team, Tampa, Florida. University of South Florida: Tools for Creating and Sustaining Human Services: Theory of Change Logic Models; 2012.
Andreas D, Ja DY, Wilson S. Peers reach out supporting peers to embrace recovery (PROSPER): a center for substance abuse treatment recovery community services program. Alcohol Treat Q. 2010;28(3):326–38. doi:10.1080/07347324.2010.488538.
AusAID. AUusAID’s mulitsectoral contribution to the national response in Papua New Guinea. Annex 10 Theory of Change 2010. http://www.ode.ausaid.gov.au/publications/documents/pnghiv/annex10-theoryofchange.pdf. Accessed date 16 Jan 2014.
Bacchus LJ, Bewley S, Vitolas CT, Aston G, Jordan P, Murray SF. Evaluation of a domestic violence intervention in the maternity and sexual health services of a UK hospital. Reprod Health Matters. 2010;18(36):147–57. doi:10.1016/s0968-8080(10)36526-810.1136/.
Barton WH, Powers GT, Morris ES, Harrison A. Evaluating a comprehensive community initiative for children, youth, and families. Adolesc Family Health. 2001;2(1):27–36.
Basson HA, Roets L. Workplace wellness for HIV/AIDS affected nurses in South Africa. Br J Nurs. 2013;22(1):38–44.
Bauld L, Judge K, Barnes M, Benzeval M, MacKenzie M, Sullivan H. Promoting social change: the experience of health action zones in England. J Soc Policy. 2005;34(3):427–45. doi:10.1017/S0047279405008858.
Beeston C, Robinson M, Craig N. Monitoring and evaluating Scotland’s alcohol strategy. Setting the scene: theory of change and baseline picture. Edinburgh: NHS Health Scotland; 2011.
Bhattacharjee P. Use of theory of change to guide the design and monitoring of an intervention intended to reduce partner violence and increase protected sex among sex workers. STRIVE Ann Meet. 2013.
Birkby BW. The design and implementation of an evaluation for a comprehensive community-wide initiative on child maltreatment (Kentucky). Kentucky University, US: ProQuest Information & Learning; 2001.
Bonner L. Using theory-based evaluation to build evidence-based health and social care policy and practice. Crit Public Health. 2003;13(1):77–92. doi:10.1080/0958159031000100224.
Brown EC, Hawkins JD, Rhew IC, Shapiro VB, Abbott RD, Oesterle S et al. Prevention system mediation of communities that care effects on youth outcomes. Prev Sci. 2013:1-10. doi:10.1007/s11121-013-0413-7
Carr SM, Lhussier M, Wilkinson J, Gleadhill S. Empowerment evaluation applied to public health practice. Crit Public Health. 2008;18(2):161–74. doi:10.1080/09581590701499327.
Carroll M, David M, Jacobs B, Judge K, Wilkes B. A realistic/theory of change approach to the evaluation of health promotion in small- and medium-sized enterprises in Sandwell. Soc Policy Soc. 2005;4(4):393.
Chandani Y, Noel M, Pomeroy A, Andersson S, Pahl MK, Williams T. Factors affecting availability of essential medicines among community health workers in Ethiopia, Malawi, and Rwanda: solving the last mile puzzle. Am J Trop Med Hyg. 2012;87(5):120–6. doi:10.4269/ajtmh.2012.11-0781.
Cole DC, Hogg-Johnson S, Manno M, Ibrahim S, Wells RP, Ferrier SE, et al. Reducing musculoskeletal burden through ergonomic program implementation in a large newspaper. Int Arch Occup Environ Health. 2006;80(2):98–108. doi:10.1007/s00420-006-0107-6.
De La Rosa IA, Perry J, Dalton LE, Johnson V. Strengthening families with first-born children: exploratory story of the outcomes of a home visiting intervention. Res Soc Work Pract. 2005;15(5):323–38. doi:10.1177/1049731505277004.
De La Rosa IA, Perry J, Johnson V. Benefits of increased home-visitation services: exploring a case management model. Family Community Health. 2009;32(1):58–75.
Dixon-Woods M, Bosk CL, Aveling EL, Goeschel CA, Pronovost PJ. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q. 2011;89(2):167–205. doi:10.1111/j.1468-0009.2011.00625.x.
Dixon-Woods M, Tarrant C, Willars J, Suokas A. How will it work? A qualitative study of strategic stakeholders’ accounts of a patient safety initiative. Qual Saf Health Care. 2010;19(1):74–8. doi:10.1136/qshc.2008.029504.
Goss-Power PJ. Evaluation of a school-based program for youth with emotional and behavioral disabilities. New England Graduate School, US: ProQuest Information & Learning; 2005.
Gray P, Seddon T. Prevention work with children disaffected from school: findings from the evaluation of two innovative community-based projects. Health Educ. 2005;105(1):62–72. doi:10.1108/09654280510572312.
Gregor M. TB alert change story map: charities evaluation services: National Performance Programme. London: Charities Evaluation Services; 2009. http://www.ces-vol.org.uk/Resources/CharitiesEvaluationServices/Documents/tbalertchangestorymaptoc-802-810.pdf. Accessed date 16 Jan 2014.
Henderson PM. Program evaluation of a long-term substance abuse treatment program for homeless men. US: ProQuest Information & Learning; 2004.
Illinois Caucus for Adolescent Health. Strategic Plan. 2013. http://icah.org/about/#strategic-plan. Accessed 16th January 2014.
Kemp L, Harris E, McMahon C, Matthey S, Vimpani G, Anderson T, et al. Benefits of psychosocial intervention and continuity of care by child and family health nurses in the pre- and postnatal period: process evaluation. J Adv Nurs. 2013;69(8):1850–61. doi:10.1111/jan.12052.
Kreger M, Sargent K, Arons A, Standish M, Brindis CD. Creating an environmental justice framework for policy change in childhood asthma: a grassroots to treetops approach. Am J Public Health. 2011;101 Suppl 1:S208–16. doi:10.2105/ajph.2011.300188.
Levinson-Johnson J. Monroe Country ACCESS Logic Model. Tools for creating and sustaining human services: theory of change logic models: University of South Florida 2012
Levison-Johnson J, Wenz-Gross M. From complexity to reality: providing useful frameworks for defining systems of care. Eval Program Plann. 2010;33(1):56–8. doi:10.1016/j.evalprogplan.2009.05.005.
Lund C, Tomlinson M, de Silva M, Fekadu A, Shidhaye R, Jordans M, et al. PRIME: a programme to reduce the treatment gap for mental disorders in five low- and middle-income countries. PLoS Med. 2012;9(12):e1001359. doi:10.1371/journal.pmed.1001359.
Macfarlane F, Greenhalgh T, Humphrey C, Hughes J, Butler C, Pawson R. A new workforce in the making?: a case study of strategic human resource management in a whole-system change effort in healthcare. J Health, Organ Manag. 2011;25(1):55–72. doi:10.1108/14777261111116824.
Mackenzie M. Benefit or burden: introducing paraprofessional support staff to health visiting teams: the case of Starting Well. Health Soc Care Community. 2006;14(6):523–31. doi:10.1111/j.1365-2524.2006.00640.x.
Mackenzie M, Blamey A. The practice and the theory: lessons from the application of a theories of change approach. Evaluation. 2005;11(2):151–68. doi:10.1177/1356389005055538.
Mackenzie M, Blamey A, Halliday E, Maxwell M, McCollam A, McDaid D, et al. Measuring the tail of the dog that doesn’t bark in the night: the case of the national evaluation of Choose Life (the national strategy and action plan to prevent suicide in Scotland). BMC Public Health. 2007;7:146. doi:10.1186/1471-2458-7-146.
Mackenzie M, Reid M, Turner F, Wang YY, Clarke J, Sridharan S, et al. Reaching the hard-to-reach: conceptual puzzles and challenges for policy and practice. J Soc Policy. 2012;41(03):511–32. doi:10.1017/S0047279412000074.
Maselli B. Thrive community youth logic model: working together for a brighter tomorrow Lewiston, Maine. Tools for Creating and Sustaining Human Services: Theory of Change Logic Models: University of South Florida 2012
Mason P. Visual data in applied qualitative research: lessons from experience. Qual Res. 2005;5(3):325–46. doi:10.1177/1468794105054458.
McQuiston C, Choi-Hevel S, Clawson M. Protegiendo nuestra comunidad: empowerment participatory education for HIV prevention. J Transcult Nurs. 2001;12(4):275–83.
Mookherji S, LaFond A. Strategies to maximize generalization from multiple case studies: lessons from the Africa Routine Immunization System Essentials (ARISE) project. Evaluation. 2013;19(3):284–303. doi:10.1177/1356389013495212.
Morilus-Black M, McCarthy V, Kernan JB. Using a logic model as a tool to apply a theory of change approach to support implementation, evaluation and strategic planning. Tools for creating and sustaining human services: theory of change logic models: University of South Florida 2012
Reid M, Botma Y. A framework to expand public services to children with biomedical healthcare needs related to HIV in the Free State, South Africa. Afr J Aids Res. 2012;11(2):91–8. doi:10.2989/16085906.2012.698054.
Riley AJ, Byng R, White C, Smith S. Utilising theories of change to understand the engagement of general practitioners in service improvement: a formative evaluation of the Lewisham Depression Programme. Qual Prim Care. 2008;16(1):17–26.
Rivera JA, Martorell R, Gonzalez W, Lutter C, Cossio TG, Flores-Ayala R, et al. Prevención de la desnutrición de la madre y el niño: el componente de nutrición de la Iniciativa Salud Mesoamérica 2015. Salud Publica Mex. 2011;53 Suppl 3:S303–11.
Rodriguez MH, Betanzos-Reyes AF. Grupo de Trabajo de Malaria del Sistema Mesoamericano de Salud P. Plan de mejoramiento del control de la malaria hacia su eliminación en Mesoamérica. Salud Publica Mex. 2011;53 Suppl 3:S333–48.
Scanlon DP, Beich J, Alexander JA, Christianson JB, Hasnain-Wynia R, McHugh MC, et al. The aligning forces for quality initiative: background and evolution from 2005 to 2012. Am J Manag Care. 2012;18:S115–25.
Schierhout G, Hains J, Si D, Kennedy C, Cox R, Kwedza R, et al. Evaluating the effectiveness of a multifaceted, multilevel continuous quality improvement program in primary health care: developing a realist theory of change. Implementation Sci. 2013;8(1):119. doi:10.1186/1748-5908-8-119.
Secker J, Bowers H, Webb D, Llanes M. Theories of change: what works in improving health in mid-life? Health Educ Res. 2005;20(4):392–401. doi:10.1093/her/cyg133.
Smith N, Barnes M. New jobs old roles—working for prevention in a whole-system model of health and social care for older people. Health Soc Care Community. 2013;21(1):79–87. doi:10.1111/j.1365-2524.2012.01089.x.
Suarez-Balcazar Y. Empowerment and participatory evaluation of a community health intervention: Implications for occupational therapy. OTJR Occup Participation Health. 2005;25(4):133–42.
Tran L. A theory of change for mental health advocacy: a case study Charities evaluation services; 2009. http://www.cesvol.org.uk/Resources/CharitiesEvaluationServices/Documents/kingsfundtoccasestudy-801-809.pdf. Accessed date 16 Jan 2014.
Tucker P, Liao Y, Giles WH, Liburd L. The REACH 2010 logic model: an illustration of expected performance. Prev Chronic Dis. 2006;3(1):A21.
Vander Stoep A, Williams M, Jones R, Green L, Trupin E. Families as full research partners: what’s in it for us? J Behav Health Serv Res. 1999;26(3):329–44.
Veerman JW, De Kemp RAT, Ten Brink LT, Slot NW, Scholte EM. The implementation of families first in the Netherlands: a one year follow-up. Child Psychiatry Hum Dev. 2003;33(3):227–44. doi:10.1023/a:1021456630670.
von dem Knesebeck O, Joksimovic L, Badura B, Siegrist J. Evaluation of a community-level health policy intervention. Health Policy. 2002;61(1):111–22.
Walker JS, Matarese M. Using a theory of change to drive human resource development for wraparound. J Child Family Studies. 2011;20(6):791–803. doi:10.1007/s10826-011-9532-6.
Weitzman BC, Mijanovich T, Silver D, Brecher C. Finding the impact in a messy intervention: using an integrated design to evaluate a comprehensive citywide health initiative. Am J Eval. 2009;30(4):495–514. doi:10.1177/1098214009347555.
Wenz-Gross W, DuBrino T. Central Massachusetts Communities of Care: applying a theory of change approach to support implementation, evaluation, and strategic planning. Tools for Creating and Sustaining Human Services: Theory of Change Logic Models: University of South Florida 2012.
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6, e1000097. doi:10.1371/journal.pmed.1000097.
Clarke DJ, Godfrey M, Hawkins R, Sadler E, Harding G, Forster A, et al. Implementing a training intervention to support caregivers after stroke: a process evaluation examining the initiation and embedding of programme change. Implementation Sci. 2013;8(1):96. doi:10.1186/1748-5908-8-96.
Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 2010;8(1):18.
Vandenbroucke JP, Von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. Ann Intern Med. 2007;147(8):W-163–94.
Möhler R, Köpke S, Meyer G. Criteria for Reporting the Development and Evaluation of Complex Interventions in healthcare: revised guideline (CReDECI 2). Trials. 2015;16(1):204.
Our sincere thanks to Solange Durao, Nicola Foster and Florian Breuer for providing comments on an earlier draft of this paper.
EB, CL and MDS are funded by UK aid from the UK Government, as part of the PRogramme for Improving Mental health carE (PRIME). However, the views expressed do not necessarily reflect the UK Government’s official policies. MDS is also funded by an LSHTM Career Fellowship along with LL by Grand Challenges Canada as part of the Mental Health Innovation Network. The funders had no role in the design, data collection, analysis or writing of this report.
The authors declare that they have no competing interests.
EB designed the study under the guidance of MDS and CL. The protocol for the review was developed by EB with input from LL, MDS and CL. EB and LL double screened all abstracts and full text papers. EB conducted the data extraction and drafted the manuscript. All authors gave feedback on the draft manuscript and read and approved the final manuscript
The use of Theory of Change to design, implement and evaluate Public Health Interventions: a systematic review protocol. (DOCX 202 kb)