Skip to main content
Download PDF
Download ePub

Mechanisms of implementing public health interventions: a pooled causal mediation analysis of randomised trials

Implementation Science volume 13, Article number: 42 (2018) Cite this article

Abstract

Background

The World Health Organization recommends that nations implement evidence-based nutritional guidelines and policies in settings such as schools and childcare services to improve public health nutrition. Understanding the causal mechanism by which implementation strategies exert their effects could enhance guideline implementation. The aim of this study was to assess the mechanisms by which implementation strategies improved schools and childcare services’ adherence to nutrition guidelines.

Methods

We conducted a mechanism evaluation of an aggregated dataset generated from three randomised controlled trials conducted in schools and childcare services in New South Wales, Australia. Each trial examined the impact of implementation strategies that targeted Theoretical Domains Framework constructs including knowledge, skills, professional role and identity, environmental context and resources. We pooled aggregated organisation level data from each trial, including quantitative assessments of the Theoretical Domains Framework constructs, as well as measures of school or childcare nutrition guideline compliance, the primary implementation outcome. We used causal mediation analysis to estimate the average indirect and direct effects of the implementation strategies and assessed the robustness of our findings to varying levels of unmeasured and unknown confounding.

Results

We included 121 schools or childcare services in the pooled analysis: 79 allocated to receive guideline and policy implementation strategies and 42 to usual practice. Overall, the interventions improved compliance (odds ratio = 6.64; 95% CI [2.58 to 19.09]); however, the intervention effect was not mediated by any of the four targeted Theoretical Domains Framework constructs (average causal mediation effects through knowledge = − 0.00 [− 0.05 to 0.04], skills = 0.01 [− 0.02 to 0.07], professional role and identity = 0.00 [− 0.03 to 0.03] and environmental context and resources = 0.00 [− 0.02 to 0.06]). The intervention had no significant effect on the four targeted Theoretical Domains Framework constructs, and the constructs were not associated with school or childcare nutrition guideline compliance. Potentially, this lack of effect could be explained by imprecise measurement of the mediators. Alternatively, it is likely that that the interventions were operating via alternative mechanisms that were not captured by the four Theoretical Domains Framework constructs we explored.

Conclusions

Even though public health implementation strategies led to meaningful improvements in school or childcare nutrition guideline compliance, these effects were not mediated by key targeted constructs of the Theoretical Domains Framework. Future research should explore the mechanistic role of other Theoretical Domains Framework constructs and evaluate system-level mechanisms informed by an ecological framework.

Trial registration

All trials were prospectively registered with the Australian New Zealand Clinical Trials Registry (ACTRN12613000543785. Registered 15/05/2013; ACTRN12614001148662. Registered 30/10/2014; ACTRN12615001032549. Registered 1/10/2015).

Peer Review reports

Background

Nutrition risk factors are the leading causes of the global disease burden [1]. Consequently, United Nations member states declared 2016–2025 as the decade of action on nutrition [2]. Dietary guidelines provide evidence-informed recommendations regarding the dietary patterns recommended for optimal health and well-being and to reduce the risk of dietary-related chronic diseases [3]. The World Health Organization has recommended that nations implement evidence-based nutritional guidelines and policies in settings such as schools and childcare services to improve public health nutrition [4]. School and childcare-based nutritional guidelines typically make recommendations regarding the types of foods and beverages that should be provided (or made available to children) and in quantity, variety and frequency [5, 6]. A considerable challenge to achieving such objectives is the limited evidence base regarding the effectiveness of implementation strategies [7, 8]. Reviews of strategies that aim to improve the implementation of nutrition policies in schools and childcare services have identified few trials and report strategies that achieved equivocal effects [9, 10].

Understanding the causal mechanism by which implementation strategies exert their effects can improve the impact of strategies to enhance guideline implementation [7]. Approaches to improve policy or practices consistent with guideline recommendation are often multi-strategic and target a range of intermediary factors (or mediators) that are hypothesised to be causally be linked to successful implementation [7, 11]. The effectiveness of such strategies may be improved by retaining (or strengthening) strategies that target mediators which cause improvements in guideline implementation. Strategies could also be refined by discarding intervention components that target mediators that do not cause improvements in guideline implementation, or those that fail to shift important mediators [7, 12]. Despite the importance of understanding mechanisms of effect, few mechanistic evaluations of implementation strategies exist, and to our knowledge, none have been conducted on trials of strategies to improve implementation of nutrition guidelines.

Recent methodological advances have developed robust analytical techniques to quantify the extent to which of intervention effects are channelled through selected mediating variables. These new methods are based on clearly outlined counterfactual definitions of causal effects along with explicit assumptions required for making causal inferences [13,14,15]. However, conducting mechanistic evaluations of pragmatic trials in settings such as schools and childcare services is particularly challenging as guideline implementation typically occurs at the organisational level. Often, an insufficient sample of organisations is recruited to allow for sufficient power to undertake mediation analyses. There is also a lack of agreement as to which constructs should be tested as possible mediators of implementation and how they should be measured [16].

In this exploratory study, we aimed to overcome power limitations by aggregating data from three homogeneous implementation trials in schools and childcare services and used a theory-driven consensus approach to identify key constructs that could plausibly mediate the effects of implementation strategies on nutrition policy uptake. Thus, the overarching objective of this study was to quantify the extent to which selected Theoretical Domains Framework (TDF) constructs mediate the effect of implementation strategies on nutrition policy uptake in schools and childcare services.

Methods

Design and data sources

We conducted a mechanism evaluation by aggregating data from three homogeneous randomised controlled trials. The primary aims of all three trials were to increase compliance with state-wide nutrition guidelines regarding the nutritional quality of foods offered to children via a school canteen or provided as part of a food service at childcare. All trials were conducted in the Hunter New England Region of New South Wales, Australia, by the Hunter New England Population Health Research Group, a partnership between the University of Newcastle and a government population health unit [17]. Each trial randomly allocated schools or childcare services to receive a multicomponent complex implementation intervention or usual care. The outcomes of two trials (CAFÉ [18] and SNACS [19]) are published, and one trial (BMI [20]) has closed data collection. For each trial, independent reviewers assessed their quality using the Cochrane Risk of Bias Tool [21] (Additional file 1). Key characteristics of all three trials are described in Table 1. Detailed information about each trial are available in published reports and trial protocols [18,19,20].

Table 1 Characteristics of trials
Full size table

Ethical approval to conduct this study was obtained from Hunter New England Human Research Ethics Committee (ref. no. 06/07/26/4.04), University of Newcastle (ref. no. H-2008-0343), and New South Wales (NSW) Department of Education (SERAP 2012277). All trials were prospectively registered with the Australian New Zealand Clinical Trials Registry (ACTRN12613000543785, ACTRN12614001148662, and ACTRN12615001032549).

Selection of putative mechanisms via the Theoretical Domains Framework

All three trials targeted a range of implementation barriers and enablers outlined by the TDF ––with the hypothesis that addressing these factors would improve guideline implementation. The TDF is an integrative theoretical framework that incorporates 33 theories of behaviour change [22]. The framework includes the following 14 constructs: knowledge; skills; social/professional role and identity; beliefs about capabilities; optimism; beliefs about consequences; reinforcement; intentions; goals; memory, attention and decision processes; environmental context and resources; social influences; behavioural regulation. Detailed definitions of these constructs are reported by Cane et al. [22].

Full details regarding the strategy development process for each trial are described in detail in the respective manuscripts [18,19,20]. Briefly, in each trial, the development of the implementation strategy was undertaken by a multidisciplinary group including implementation and behavioural scientists, health promotion practitioners, dieticians and those with ‘setting’ (school and childcare service) expertise and followed formative evaluation including literature reviews, quantitative surveys and setting-based observations to identify factors (barriers and enablers). In the SNACS [19] and BMI [20] trial, mapping processes were then undertaken to select implementation strategies that could be employed to address those barriers (or facilitators) to guideline implementation using the TDF strategy selection matrix and further refined following consultations with stakeholders and consideration of project resources and feasibility. The selection of specific Canteen Manager behaviour change techniques in the CAFÉ trial [18] was based on Control Theory [23]. This theory was used to select key behaviour change techniques that would target knowledge gaps and skill barriers.

The constructs targeted by the implementation strategies varied across trials. Thus, for the purposes of testing the most plausible and likely causal mechanisms that would explain how the implementation strategies worked, and limiting the inflation of familywise (type 1) error rate in the aggregate analysis [24], we used a ranking and consensus approach to select four key TDF constructs that converged across all three trials. To do this, we asked a lead investigator from each trial to map and rank all 14 TDF constructs from ‘most likely mechanism’ to ‘least likely mechanism’. All lead investigators participated in each phase of the research project, including the implementation strategy development, stakeholder consultation and implementation strategy delivery. This process was completed independently across trials by three separate trial leads and without any post-hoc knowledge about the intervention effect on any of the TDF constructs. We did this so that the analysis was completely a theory and not data-driven. The independently ranked items were then combined by HL (who was not involved in the planning or conduct of all three trials) in an excel document, and the first four TDF constructs that converged across all three trials were selected for analysis. We did this to avoid the analysts (HL and AH) from having any control over the selection of the TDF domains.

Measures

We harmonised the outcomes from each trial by computing a binary variable that captured whether a school or childcare centre improved or did not improve in their implementation of nutrition guidelines from baseline [5]. In all trials, canteen managers or childcare service cooks (those primarily responsible for guideline implementation) completed a quantitative TDF survey that assessed all constructs. The survey was adapted from previous TDF instruments to suit the schools and childcare context and was independently validated [25] (Additional file 2). From these surveys, we used measures of four key TDF constructs to represent the primary mediators in this aggregate analysis, including knowledge (5 items), skills (3 items), professional role and identity (3 items), and environmental resources (7 items). We averaged the TDF scores for each construct and computed a standardised score. All TDF constructs and outcomes were measured at the final endpoint of each trial.

Construction of causal models

We specified four independent single mediator models with knowledge, skills, professional role and identity, and environmental resources as hypothesised mediators of the intervention effect on policy uptake (directed acyclic graphs shown in Fig. 1). In each model, we assumed that the intervention-mediator and intervention-outcome paths would be unconfounded because of random allocation of treatment [26]. However, for the mediator-outcome effect, we made an explicit assumption that this path could be confounded by unmeasured or unknown confounders. We also assumed that the four mediators were independent of one another [27]. Although it is plausible that the mediators are causally related (for example, better knowledge could increase sense of professional role and identity), given the exploratory nature of this analysis, we decided not to account for intertwined meditators and assumed independence between mediators. Because the causal effect of the mediator on the outcome could depend on the intervention status, we included an intervention-mediator interaction term into the models.

Fig. 1
figure 1

Directed acyclic graph of hypothesised mechanisms. Blue arrows = average causal mediation effect, black arrow = average direct effect, green arrow = intervention-mediator interaction, red arrows = confounding effects. Here we assume that the four mediators are independent of one another

Full size image

Statistical analysis

We conducted all analyses in R (The R Foundation for Statistical Computing) using the ‘mediation’ package (version 4.4.6) [28]. We used causal mediation analysis under the counterfactual framework to analyse the aggregate data [13]. For each of the four models, we estimated the average causal mediation effect (ACME), average direct effect (ADE) and the average total effect (ATE). The ACME is the effect of the intervention on the outcome exerted through the hypothesised mediator. The ADE represents the remaining effect of the intervention on the outcome that is not exerted through the selected mediator. Thus, the sum of the ACME and ADE equals the ATE. The proportion of the ATE that is channelled through the mediator (ACME) is termed the proportion mediated.

To analyse each causal model, we fit two regression models: the mediator model and the outcome model. We fit the mediator model using linear regression, specifying intervention allocation and trial ID as independent variables and the mediator as the dependent variable. We then fit the outcome model using binomial probit regression, specifying intervention allocation, trial ID, and mediator as independent variables, and compliance to nutrition policies or guidelines as the dependent variable. In the outcome model, we accounted for the possibility of an intervention-mediator interaction by including the product of intervention allocation and selected mediator into the regression models. To obtain unstandardised point estimates of the quantities of interest (ACME, ADE, ATE), we passed the mediator and outcome models through the mediate command with 1000 bootstrapped simulations [28].

We conducted the primary analysis on complete cases. However, as 16% (n = 19) of the primary outcome was missing, we conducted a post hoc sensitivity analysis by using Multiple Imputation by Chained Equations (MICE) [29] to assess the possible impact of missing data. We imputed 20 datasets with 50 iterations and used the bootstrap method to estimate standard errors. We used predictive mean matching to impute continuous variables (TDF domain scores) and logistic regression to impute the binary variable (primary outcome). All four TDF domains, trial, intervention group and the outcome were included in the imputation model. We pooled the estimates and standard errors using Rubin’s rule and calculated 95% CIs as outlined by Carlin et al. [30].

We examined violations in the assumption of normality for the linear regression models (mediator models) via visual inspections of the residual histogram and normal quantile (Q-Q) plots. If normality was violated, we transformed the TDF-construct variable using a BoxCox transformation using the log-likelihood procedure to obtain the optimal power coefficient (lambda) for transformation [31]. We then conducted sensitivity analyses to compare the results from the mediation analysis using transformed TDF-construct variables against the results from the original planned analyses.

Sensitivity analysis for sequential ignorability

In a single mediator model, we cannot assume that the mediator-outcome effect is un-confounded because the mediator is not randomised [32]. Thus, to explore the robustness of the ACME to violation of this assumption (sequential ignorability), we conducted sensitivity analyses. The level of confounding due to unknown and unmeasured confounders is represented by the correlation between the residuals from the mediator and outcome regression models (σ). We explored if changing the levels of σ between − 1 and 1 would influence the ACME and plotted the results.

Post hoc power calculation

To gain a general appreciation for the required sample size to detect an indirect effect through the TDF constructs in single mediator models, we used the sample size estimator for joint indirect effects developed by Vittinghoff and Neilands [33]. With a two-sided alpha of 0.05, exposure-mediator error term correlation coefficient of 0, and mediator-outcome error term correlation coefficient of 0.1, a sample of 121 provides 80% power to detect a proportion-mediated of 50%, with meaningful treatment-mediator (r = 0.5) and mediator-outcome (r = 0.4) effects. This post hoc power calculation provides indication that the pooled analysis would be powered to detect an indirect effect that consists of moderate treatment-mediator and mediator-outcome effects.

Results

Descriptive results

In the aggregate dataset (n = 121), a total of 42 organisations were randomised to a control group and 79 to an intervention group. The CAFÉ trial contributed 14 control and 23 intervention organisations, SNACS contributed 15 control and 21 intervention organisations and BMI contributed 13 control and 35 intervention organisations. List-wise deletion removed nine (21%) observations from the control group and 10 (11%) observations from the intervention group due to missing data. Nineteen observations were missing for the outcome, and one observation was missing for the TDF domain “Professional Role and Identity” and one from “Environmental context and resources.” In the control group, 8 organisations improved practice and 25 did not. In the intervention group, 44 organisations improved practice and 25 did not. These data stratified by trial are presented in Fig. 2.

Fig. 2
figure 2

Count of organisations that improved or did not change practice, stratified by trial

Full size image

Key findings

Overall, the intervention group had higher odds of improving adherence to nutritional policy when compared to the control group (odds ratio = 6.64; 95% CI [2.58 to 19.09]). Mediation analyses showed that none of the four TDF constructs explained how the interventions improved policy implementation. This is represented by the small and non-significant ACME and proportion mediated. The analyses showed that most of the intervention effect (ATE) is left unexplained––indicated by significant ADEs that represent unspecified mechanisms. The decompositions of these effects and their precision estimates are shown in Fig. 3.

Fig. 3
figure 3

Effect decomposition plots for each mediator model. ACME average causal mediation effect, ADE average direct effect. Solid dots and lines represent point estimates and 95% confidence limits for the intervention group; the hollow dots and broken lines represent point estimates and confidence limits for the usual care group. The total effect is displayed as an average effect. All effects are reported unstandardized with their 95% confidence intervals

Full size image

The regression models for the intervention-mediator path showed that the interventions had a negligible and non-significant effect on all four TDF constructs (second column, Table 2). The regression models for the mediator-outcome path also showed that all four TDF constructs had negligible and non-significant effects on guideline implementation (third column, Table 2). These results indicate that our hypothesised causal mechanism broke down at both the intervention-mediator pathway (action theory) and mediator-outcome pathway (conceptual theory).

Table 2 Effect decomposition for four TDF constructs as hypothesised mediators
Full size table

Pooled estimates obtained from the imputed datasets were similar to the results obtained from the complete case analysis. The results are presented in Additional file 3. We observed violations of normality in three linear regression models assessing the following TDF constructs: knowledge, skills, and environmental context and resources. To overcome violations of normality, we used a BoxCox transformation on these three TDF constructs and conducted sensitivity analyses of the mediation models using the transformed variables. We did not observe extreme deviations in the estimates from the sensitivity analyses when compared with the original analyses that used percentage scores of the TDF constructs. The results of the sensitivity analyses are presented in Additional file 4.

The sensitivity analyses for the sequential ignorability assumption indicated that the ACME estimates were robust. All ACMEs remained stable across low to high levels of unknown and unmeasured confounding (Fig. 4).

Fig. 4
figure 4

Sensitivity plots. The average mediation effects are plotted as a function of the sensitivity parameter (magnitude of residual confounding). A sensitivity parameter of 0 represents null hypothesised levels of residual confounding and the extremes of − 1 and 1 represent maximum hypothesised levels of residual confounding. Grey zones represent 95% confidence limits of the estimated mediation effect across a range of hypothesised levels of residual confounding

Full size image

Discussion

This exploratory study sought to identify the mechanisms by which implementation strategies improve nutrition guideline implementation in schools and childcare services. Our aggregate causal mediation analysis found that although the implementation strategies increased adherence to policy and guidelines, none of the theoretically targeted factors (knowledge, skills, professional role and identity, and environmental resources) mediated this effect.

The implementation strategies evaluated in the included trials targeted a range of implementation barriers that were not analysed in this study. It is possible that the interventions were operating via mechanisms that were not captured by the four TDF constructs we explored. For example, previous studies have reported a range of factors associated with successful implementation of healthy eating interventions in childcare––including the support of parents, service management, and the structural availability of time and resources [34]. These factors could have been the active mechanisms of the interventions evaluated in this study but were not captured by the four TDF constructs we explored. Previous work has posited that organisational level change often involves the interplay between many individuals, group and environmental factors [35]. Future research should consider investigating the casual mechanisms that work through system-level factors that could impact the implementation of nutritional policies within childcare services and schools. An ecological systems framework that considers the complex interactions between individuals and their social structures may be appropriate to guide such investigations [36]. Furthermore, the use of the Control Theory in the CAFÉ trial may have caused the intervention to work through mechanisms that were not captured by the TDF.

Another explanation for our findings is that we may have failed to measure the targeted TDF constructs with adequate precision. While the TDF questionnaire used in this analysis has some evidence of validity and reliability [20], it has some limitations. For instance, only one of the three goodness-of-fit statistics from the original confirmatory factor analysis met acceptable criteria [25], which indicates limitations in construct validity. Furthermore, all four constructs of the TDF were negatively skewed for both control and the intervention groups, with most organisations reporting high scores (i.e. low levels of barriers) at follow-up. The skew towards high scores on all four TDF constructs for both groups may reflect possible ceiling effects. If ceiling effects are present, it is possible that our measure was not sensitive enough to detect between group differences, as the measure cannot distinguish respondents at the upper end of the construct [37]. Ceiling effects can attenuate statistical associations, thus resulting in a possible underestimation in the relationship between variables [38, 39]. This may explain why we did not detect a mediating effect through these variables. Future work should seek to make improvements in the current TDF measures to allow for greater sensitivity to detect underlying mechanisms. Possible strategies that may be considered to help reduce the ceiling effects and increase the response variability of the TDF measure could include (i) using more extreme response options at the positive end of the scale, which could help differentiate people who score high. This strategy has been suggested to reduce ceiling effects in other surveys [40, 41], (ii) including survey items that assess respondents’ actual behaviour rather than self-perceived behaviour. For example, rather than asking respondents to indicate whether they are aware of guideline content, it would be more precise to specifically assess the respondents’ actual knowledge of the guideline content. This should help provide a more objective and standardised assessment of barriers.

A strength of this study is the use of pooled data from three relatively large randomised trials in the public health nutrition setting. This was possible through the aggregation of a homogenous collection of trials (same geographic location, similar interventions, similar population and matched target behaviour). This is a key strength as it is often difficult to collect large organisation-level datasets for mechanistic evaluations. Many studies have used the TDF to guide the development of interventions; however, no study has quantitatively tested the TDF constructs as causal mechanisms to refine future implementation strategies [42]. Building an evidence base for the mechanistic role of the TDF constructs will assist in future intervention design and adaptation [7, 12, 42]. Research partnerships between health organisations and clinical trial units should employ similar approaches to conduct mechanism-focused implementation studies. Planning and executing a concerted set of trials that assess similar implementation mechanisms could yield robust evidence for how implementation strategies work or do not work. These techniques can and should be applied across various settings for better implementation of preventive and healthcare strategies.

Our findings should also be interpreted in the context of its limitations. We were unable to adjust for any confounders of the mediator-outcome effect. However, our sensitivity analysis indicated that our estimations of the mediation effect would remain stable even at high levels of residual confounding. We may have measured the TDF constructs with error and poor precision. It is possible that the questionnaire used to assess the TDF constructs was limited in its construct validity and displayed ceiling effects. The temporal precedence between the mediator, and outcome is unclear in our analyses. Future work should aim to measure the mediator prior to the outcome and assess the possibility of reverse causation. The trial participants may have felt under pressure to report excellent knowledge and skills after training (social desirability bias). However, given that we did not detect between group differences on knowledge and skills, the likelihood of this bias is low. Lastly, 19 (16%) organisations were lost to follow-up, and this may have induced bias if the missingness mechanism was not at random.

Conclusions

Understanding the casual mechanisms of a complex implementation strategy can inform the development and adaptation of future strategies, as well as provide an opportunity to understand and improve the theoretical underpinnings of implementation science. As the first study of its kind in this setting, an aggregated causal mediation analysis of three complex intervention trials that aimed to improve the implementation of nutritional guidelines across Australian primary schools and childcare services showed that none of the four hypothesised TDF constructs were meaningful mediators. Future research should employ similar methods and techniques to explore the mechanistic role of other TDF constructs and evaluate system-level mechanisms informed by an ecological framework.

Abbreviations

ACME:

Average causal mediation effect

ADE:

Average direct effect

ATE:

Average total effect

TDF:

Theoretical Domains Framework

References

  1. Lim SS, Vos T, Flaxman AD, Danaei G, Shibuya K, Adair-Rohani H, et al. A comparative risk assessment of burden of disease and injury attributable to 67 risk factors and risk factor clusters in 21 regions, 1990–2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet [Internet]. 2012;380:2224–60. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4156511/.

    Article  Google Scholar 

  2. World Health Organization. General Assembly proclaims the Decade of Action on Nutrition [Internet]. 2016. Available from: http://www.who.int/nutrition/GA_decade_action/en/. Accessed 3 Nov 2017.

  3. NHMRC. Australian Dietary Guidelines [Internet]. Natl Heal Med Res Counc. 2013. Available from: www.nhmrc.gov.au/guidelines-publications/n55. Accessed 3 Nov 2017.

  4. World Health Organization. Ambition and Action in Nutrition 2016-2025 [Internet]. 2017. Available from: http://www.who.int/nutrition/publications/nutrition-strategy-2016to2025/en/. Accessed 3 Nov 2017.

  5. NSW Department of Health, NSW Department of Education and Training. Fresh Tastes @ School NSW healthy school canteen strategy: canteen menu planning guide. 3rd ed. Sydney: NSW Department of Health, NSW Department of Education and Training; 2012.

  6. NSW Ministry of Health. Caring for children. 2014.

    Google Scholar 

  7. Geng EH, Peiris D, Kruk ME. Implementation science: relevance in the real world without sacrificing rigor. PLoS Med. 2017;14:1–5.

    Google Scholar 

  8. Sanders D, Haines A. Implementation research is needed to achieve international health goals. PLoS Med. 2006;3:0719–22.

    Article  Google Scholar 

  9. Wolfenden L, Jones J, Cm W, Finch M, Rj W, Kingsland M, et al. Strategies to improve the implementation of healthy eating, physical activity and obesity prevention policies, practices or programmes within childcare services. Cochrane Database Syst Rev [Internet]. 2016; Available from: http://www.crd.york.ac.uk/prospero/display_record.asp?src=trip&ID=CRD42015028090. Accessed 16 Nov 2017.

  10. Rabin BA, Glasgow RE, Kerner JF, Klump MP, Brownson RC. Dissemination and implementation research on community-based cancer prevention: a systematic review. Am J Prev Med Netherlands. 2010;38:443–56.

    Article  Google Scholar 

  11. Anselmi L, Binyaruka P, Borghi J. Understanding causal pathways within health systems policy evaluation through mediation analysis: an application to payment for performance (P4P) in Tanzania. Implement Sci. 2017;12:10.

  12. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ [Internet]. 2015;350:h1258. Available from: http://www.bmj.com/cgi/doi/10.1136/bmj.h1258.

    Article  Google Scholar 

  13. Imai K, Keele L, Tingley D. A general approach to causal mediation analysis. Psychol Methods [Internet]. 2010 [cited 2013 Sep 23];15:309–34. Available from: http://www.ncbi.nlm.nih.gov/pubmed/20954780.

  14. VanderWeele T. Explanation in causal inference: methods for mediation and interaction. UK: Oxford University Press; 2015.

  15. Dunn G, Emsley R, Liu H, Landau S, Green J, White I, et al. Evaluation and validation of social and psychological markers in randomised trials of complex interventions in mental health. NIHR Rep. 2015;19(93):1–115.

  16. Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci [Internet]. 2013;8:20. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3598720&tool=pmcentrez&rendertype=abstract.

  17. Wolfenden L, Yoong SL, Williams C, Grimshaw J, Durrheim DN, Gillham K, et al. Embedding researchers in health service organizations improves research translation and health service performance: The Australian Hunter New England Population Health example. J Clin Epidemiol [Internet]. Elsevier Inc.; 2017;0. Available from: http://www.ncbi.nlm.nih.gov/pubmed/28341367.

  18. Yoong SL, Nathan N, Wolfenden L, Wiggers J, Reilly K, Oldmeadow C, et al. CAFÉ: a multicomponent audit and feedback intervention to improve implementation of healthy food policy in primary school canteens: a randomised controlled trial. Int J Behav Nutr Phys Act [Internet]. 2016;13:126. Available from: http://ijbnpa.biomedcentral.com/articles/10.1186/s12966-016-0453-z.

  19. Nathan N, Yoong SL, Sutherland R, Reilly K, Delaney T, Janssen L, et al. Effectiveness of a multicomponent intervention to enhance implementation of a healthy canteen policy in Australian primary schools: a randomised controlled trial. Int J Behav Nutr Phys Act [Internet]. 2016;13:106. Available from: http://ijbnpa.biomedcentral.com/articles/10.1186/s12966-016-0431-5.

  20. Seward K, Wolfenden L, Finch M, Wiggers J, Wyse R, Jones J, et al. Multistrategy childcare-based intervention to improve compliance with nutrition guidelines versus usual care in long day care services: a study protocol for a randomised controlled trial. BMJ Open [Internet]. 2016;6:10786. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27301484.

  21. Higgins JPT, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman a. D, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. Bmj [Internet]. 2011;343:d5928. Available from: http://www.bmj.com/cgi/doi/10.1136/bmj.d5928.

  22. Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:37.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Gardner B, Whittington C, McAteer J, Eccles MP, Michie S. Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Soc Sci Med. 2010;70:1618–25.

    Article  PubMed  Google Scholar 

  24. Pocock SJ, Geller NL, Tsiatis AA. The analysis of multiple endpoints in clinical trials. Biometrics JSTOR. 1987:487–98.

  25. Seward K, Wolfenden L, Wiggers J, Finch M, Wyse R, Oldmeadow C, et al. Measuring implementation behaviour of menu guidelines in the childcare setting: confirmatory factor analysis of a theoretical domains framework questionnaire (TDFQ). Int J Behav Nutr Phys Act [Internet]. 2017;14:45. Available from: http://ijbnpa.biomedcentral.com/articles/10.1186/s12966-017-0499-6.

  26. Emsley R, Dunn G, White IR. Mediation and moderation of treatment effects in randomised controlled trials of complex interventions. Stat Methods Med Res. 2010;19:237–70.

    Article  PubMed  Google Scholar 

  27. Daniel RM, De Stavola BL, Cousens SN, Vansteelandt S. Causal mediation analysis with multiple mediators. Biometrics [Internet]. 2015 [cited 2014 Oct 31];71:1–14. Available from: http://www.ncbi.nlm.nih.gov/pubmed/25351114.

  28. Tingley D, Yamamoto T, Hirose K, Keele L, Imai K. Mediation: R package for causal mediation analysis. J Stat Softw. 2014;59:1–38.

    Article  Google Scholar 

  29. van Buuren S, Groothuis-Oudshoorn K. Mice: multivariate imputation by chained equations in R. J Stat Softw. University of California, Los Angeles; 2010;1–68.

  30. Carlin JB, Li N, Greenwood P, Coffey C. Tools for analyzing multiple imputed datasets. Stata J. 2003;3:1–20.

    Google Scholar 

  31. Box GEP, Cox DR. An analysis of transformations revisited, rebutted. J Am Stat Assoc [Internet]. 1982;77:209. Available from: http://www.jstor.org/stable/2287791?origin=crossref.

  32. Keele L. Causal mediation analysis: warning! Assumptions ahead. Am J Eval. 2015;36:500–13.

    Article  Google Scholar 

  33. Vittinghoff E, Neilands TB. Sample size for joint testing of indirect effects. Prev Sci [Internet]. 2014 [cited 2014 Nov 28];16:1128–35. Available from: http://www.ncbi.nlm.nih.gov/pubmed/25418811.

  34. Wolfenden L, Finch M, Nathan N, Weaver N, Wiggers J, Yoong SL, et al. Factors associated with early childhood education and care service implementation of healthy eating and physical activity policies and practices in Australia: a cross-sectional study. Transl Behav Med. 2015;5:327–34.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Bonell CP, Fletcher A, Jamal F, Wells H, Harden A, Murphy S, et al. Theories of how the school environment impacts on student health: systematic review and synthesis. Health Place. 2013;24:242–9.

    Article  CAS  PubMed  Google Scholar 

  36. Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol. 2009;43:267–76.

    Article  PubMed  Google Scholar 

  37. Terwee CB, Bot SDM, de Boer MR, van der Windt DAWM, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60:34–42.

    Article  PubMed  Google Scholar 

  38. Huang I-C, Frangakis C, Atkinson MJ, Willke RJ, Leite WL, Vogel WB, et al. Addressing ceiling effects in health status measures: a comparison of techniques applied to measures for people with HIV disease. Health Serv Res. Wiley Online Library; 2008;43:327–339.

  39. Wang L, Zhang Z, McArdle JJ, Salthouse TA. Investigating ceiling effects in longitudinal data analysis. Multivariate Behav Res. Taylor & Francis; 2008;43:476–496.

  40. Sirgy MJ, Samli AC. New dimensions in marketing/quality-of-life research. Greenwood Publishing Group; 1995.

  41. Vita S, Coplin H, Feiereisel KB, Garten S, Mechaber AJ, Estrada C. Decreasing the ceiling effect in assessing meeting quality at an academic professional meeting. Teach Learn Med. Taylor & Francis; 2013;25:47–54.

  42. Curran J, Brehaut J, Patey A, Osmond M, Stiell I, Grimshaw J. Understanding the Canadian adult CT head rule trial: use of the theoretical domains framework for process evaluation. Implement Sci [Internet]. 2013;8:25. Available from: http://jama.jamanetwork.com/article.aspx?doi=10.1001/jama.1997.03540300056034.

Download references

Acknowledgements

We thank Li Kheng Chai for providing risk of bias assessments of the included trials.

Funding

This project received funding from the Priority Research Centre for Health Behaviour, University of Newcastle Australia and Hunter New England Population Health Wallsend Australia. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. HL is funded by the Australian National Health and Medical Research Council (grant no. APP1126767); receives project funding from the Berkeley Initiative for Transparency in the Social Sciences, a program of the Center for Effective Global Action (CEGA), with support from the Laura and John Arnold Foundation. HL is supported by funding from the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care Oxford at Oxford Health NHS Foundation Trust.

Availability of data and materials

The datasets and code used in the current study are available from the corresponding author on reasonable request.

Author information

Authors and Affiliations

  1. Centre for Statistics in Medicine, Nuffield Department of Orthopaedics Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, United Kingdom

    Hopin Lee

  2. School of Medicine and Public Health, University of Newcastle, Newcastle, New South Wales, Australia

    Hopin Lee, Alix Hall, Nicole Nathan, Kathryn L. Reilly, Kirsty Seward, Christopher M. Williams, Serene Yoong, Meghan Finch, John Wiggers & Luke Wolfenden

  3. Neuroscience Research Australia, Sydney, New South Wales, Australia

    Hopin Lee

  4. Hunter Medical Research Institute, Newcastle, New South Wales, Australia

    Alix Hall, Christopher M. Williams, Serene Yoong, John Wiggers & Luke Wolfenden

  5. Hunter New England Population Health, Newcastle, New South Wales, Australia

    Nicole Nathan, Kathryn L. Reilly, Kirsty Seward, Christopher M. Williams, Serene Yoong, Meghan Finch, John Wiggers & Luke Wolfenden

Authors
  1. Hopin Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alix Hall
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicole Nathan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kathryn L. Reilly
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kirsty Seward
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Christopher M. Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Serene Yoong
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Meghan Finch
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. John Wiggers
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Luke Wolfenden
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

HL, AH and LW conceived, planned and drafted this paper. NN, KS, KR and LW supplied the data. HL and AH conducted the analysis. HL, AH and LW wrote the first draft of the manuscript. All authors contributed to subsequent revisions and approved the final version.

Corresponding author

Correspondence to Hopin Lee.

Ethics declarations

Ethics approval and consent to participate

Ethical approval to conduct this study was obtained from Hunter New England Human Research Ethics Committee (ref. no. 06/07/26/4.04), University of Newcastle (ref. no. H-2008-0343) and New South Wales (NSW) Department of Education (SERAP 2012277).

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

Cochrane Risk of Bias Assessment. (DOCX 110 kb)

Additional file 2:

School survey questions and modified questions. (DOCX 35 kb)

Additional file 3:

Sensitivity analysis using multiple imputation to replace missing values. (DOCX 17 kb)

Additional file 4:

Sensitivity analysis using transformed TDF variables. (DOCX 60 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, H., Hall, A., Nathan, N. et al. Mechanisms of implementing public health interventions: a pooled causal mediation analysis of randomised trials. Implementation Sci 13, 42 (2018). https://doi.org/10.1186/s13012-018-0734-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13012-018-0734-9

Keywords

Implementation Science

ISSN: 1748-5908