- Open Access
Is fidelity to a complex behaviour change intervention associated with patient outcomes? Exploring the relationship between dietitian adherence and competence and the nutritional status of intervention patients in a successful stepped-wedge randomised clinical trial of eating as treatment (EAT)
Implementation Science volume 16, Article number: 46 (2021)
Behavioural medicine is characterised by findings for the effectiveness and efficacy of complex behaviour change interventions. Comparatively, scant attention has been paid to key intervention components or mechanisms of action. Evaluating relationships between process variables (fidelity) and intervention effects is central to addressing this imbalance. Accordingly, in the current study, we sought to explore the magnitude and direction of effect between fidelity predictors (dietitian adherence and competence) and intervention effects (patient nutritional status) during the intervention phase of a real-world, stepped-wedge evaluation of ‘EAT: Eating As Treatment’.
The EAT clinical trial was conducted within five major Australian hospitals located in Queensland, Western Australia, Victoria and South Australia between 2013 and 2016. EAT is a dietitian-delivered health behaviour change intervention designed to reduce malnutrition in head and neck cancer (HNC) patients undergoing radiotherapy. Dietitian adherence and competence ratings were derived from a 20% random sample of audio-recorded dietetic consultations (n=194) conducted after dietitians (n=18) were trained in EAT. Sessions were coded by trained, independent, coders using a study checklist, the Behaviour Change Counselling Index (BECCI) and items from the Cognitive Therapy Scale-Revised (CTS-R). Patient nutritional status was measured using the Patient-Generated Subjective Global Assessment (PGSGA).
Dietitian adherence to a written nutrition plan (β=7.62, 95% CI=0.65 to 14.58, p=0.032), dietitian adherence to behaviour change counselling (β=0.69, 95% CI =0.02 to 1.38, p=0.045) and competence in delivering behaviour change counselling (β=3.50, 95% CI =0.47 to 6.53, p=0.024) were significant predictors of patient nutritional status. Dietitian adherence and competence ratings were higher during consultations with intervention patients at greater risk of malnutrition.
This study contributes new insights into the relationship between fidelity and treatment outcome by demonstrating that dietitian adherence and competence is greater when working with more challenging patients. This is likely central to the demonstrated success of the EAT intervention in reducing malnutrition and highlights the importance of ensuring that providers are adequately equipped to flexibly integrate intervention elements according to patient need.
This study is a process analysis of a stepped-wedge randomised controlled trial prospectively registered on the Australian New Zealand Clinical Trials Registry (ACTRN12613000320752; Date of registration 21/03/2013).
Fidelity is a multi-component and multidimensional construct  that is relevant to the design, conduct, evaluation and reporting of behavioural interventions [2,3,4]. When applied to the delivery of an intervention, fidelity refers to whether an intervention was delivered as intended (integrity) and the degree to which it is distinguishable from comparison conditions (differentiation [2, 3]). Key concepts of integrity include adherence (the degree to which an intervention was delivered as intended) and competence (the skill with which the intervention was delivered ). Accurate interpretation of treatment effects relies, in part, on evidence regarding treatment fidelity [6,7,8]. For example, demonstrating that an intervention was delivered with fidelity improves the confidence with which observed changes in patient outcome can be attributed to the intervention under evaluation [2, 3, 8]. Rigorous fidelity assessment is therefore critical for enhancing the external validity of a study .
Cognitive behaviour therapy (CBT)  and/or motivational interviewing (MI)  are commonly used evidence-based approaches for promoting change across a range of health behaviours [12,13,14]. These psychological approaches to health behaviour change can be defined as ‘complex’ due to the multiple interacting components employed . Current understanding of complex behaviour change interventions is limited by inadequate consideration of treatment fidelity [16,17,18]. Systematic reviews and/or meta-analyses consistently demonstrate that fidelity assessment is rarely conducted [13, 19,20,21,22,23,24,25]. Furthermore, even when fidelity assessment is conducted, the resultant data (i.e. the level of adherence and/ or competence achieved) is rarely reported [20,21,22,23,24]. For example, in a recent review of MI for health behaviour change, of the 37 studies included, only 13 assessed fidelity and of those, more than half (53%) did not report the findings . This means that, for the vast majority of evaluations, the concordance between the intended intervention and what was actually delivered is unclear. This compromises current understanding of whether and how complex behaviour change interventions affect change in patient behaviour. Secondary analyses are therefore recommended to explore the relationship between important process variables, such as fidelity, and treatment outcomes .
Intuitively, it may be reasonable to hypothesise that the more faithful a clinician is to delivering an intervention ‘as intended’ (i.e. greater levels of adherence and/or competence) the more beneficial the intervention will be. However, evidence regarding the relationship between fidelity and outcome is limited and complex [27, 28]. For example, in a seminal review of the health prevention literature, of the 162 publications included, only five examined the relationship between adherence and treatment outcome, and the presence and direction of effect varied within and between studies . Similarly, adherence to the relational [21, 30, 31] and/ or technical [21, 31] components of MI has been linked to improved patient outcomes , worse patient outcomes  and no effect [21, 30, 31]. Greater attention to fidelity [1, 9], including the conduct of process analyses , is central to improving understanding of the relationship between fidelity and intervention effects.
Accordingly, we recently applied published recommendations [2,3,4] to the design and conduct of a comprehensive fidelity evaluation [32, 33] within a real-world multi-site stepped-wedge randomised controlled trial of ‘EAT: Eating as Treatment’. EAT is a complex health behaviour change intervention designed to reduce malnutrition in head and neck cancer (HNC) patients undergoing radiotherapy [34, 35]. Our findings demonstrated that the primary outcome of nutritional status (as measured by the Patient-Generated Subjective Global Assessment; PGSGA ) was superior for intervention relative to control participants (β = −1.53; confidence interval = −2.93 to −0.13 ). Furthermore, fidelity analysis  confirmed that (a) objective ratings of dietitian adherence and competence favoured intervention relative to control sessions , and (b) intervention sessions were clearly distinct regarding core motivational and behavioural intervention elements, whereas generic or ‘common’ intervention elements remained stable . In light of this evidence for both intervention effects and intervention fidelity, we now take the next step  in the evaluation of the EAT intervention and conduct a process analysis.
Objectives and importance
In the current paper, we explore the potential functional elements of EAT by examining the relationship between dietitian adherence and competence ratings and patient nutritional status in a randomly selected sample of intervention participants. Given the limited, and often contradictory nature of the literature, we make no predictions regarding the potential direction of effect. Rather, the current analysis is exploratory and aims to:
Examine the magnitude and direction of association between objective ratings of dietitian adherence to the EAT intervention and the nutritional status of intervention patients.
Examine the magnitude and direction of association between objective ratings of dietitian competence in delivering the EAT intervention and the nutritional status of intervention patients.
Setting and trial design
The current paper is derived from data collected during the intervention phase of the EAT clinical trial (ACTRN12614000876695; Date of registration April 21, 2013), conducted between 2013 and 2016 within five major Australian hospitals located in Queensland, Western Australia, Victoria and South Australia. This research was conducted in accordance with the National Statement on Ethical conduct in Human Research . Further information regarding study methods and findings are available in published protocol and outcome papers from the fidelity evaluation [32, 33] and overarching clinical trial [34, 35].
Inclusion criteria [34, 35] for participants in the EAT clinical trial were aged 18 years or older, pathologically confirmed diagnosis of HNC, undergoing radiotherapy (definitive or post-operative) with curative intent and receiving a prescribed dose of at least 60 Gy and nodal irradiation. Participants were required to provide written, informed consent and be available for follow-up for at least 6 months.
Eighteen oncology dietitians employed by the study sites were identified (via self and/or head of department selection) to undergo training in the EAT intervention. All intervention dietitians had attained bachelor level qualifications. Half (n=9) had completed postgraduate level training in nutrition and dietetics (including masters or postgraduate diploma in nutrition/dietetics). Experience working with HNC patients ranged from a few months to more than 20 years (mean = 3.38 years; mode = 1 year).
Training in EAT comprised a 2-day workshop, 1-day clinical ‘shadowing’, ongoing supervision and coaching (at least monthly) and a follow-up ‘booster’ workshop (comprising 1-day workshop, and 1-day shadowing). EAT training was delivered by the same facilitators (trial Clinical/Health Psychologists and authors BB, ALB and/or AKB) as part of an overall package designed to support systems level change .
The EAT intervention
To support flexible integration throughout dietetic consultations (i.e. according to clinical presentation and/or patient need), EAT is informed by motivational and behavioural principles (Fig. 1).
MI is used throughout consultations to elicit and reinforce patient reason(s) for maintaining adequate nutrition, the performance of helpful nutrition-related behaviours and to collaboratively identify pertinent nutrition-related goals. A written nutrition planner is used to simplify, document, review and reinforce patient goals. An objective measure of nutritional status is provided via the conduct of a standardised nutrition assessment.
The “EAT to LIVE” conversation (see Fig. 1) integrates the aforementioned behaviour change principles and strategies. This conversation requires that dietitians (i) elicit patient motivation for undergoing radiotherapy treatment (i.e., survival); (ii) explicitly highlight the relationship between adequate nutritional status and radiotherapy treatment outcomes; (iii) explore the (in) consistency between patient behaviour, motivation to survive, nutritional status and requirements for maintaining an adequate nutritional status; and (iv) work towards developing a concrete plan.
Assessment of fidelity of intervention delivery
Fidelity assessment in the EAT clinical trial was conducted using a randomly selected 20% sample of audio-recorded dietetic consultations conducted before (n=196) and after (n=194) training in EAT [32, 33]. In the current paper, we focus on the dietitian adherence and competence outcomes derived from the 194 randomly selected sessions conducted during the intervention phase of the EAT clinical trial. Audio-recordings were coded by an independent assessor (KB) blind to the schedule of training and intervention content. Each audio-recording was listened to from start to finish, and coding was based on the entire session. Inter-rater and intra-rater reliability was derived from a randomly selected 20% subsample of the coded intervention recordings (n=47). Inter-rater reliability was, on average, ‘substantial’  (0.61), with a range from 0.24 to 1.0. Overall intra-rater reliability for both rater one (KB; 0.96, range 0.92 to 1.0) and rater two (KM; 0.94, range 0.84 to 1.0) was ‘almost perfect’  (Supplementary Table 2).
Dietitian adherence to key elements of the EAT intervention was assessed using a checklist developed by the research team (see Table 1). Each of the items was rated as being present (‘yes’) or absent (‘no’).
The behaviour change Counselling index (BECCI)
The 11-item BECCI  was used to assess dietitian adherence to MI. Each item is rated on a five-point scale (0=not at all, 1=minimally, 2=to some extent, 3=a good deal, 4=a great extent) and used to calculate an overall ‘practitioner score’.
The cognitive therapy scale–revised (CTS-R)
To assess the degree to which dietitians embodied the ‘spirit’ of intervention delivery (empathy, genuineness and warmth), we used the ‘interpersonal effectiveness’ item from the CTS-R . This item is rated on a scale of 0 to 6, with higher ratings indicating greater expression of warmth, concern, confidence, genuineness and professionalism.
Due to the paucity of brief, validated tools available at study inception to assess competence in behaviour change counselling, the CBT competence item from the CTS-R  was modified to reflect competence in skills of behaviour change counselling. This item is rated on a scale of 0 to 6, with higher ratings indicating greater competence.
Primary patient outcome
Participant nutritional status was measured on four occasions (first and final week of radiotherapy, 4 and 12 weeks post-radiotherapy) using the Patient-Generated Subjective Global Assessment (PGSGA ). This instrument combines patient reported medical history (weight loss, nutrition impact symptoms, intake and functional capacity) and clinician examination (comprising diagnosis, age, metabolic stress and physical examination of fat, muscle stores and fluid status) . Items are scored based on their likely impact on nutrition. The PGSGA can be used to calculate a numerical score (0–35), with higher scores indicative of elevated risk of malnutrition .
Data analysis was conducted using the IBM Statistical Package for the Social Sciences (SPSS) for windows, version 26.0 (2019).
To allow for evaluation of within-dietitian, between-patients effects of fidelity on treatment outcome; adherence and competence scores assigned to each consultation were used to generate average adherence and competence ratings for each dietitian sampled during the intervention phase. For the study checklist, dietitian adherence scores for each item were generated by calculating the proportion of their consultations coded ‘yes’ for that skill. The total BECCI scores assigned to each consultation were averaged to generate a mean BECCI adherence score for each dietitian. Average scores for the CTS-R interpersonal effectiveness item and CTS-R competence item were also produced to generate mean ratings of interpersonal effectiveness and competence, respectively, for each dietitian. For each predictor variable, intervention participants were assigned the score of the dietitian they saw in the sampled consultation(s). When an individual participant had sessions sampled from multiple dietitians, individual dietitian scores were averaged to produce an overall score for each of the predictors.
Baseline characteristics of intervention participants (n=156) with at least one dietetic consultation randomly selected for inclusion in the fidelity sample (n=107) were compared to intervention participants not included in the fidelity sample (n=49) using independent samples t tests, chi-square or Fisher’s exact test, as appropriate. Linear mixed model regressions were used to analyse associations between the nine fidelity predictors and the dependent variable of patient nutritional status. Predictors were analysed separately to avoid interactions between them. The model utilised unstructured covariance and included the following: cluster level variable of site (to adjust for differences between hospitals), individual level variables of baseline PGSGA (to adjust for differences in baseline nutritional status), tumour site and stage (to adjust for any differences between participants’ nutritional status because of differences in malignancies), calendar time (to adjust for any background temporal effects that could have confounded the stepped-wedge design) and assessment interval (to adjust for differences in nutritional status through radiotherapy and recovery). The model included a random individual level intercept to account for the repeated measures on individuals over assessment moments and a random coefficient for assessment moment to allow for heterogeneity in subject specific trends. The alpha level was set to p=0.05.
During the intervention phase of the EAT clinical trial, the 156 intervention participants attended a total of 1568 dietetic consultations. The coded sample of dietetic consultations was conducted by the 18 trained dietitians and two locum dietitians. The mean number of sessions sampled per dietitian was 9.7 (range = 1 to 43). 107 intervention participants (68%) were represented in the fidelity sample. The average number of coded sessions per participant was 2 (range = 1–6). The demographic and clinical characteristics of patients included in the fidelity sample were largely comparable to those who were not sampled, with the exception of fewer radiotherapy fractions received and poorer baseline nutritional status (Supplementary Table 1).
Adherence and competence
Average dietitian adherence and competence ratings are presented in Table 2. The use of each study checklist item ranged from an average of 13 to 96% of each dietitians’ sampled intervention consultations. The mean overall BECCI practitioner score was 23.17, meaning that on average, each dietitian used behaviour change counselling ‘to some extent’  with intervention patients. The average dietitian score on the CTS-R interpersonal effectiveness item was in the ‘expert’ [44, 45] range, meaning that intervention sessions were clearly delivered within the required ‘spirit’ (i.e. a high degree of empathy, warmth, genuineness and concern). The average dietitian score on the CTS-R competence item suggests that on average, intervention dietitians were ‘advanced beginners’ [44, 45].
Relationship between fidelity and the nutritional status of intervention patients
Of the nine fidelity predictors (six study checklist items, BECCI practitioner score, CTS-R competence score, CTS-R interpersonal effectiveness score), three were found to be significantly related to the nutritional status of intervention participants (Table 3). From the study checklist, dietitian adherence to the nutrition plan was the only item that emerged as a significant predictor of patient nutritional status (β = 7.62, 95% CI = 0.65–14.58, p = 0.032). Dietitian adherence to behaviour change counselling (β = 0.69, 95% CI = 0.02–1.38, p = 0.045) and dietitian competence (β = 3.50, 95% CI = 0.47–6.53, p = 0.024) were also found to be significant predictors. Amongst intervention patients (i.e. whose nutritional status on average was found to be significantly better than that of controls), participants whose dietitians used either the nutrition planner or behaviour change counselling skills more frequently than other dietitians, or who had a dietitian who was more skilful in intervention delivery, demonstrated higher PGSGA scores (i.e. at greater risk of malnutrition).
We found evidence that dietitian adherence and competence were inversely related to patient nutritional status in a sample of HNC patients enrolled in the intervention phase of the EAT clinical trial. Aligned with published differentiation outcomes , significant predictors of nutritional status only emerged from those fidelity outcomes that could be used to distinguish intervention from control sessions. Higher levels of dietitian adherence to the nutritional planner or MI were associated with significantly higher PGSGA scores (i.e. poorer patient nutritional status). Similarly, a significant relationship emerged between higher levels of dietitian competence and higher (worse) PGSGA scores for intervention participants. These findings are consistent with the inverse relationship detected between fidelity and treatment outcome within other evaluations of health behaviour change interventions [29, 31, 46], and the psychotherapeutic literature more broadly [27, 47].
How then, do we interpret these results within the context of the intervention being effective compared to randomised controls, and clinician adherence and competence being significantly higher amongst intervention dietitians? Taken together with evidence for null [21, 30, 31] or mixed  findings, it is clear that the relationship between clinician adherence, clinician competence and patient outcome is inherently more complex than the assumption of a necessarily positive relationship (i.e. higher adherence should optimise intervention efficacy). Although seemingly intuitive, this assumption does not account for a range of factors , including the complex interaction between patient and clinician behaviours that are central to MI informed interventions [11, 30, 48]. Of particular relevance to the current study is the potential impact of ‘challenging’ patients on therapist behaviour.
Two hypotheses have been proposed regarding the direction of this effect . Firstly, therapist adherence and competence may be adversely affected . For example, ‘behavioural resistance’ on the part of the client has been linked to lower levels of clinician adherence . Similarly, patient resistance can elicit clinician confrontation  (a behaviour which runs contrary to the empathic, collaborative non-judgemental stance of motivational approaches), thereby undermining clinician competence. A second hypothesis , and one that is consistent with both the findings from the current study, and effective training in MI , is that adherence and competence are expected to increase during consultations with challenging patients. For example, patient resistance should elicit greater clinician use of MI, which in turn, provides a foundation for engaging patients in behaviour change strategies (e.g. nutrition planner). Under these circumstances, competence ratings are also likely to be greater due to clinician ‘responsivity’  and patient complexity .
Consistent with this second hypothesis, an evaluation of motivational enhancement therapy (MET)  demonstrated that patient motivation was a significant predictor of within clinician variability in MET adherence. That is, when patients were less motivated to change their behaviour, clinicians were more likely to implement MET skills. Applied to the current trial, this finding is encouraging, since it suggests that EAT provided dietitians with tools to utilise with ‘hard to treat’ patients. Clearly, further research is warranted to identify and understand the relationship between patient and clinician variables, intervention delivery and treatment outcome.
Within the current analysis, two of the EAT intervention features previously shown to differentiate intervention from control sessions  (eliciting patient motivation for undergoing radiotherapy treatment; explicitly highlighting the relationship between adequate nutritional status and radiotherapy treatment outcomes) did not emerge as significant predictors of the nutritional status of intervention participants. On the one hand, this finding could be interpreted as evidence for the redundancy of these intervention elements. An alternate view is that since both intervention elements are grounded in established principles of behaviour change (i.e. amplifying the intrinsic value of the new behaviour and/or reducing the value of the old behaviour .), we expect that the answer lies in the complex nature of the EAT intervention. That is, by definition, the synergistic action of multiple, interacting components is expected to be greater than any single intervention component . Building from the current findings, mixed methods research is now needed to specifically model and evaluate the complex pathways by which EAT may exert its effect, both in terms of intervention components and contextual factors .
Strengths and limitations
This study makes an important contribution to the literature concerning complex behaviour change interventions and patient outcomes. To the best of our knowledge, this is the first real-world evaluation of the relationship between clinician adherence, competence and patient outcome within a tertiary care setting. The EAT intervention was delivered by a range of real-world clinicians within the context of every day practice. Patient nutritional status was assessed by independent, trained assessors  using a gold standard approach to nutrition assessment [36, 42]. Fidelity outcomes are derived from independent coding of a randomly selected sample of real-world consultations, by a trained coder blind to treatment allocation. Coder reliability was good, lending confidence that our findings reflect true treatment effects (i.e. rather than measurement error). Consistent with published recommendations our approach to analysing the relationship between fidelity and treatment outcome allowed us to control for temporal confounds and plausible third variables .
There are also a number of limitations to consider when interpreting results. First, findings are derived from a 20% sample of audio-recorded consultations conducted during the intervention phase of the EAT clinical trial. Although objective coding of real consultations is considered gold standard [2, 3, 56], the associated time and cost limits the number of sessions that can be coded . Accordingly, although our methods lend confidence in the accuracy of our findings, we are restricted in our ability to definitively comment on clinician adherence and competence outside of the current sample of recordings. Recent advances in automated approaches to coding  may help to address this common limitation within the fidelity literature. Secondly, although subgroup analysis lends confidence in the representativeness of the sample, not all intervention participants were represented in the sample of intervention consultations randomly selected for fidelity assessment. Thirdly, fidelity of intervention delivery was operationalised according to observable skills, relative to the purpose or function behind using the skill. This is a common feature of tools used to evaluate clinician adherence and competence [53, 58]. However, given that intervention implementation is influenced by a range of clinician, patient and contextual factors , ongoing monitoring and assessment of clinician understanding of how an intervention works and their decision making process when it comes to intervention implementation is essential. In the current study, we utilised regular supervision and coaching to support theory informed treatment decisions [33, 35]. Future evaluations should also assess other important clinician (e.g. personality ), patient (e.g. change talk ) and contextual factors (e.g. organisational social context and readiness for change ) that may influence intervention delivery. Finally, the majority of our intervention providers were women, working with English speaking patients undergoing radiotherapy for HNC. Further research is therefore needed to understand whether the current findings generalise to male clinicians, non-English speakers and other patient groups at risk of malnutrition.
Implications for research and practice
This paper addresses an important methodological limitation within the complex behaviour change literature. Improved understanding of the relationship between fidelity and treatment outcome represents an important research priority. This paper therefore makes an important contribution to the science of fidelity and the quality with which complex behaviour change interventions are evaluated and reported. The current findings also have direct implications for future training and supervision in the EAT intervention. For example, the utility of the nutrition planner and MI should be highlighted and efforts made to ensure that clinicians are equipped to competently deliver these skills to challenging patients. Conversely, efforts should be made to ensure that less challenging patients do not miss out on important intervention elements.
To optimise the effectiveness of complex behaviour change interventions more broadly, future research designed to explicitly manipulate and evaluate the interplay between patient and provider behaviours, intervention delivery and treatment outcome is needed. This research would also have important implications for intervention design, delivery and training—for example by providing evidence for patient ‘cues’ that providers can use to tailor and optimise intervention delivery and provider ‘cues’ that trainers can use to tailor and optimise training and supervision. Such ‘responsivity’  is especially important for intervention delivery and dissemination within the dynamic context of real-world service provision .
The current paper lends further support to the complex interplay between patient behaviour and clinician fidelity. Dietitians’ adherence to either MI or a nutrition plan and their competence were all found to be inversely related to patient nutritional status. Dietitians were especially likely to draw upon and skilfully execute these intervention components when working with challenging clients. This is likely central to the demonstrated success of the EAT intervention in reducing malnutrition and highlights the importance of ensuring that providers are adequately equipped to flexibly integrate intervention elements according to patient need. Identifying the clinician, patient and contextual factors that mediate and/or moderate the relationship between fidelity and treatment outcome represents an important challenge for future research.
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
Eating As Treatment
Head and neck cancer
Patient-Generated Subjective Global Assessment
Human Research Ethics Committee
Behaviour Change Counselling Index
The Cognitive Scale–Revised
Motivational enhancement therapy
McGee D, Lorencatto F, Matvienko-Sikar K, Toomey E. Surveying knowledge, practice and attitudes towards intervention fidelity within trials of complex healthcare interventions. Trials. 2018;19(1):504. https://doi.org/10.1186/s13063-018-2838-6.
Bellg AJ, Borrelli B, Resnick B, Hecht J, Minicucci DS, Ory M, et al. Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH behavior change consortium. Health Psychol. 2004;23(5):443–51. https://doi.org/10.1037/0278-6188.8.131.523.
Borrelli B, Sepinwall D, Ernst D, Bellg AJ, Czajkowski S, Breger R, et al. A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. J Consult Clin Psychol. 2005;73(5):852–60. https://doi.org/10.1037/0022-006X.73.5.852.
Gearing RE, El-Bassel N, Ghesquiere A, Baldwin S, Gillies J, Ngeow E. Major ingredients of fidelity: a review and scientific guide to improving quality of intervention research implementation. Clin Psychol Rev. 2011;31(1):79–88. https://doi.org/10.1016/j.cpr.2010.09.007.
Perepletchikova F, Treat TA, Kazdin AE. Treatment integrity in psychotherapy research: analysis of the studies and examination of the associated factors. J Consult Clin Psychol. 2007;75(6):829–41. https://doi.org/10.1037/0022-006x.75.6.829.
Onken LS, Carroll KM, Shoham V, Cuthbert BN, Riddle M. Reenvisioning clinical science: unifying the discipline to improve the public health. Clin Psychol Sci. 2014;2(1):22–34. https://doi.org/10.1177/2167702613497932.
Michie S, Fixsen D, Grimshaw JM, Eccles MP. Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implement Sci. 2009;4(1):40. https://doi.org/10.1186/1748-5908-4-40.
Moncher FJ, Prinz RJ. Treatment fidelity in outcome studies. Clin Psychol Rev. 1991;11(3):247–66. https://doi.org/10.1016/0272-7358(91)90103-2.
Mars T, Ellard D, Carnes D, Homer K, Underwood M, Taylor SJC. Fidelity in complex behaviour change interventions: a standardised approach to evaluate intervention integrity. BMJ Open. 2013;3(11). https://doi.org/10.1136/bmjopen-2013-003555.
Beck JS. Cognitive behavior therapy: basics and beyond. 2nd ed. New York: Guilford Press; 2011.
Miller WR, Rollnick S. Motivational interviewing: helping people change. 3rd ed. New York: Guildford Press; 2013.
Dale H, Brassington L, King K. The impact of healthy lifestyle interventions on mental health and wellbeing: a systematic review. Ment Health Rev J. 2014;19(1):1–26. https://doi.org/10.1108/MHRJ-05-2013-0016.
McKenzie KJ, Pierce D, Gunn JM. A systematic review of motivational interviewing in healthcare: the potential of motivational interviewing to address the lifestyle factors relevant to multimorbidity. JOC. 2015;5:162–74. https://doi.org/10.15256/joc.2015.5.55.
Barrett S, Begg S, O'Halloran P, Kingsley M. Integrated motivational interviewing and cognitive behaviour therapy for lifestyle mediators of overweight and obesity in community-dwelling adults: a systematic review and meta-analyses. BMC Public Health. 2018;18(1):1160. https://doi.org/10.1186/s12889-018-6062-9.
Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337. https://doi.org/10.1136/bmj.a1655.
Frost H, Campbell P, Maxwell M, O'Carroll RE, Dombrowski SU, Williams B, et al. Effectiveness of motivational interviewing on adult behaviour change in health and social care settings: a systematic review of reviews. PLoS One. 2018;13(10):e0204890. https://doi.org/10.1371/journal.pone.0204890.
DiClemente CC, Corno CM, Graydon MM, Wiprovnick AE, Knoblach DJ. Motivational interviewing, enhancement, and brief interventions over the last decade: a review of reviews of efficacy and effectiveness. Psychol Addict Behav. 2017;31(8):862–87. https://doi.org/10.1037/adb0000318.
Toomey E, Hardeman W, Hankonen N, Byrne M, McSharry J, Matvienko-Sikar K, et al. Focusing on fidelity: narrative review and recommendations for improving intervention fidelity within trials of health behaviour change interventions. Health Psychol Behavior Med. 2020;8(1):132–51. https://doi.org/10.1080/21642850.2020.1738935.
Morton K, Beauchamp M, Prothero A, Joyce L, Saunders L, Spencer-Bowdage S, et al. The effectiveness of motivational interviewing for health behaviour change in primary care settings: a systematic review. Health Psychol Rev. 2015;9(2):205–23. https://doi.org/10.1080/17437199.2014.882006.
Spencer JC, Wheeler SB. A systematic review of motivational interviewing interventions in cancer patients and survivors. Patient Educ Couns. 2016;99(7):1099–105. https://doi.org/10.1016/j.pec.2016.02.003.
Copeland L, McNamara R, Kelson M, Simpson S. Mechanisms of change within motivational interviewing in relation to health behaviors outcomes: a systematic review. Patient Educ Couns. 2015;98(4):401–11. https://doi.org/10.1016/j.pec.2014.11.022.
O'Halloran PD, Blackstock F, Shields N, Holland A, Iles R, Kingsley M, et al. Motivational interviewing to increase physical activity in people with chronic health conditions: a systematic review and meta-analysis. Clin Rehabil. 2014;28(12):1159–71. https://doi.org/10.1177/0269215514536210.
Dalgetty R, Miller CB, Dombrowski SU. Examining the theory-effectiveness hypothesis: a systematic review of systematic reviews. Br J Health Psychol. 2019;24(2):334–56. https://doi.org/10.1111/bjhp.12356.
Steed L, Sohanpal R, Todd A, Madurasinghe VW, Rivas C, Edwards EA, et al. Community pharmacy interventions for health promotion: effects on professional practice and health outcomes. Cochrane Database Syst Rev. 2019;12. https://doi.org/10.1002/14651858.CD011207.pub2.
Waltman SH, Sokol L, Beck AT. Cognitive behavior therapy treatment fidelity in clinical trials: review of recommendations. Curr Psychiatr Rev. 2017;13(4):311–5. https://doi.org/10.2174/1573400514666180109150208.
Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350(mar19 6):h1258. https://doi.org/10.1136/bmj.h1258.
Webb CA, Derubeis RJ, Barber JP. Therapist adherence/competence and treatment outcome: a meta-analytic review. J Consult Clin Psychol. 2010;78(2):200–11. https://doi.org/10.1037/a0018912.
Truijens F, Zühlke-van Hulzen L, Vanheule S. To manualize, or not to manualize: is that still the question? A systematic review of empirical evidence for manual superiority in psychological treatment. J Clin Psychol. 2019;75(3):329–43. https://doi.org/10.1002/jclp.22712.
Dane AV, Schneider BH. Program integrity in primary and early secondary prevention: are implementation effects out of control? Clin Psychol Rev. 1998;18(1):23–45. https://doi.org/10.1016/s0272-7358(97)00043-3.
Magill M, Apodaca TR, Borsari B, Gaume J, Hoadley A, Gordon REF, et al. A meta-analysis of motivational interviewing process: technical, relational, and conditional process models of change. J Consult Clin Psychol. 2018;86(2):140–57. https://doi.org/10.1037/ccp0000250.
Pace BT, Dembe A, Soma CS, Baldwin SA, Atkins DC, Imel ZE. A multivariate meta-analysis of motivational interviewing process and outcome. Psychol Addictive Behaviors. 2017;31(5):524–33. https://doi.org/10.1037/adb0000280.
Beck AK, Baker AL, Carter G, Wratten C, Bauer J, Wolfenden L, et al. Assessing adherence, competence and differentiation in a stepped-wedge randomised clinical trial of a complex behaviour change intervention. Nutrients. 2020;12(8). https://doi.org/10.3390/nu12082332.
Beck AK, Baker A, Britton B, Wratten C, Bauer J, Wolfenden L, et al. Fidelity considerations in translational research: eating as treatment — a stepped wedge, randomised controlled trial of a dietitian delivered behaviour change counselling intervention for head and neck cancer patients undergoing radiotherapy. Trials. 2015;16(1):465. https://doi.org/10.1186/s13063-015-0978-5.
Britton B, Baker A, Wolfenden L, Wratten C, Bauer J, Beck AK, et al. Eating as treatment (EAT): a stepped-wedge, randomised controlled trial of a health behaviour change intervention provided by dietitians to improve nutrition in patients with head and neck cancer undergoing radiotherapy. Int J Radiat Oncol Biol Phys. 2019;103(2):353–62. https://doi.org/10.1016/j.ijrobp.2018.09.027.
Britton B, McCarter K, Baker A, Wolfenden L, Wratten C, Bauer J, et al. Eating as treatment (EAT) study protocol: a stepped-wedge, randomised controlled trial of a health behaviour change intervention provided by dietitians to improve nutrition in patients with head and neck cancer undergoing radiotherapy. BMJ Open. 2015;5(7):e008921. https://doi.org/10.1136/bmjopen-2015-008921.
Ottery FD. Definition of standardized nutritional assessment and interventional pathways in oncology. Nutrition. 1996;12(1 Suppl):S15–9. https://doi.org/10.1016/0899-9007(96)90011-8.
NHMRC. In: Government A, editor. National Health and medical research council National Statement on ethical conduct in human research (2007) - updated. Canberra: ACT; 2015.
McCarter K, Baker A, Britton B, Beck AK, Carter G, Bauer J, et al. Effectiveness of clinical practice change strategies in improving dietitian care for head and neck cancer patients according to evidence based clinical guidelines: a stepped wedge randomised controlled trial. Transl Behav Med. 2018;8(2):166–74. https://doi.org/10.1093/tbm/ibx016.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74. https://doi.org/10.2307/2529310.
Lane C, Huws-Thomas M, Hood K, Rollnick S, Edwards K, Robling M. Measuring adaptations of motivational interviewing: the development and validation of the behavior change counseling index (BECCI). Patient Educ Couns. 2005;56(2):166–73. https://doi.org/10.1016/j.pec.2004.01.003.
Blackburn I-M, James I, Milne D, Reichelt FK. Cognitive therapy scale - revised (CTS-R). 2000. https://ebbp.org/resources/CTS-R.pdf.
Bauer J, Capra S, Ferguson M. Use of the scored patient-generated subjective global assessment (PG-SGA) as a nutrition assessment tool in patients with cancer. Eur J Clin Nutr. 2002;56(8):779–85. https://doi.org/10.1038/sj.ejcn.1601412.
Lane C. The behaviour change counselling index (BECCI): manual for coding behaviour change counselling. Wales: University of Wales College of Medicine; 2002.
Dreyfus SE. The five-stage model of adult skill acquisition. BSTS. 2004;24(3):177–81. https://doi.org/10.1177/0270467604264992.
Blackburn I-M, James IA, Milne DL, Baker C, Standart S, Garland A, et al. The revised cognitive therapy scale (CTS-R): psychometric properties. Behav Cogn Psychother. 2001;29(4):431–46. https://doi.org/10.1017/S1352465801004040.
Dogar O, Boehnke JR, Lorencatto F, Sheldon TA, Siddiqi K. Measuring fidelity to behavioural support delivery for smoking cessation and its association with outcomes. Addiction. 2020;115(2):368–80. https://doi.org/10.1111/add.14804.
Huppert JD, Barlow DH, Gorman JM, Shear MK, Woods SW. The interaction of motivation and therapist adherence predicts outcome in cognitive behavioral therapy for panic disorder: preliminary findings. Cogn Behav Pract. 2006;13(3):198–204. https://doi.org/10.1016/j.cbpra.2005.10.001.
Villarosa-Hurlocker MC, O'Sickey AJ, Houck JM, Moyers TB. Examining the influence of active ingredients of motivational interviewing on client change talk. J Subst Abus Treat. 2019;96:39–45. https://doi.org/10.1016/j.jsat.2018.10.001.
Imel ZE, Baer JS, Martino S, Ball SA, Carroll KM. Mutual influence in therapist competence and adherence to motivational enhancement therapy. Drug Alcohol Depend. 2011;115(3):229–36. https://doi.org/10.1016/j.drugalcdep.2010.11.010.
Zickgraf HF, Chambless DL, McCarthy KS, Gallop R, Sharpless BA, Milrod BL, et al. Interpersonal factors are associated with lower therapist adherence in cognitive-behavioural therapy for panic disorder. Clin Psychol Psychother. 2016;23(3):272–84. https://doi.org/10.1002/cpp.1955.
Codern-Bové N, Pujol-Ribera E, Pla M, González-Bonilla J, Granollers S, Ballvé JL, et al. Motivational interviewing interactions and the primary health care challenges presented by smokers with low motivation to stop smoking: a conversation analysis. BMC Public Health. 2014;14(1):1225. https://doi.org/10.1186/1471-2458-14-1225.
Stiles WB, Horvath AO. Appropriate responsiveness as a contribution to therapist effects. How and why are some therapists better than others?: understanding therapist effects. Washington, DC: American Psychological Association; 2017. p. 71–84.
Kühne F, Meister R, Maaß U, Paunov T, Weck F. How reliable are therapeutic competence ratings? Results of a systematic review and meta-analysis. Cognit Ther Res. 2020;44(2):241–57. https://doi.org/10.1007/s10608-019-10056-5.
Greenhalgh T, Papoutsi C. Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med. 2018;16(1):95. https://doi.org/10.1186/s12916-018-1089-4.
Fletcher A, Jamal F, Moore G, Evans RE, Murphy S, Bonell C. Realist complex intervention science: applying realist principles across all phases of the Medical Research Council framework for developing and evaluating complex interventions. Evaluation. 2016;22(3):286–303. https://doi.org/10.1177/1356389016652743.
Borrelli B. The assessment, monitoring, and enhancement of treatment fidelity in public health clinical trials. J Public Health Dent. 2011;71(s1):S52–63. https://doi.org/10.1111/j.1752-7325.2011.00233.x.
Tanana M, Hallgren KA, Imel ZE, Atkins DC, Srikumar V. A comparison of natural language processing methods for automated coding of motivational interviewing. J Subst Abus Treat. 2016;65:43–50. https://doi.org/10.1016/j.jsat.2016.01.006.
Walton H, Spector A, Tombor I, Michie S. Measures of fidelity of delivery of, and engagement with, complex, face-to-face health behaviour change interventions: a systematic review of measure quality. Br J Health Psychol. 2017;22(4):872–903. https://doi.org/10.1111/bjhp.12260.
Wiltsey Stirman S, Gutner CA, Crits-Christoph P, Edmunds J, Evans AC, Beidas RS. Relationships between clinician-level attributes and fidelity-consistent and fidelity-inconsistent modifications to an evidence-based psychotherapy. Implement Sci. 2015;10:115. https://doi.org/10.1186/s13012-015-0308-z.
Williams NJ, Glisson C. Changing organizational social context to support evidence-based practice implementation: a conceptual and empirical review. Implement Sci. 2020;30:145–72.
Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8(1):117. https://doi.org/10.1186/1748-5908-8-117.
The authors would like to acknowledge the support of The Trans-Tasman Radiation Oncology Group (TROG), The Calvary Mater Newcastle Nutrition and Dietetics Department and the research, dietetic and oncology staff at participating sites. We also wish to thank project staff including Erin Forbes and Katrina Bell.
AKB is the recipient of an Australian Government Research Training Program Scholarship (Fee Offset). Funding for the EAT clinical trial was provided by a Project Grant from the National Health and Medical Research Council (APP1021018; 2011/3654). The funding body had no direct input into the design of the study; collection, analysis and interpretation of data; or writing and submission of the manuscript.
Ethics approval and consent to participate
Ethical approval was granted by the Human Research Ethics Committee (HREC) of Hunter New England Health (HREC/12/HNE/108; Approval date May 09, 2012). Site specific HREC approvals were also granted. This study was conducted in accordance with the National Statement on Ethical conduct in Human Research . All participants provided written informed consent.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional file 1: Table 1
Baseline Characteristics of Intervention Participants According to Whether or Not They Had Audio-recorded Dietetic consultations(s) Randomly Selected for Inclusion in the Fidelity Sample.
Additional file 2: Table 2
. Inter- and Intra-Rater Reliability for Intervention Recordings.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Beck, A.K., Baker, A.L., Carter, G. et al. Is fidelity to a complex behaviour change intervention associated with patient outcomes? Exploring the relationship between dietitian adherence and competence and the nutritional status of intervention patients in a successful stepped-wedge randomised clinical trial of eating as treatment (EAT). Implementation Sci 16, 46 (2021). https://doi.org/10.1186/s13012-021-01118-y
- Behavioural medicine
- Translational research
- Implementation science
- Motivational interviewing
- Behaviour change counselling