- Study protocol
- Open Access
A multidimensional approach to assessing intervention fidelity in a process evaluation of audit and feedback interventions to reduce unnecessary blood transfusions: a study protocol
Implementation Science volume 11, Article number: 163 (2016)
In England, NHS Blood and Transplant conducts national audits of transfusion and provides feedback to hospitals to promote evidence-based practice. Audits demonstrate 20% of transfusions fall outside guidelines. The AFFINITIE programme (Development & Evaluation of Audit and Feedback INterventions to Increase evidence-based Transfusion practIcE) involves two linked, 2×2 factorial, cluster-randomised trials, each evaluating two theoretically-enhanced audit and feedback interventions to reduce unnecessary blood transfusions in UK hospitals. The first intervention concerns the content/format of feedback reports. The second aims to support hospital transfusion staff to plan their response to feedback and includes a web-based toolkit and telephone support. Interpretation of trials is enhanced by comprehensively assessing intervention fidelity. However, reviews demonstrate fidelity evaluations are often limited, typically only assessing whether interventions were delivered as intended. This protocol presents methods for assessing fidelity across five dimensions proposed by the Behaviour Change Consortium fidelity framework, including intervention designer-, provider- and recipient-levels.
(1) Design: Intervention content will be specified in intervention manuals in terms of component behaviour change techniques (BCTs). Treatment differentiation will be examined by comparing BCTs across intervention/standard practice, noting the proportion of unique/convergent BCTs. (2) Training: draft feedback reports and audio-recorded role-play telephone support scenarios will be content analysed to assess intervention providers’ competence to deliver manual-specified BCTs. (3) Delivery: intervention materials (feedback reports, toolkit) and audio-recorded telephone support session transcripts will be content analysed to assess actual delivery of manual-specified BCTs during the intervention period. (4) Receipt and (5) enactment: questionnaires, semi-structured interviews based on the Theoretical Domains Framework, and objective web-analytics data (report downloads, toolkit usage patterns) will be analysed to assess hospital transfusion staff exposure to, understanding and enactment of the interventions, and to identify contextual barriers/enablers to implementation. Associations between observed fidelity and trial outcomes (% unnecessary transfusions) will be examined using mediation analyses.
If the interventions have acceptable fidelity, then results of the AFFINITIE trials can be attributed to effectiveness, or lack of effectiveness, of the interventions. Hence, this comprehensive assessment of fidelity will be used to interpret trial findings. These methods may inform fidelity assessments in future trials.
ISRCTN 15490813. Registered 11/03/2015
Blood transfusion is one of the most widely and frequently used clinical interventions. However, a substantial proportion of transfusions is considered unnecessary, in that they are administered to patients where clinical studies suggest no clear benefit . Not only are blood components (i.e. red cells, platelets, plasma) scarce and costly resources, but unnecessary transfusions place patients at risk of harm (e.g. of medical adverse events, such as transfusion reactions, or errors in patient identification) . Hence, strategies are needed to reduce unnecessary use of blood components by changing current transfusion practice.
One strategy with the potential to achieve such change is audit and feedback (A&F). A&F is defined as a ‘summary of clinical performance of healthcare over a specified time period, to provide healthcare professionals with data on performance’ . In England, the National Health Service Blood and Transplant (NHSBT) National Comparative Audit (NCA) programme runs national audits designed to assess whether blood components are being used appropriately and safely across clinical specialties . The NCA’s approach to A&F has remained largely unchanged since its establishment in 2003. In brief, an audit-writing group is convened, usually consisting of an audit lead (typically a consultant haematologist with an interest in transfusion), statistician and clinical staff representatives from the clinical specialty being audited (e.g. orthopaedics). This group is responsible for agreeing the audit standards against which clinical practice will be compared, data to be collected, and the findings and recommendations to be included in feedback reports. Resulting feedback reports are subsequently uploaded and delivered to the hospital transfusion team (i.e. transfusion practitioner, consultant haematologist, transfusion laboratory manager) via a hospital-specific NCA audit webpage. It is intended or assumed that this team will subsequently disseminate these reports within their hospitals, and, where feedback indicates discrepancies between current practice and audit standards, lead on a planning process to encourage practice change. However, the NCA currently provides no formal support to facilitate such planning.
Although there have been some incremental improvements (i.e. reductions) in use of blood products over recent years, a consistent finding from these audits is that usage of blood components by different clinical specialities remains higher than indicated by evidence-based guidelines; with approximately 1 in 5 transfusions falling outside national recommendations . This raises questions regarding the effectiveness of current A&F strategies in this context—a concern mirrored in the wider A&F literature [3, 6]. A Cochrane review demonstrated that although A&F interventions have modest, albeit worthwhile, effects on clinical practice (median +4%), outcomes across A&F interventions are highly variable (IQR 0.5 to 16%) . Reasons underpinning this heterogeneity are unclear [3, 7]. It has been argued that the systematic application of evidence and behaviour change theory has the potential to enhance the effectiveness of A&F interventions .
Development & Evaluation of enhanced Audit & Feedback Interventions to Increase uptake of evidence-based transfusion practice: the AFFINITIE programme
The AFFINITIE research programme is funded by the UK’s National Institute of Health Research and is conducted in collaboration with NHSBT. AFFINITIE uses the NCA as a platform for applying existing evidence as to what makes feedback more effective (i.e. A&F Cochrane review ), and behavioural theory (i.e. control theory ) to develop and evaluate two enhanced feedback interventions. AFFINITIE consists of four workstreams (Fig. 1). The protocol for intervention development (Workstream 1; Fig. 1) has been published elsewhere .
Briefly, Intervention 1 (‘enhanced content’) concerns the content and format of feedback reports delivered to hospitals. It is a cascade intervention consisting of two components. First, an enhancement guidance manual targeted at the NCA audit-writing group, which includes guidance on how to apply five proposed enhancements for writing feedback reports with evidence- and theory-based content. The proposed enhancements were identified following a content analysis of previous NCA feedback reports. This examined whether effective components of A&F identified in the A&F Cochrane Review, and behaviour change techniques (BCTs)  consistent with control theory (e.g. goal setting, feedback, action planning)  featured in existing reports. It is intended that the audit-writing group will apply this guidance to produce a template feedback report with enhanced content, which will subsequently be populated with hospital-specific audited data. The second component in the cascade is the resulting feedback report, which will be uploaded to each hospital’s individual NCA webpage, where intervention recipients (i.e. hospital transfusion team) can access and download their reports.
Intervention 2 (‘enhanced follow-on’) concerns the actions taken in hospitals in response to feedback reports and aims to support relevant hospital transfusion staff to plan their response to the feedback. It consists of a web-based toolkit for use by the hospital transfusion team. The toolkit aims to facilitate three behaviours in response to feedback: (i) dissemination of findings to all relevant clinical staff involved in transfusion decision making; (ii) goal setting, problem solving and action planning to facilitate practice changes in response to feedback; and (iii) continued re-monitoring of the clinical practices that were audited. The toolkit will be accessible to hospital staff via a web-link uploaded to the hospital-specific NCA webpage. As a co-intervention to prompt engagement with the toolkit, hospital transfusion teams will receive an initial telephone support call from an intervention provider, offering support and advice on how to use the toolkit. A telephone line will subsequently be available for hospitals to contact intervention providers for further support as needed.
The effectiveness and cost-effectiveness of both interventions will be evaluated in two sequential, linked, 2×2 factorial, pragmatic cluster-randomised controlled trials (RCTs) (Workstream 2; Figs. 1 and 2). Trial 1 audits clinical management and transfusion decision-making for elective surgical patients. Trial 2 audits appropriateness of red cell and platelet transfusions in haematology patients. The comparator for both interventions is NCA’s existing, standard A&F practice (i.e. clinically led feedback report; no formalised follow-on support). The primary outcome is the proportion of unnecessary transfusions administered, to be evaluated through a re-audit of key clinical behaviours approximately 9 months following intervention delivery.
However, A&F is a complex, multilevel, multicomponent intervention and is therefore susceptible to variation in implementation. Intervention fidelity refers to ongoing assessment, monitoring and enhancement of the reliability and internal validity of an intervention study . It broadly involves investigating the extent to which interventions are implemented as originally intended . Therefore, the AFFINITIE programme also includes a parallel process evaluation (Workstream 3; Fig. 1) focused on assessing fidelity in each cluster-RCT to explore how the interventions succeed or fail in reducing unnecessary transfusions.
The AFFINITIE process evaluation
Theoretical models and frameworks of fidelity argue that fidelity is a multidimensional concept, relevant at intervention designer-, provider- and recipient-levels . However, fidelity assessments of complex interventions rarely look at the ‘whole picture,’ and often focus on investigating a single dimension of fidelity; typically whether or not intervention providers delivered interventions as specified, and less so whether intervention recipients comprehend, engage with or enact interventions as intended [13–15]. Assessing recipient-level fidelity is particularly important for interventions such as the AFFINITIE feedback interventions. Both interventions’ components are standardised across recipients and are delivered only once (i.e. both the feedback report link and web link to the toolkit are delivered once, digitally, although both resources are available to recipients throughout the 9-month intervention period). The interventions are thus likely to display limited variability in designer/provider-level fidelity. Rather, it is likely that any variability in observed outcomes may in part be attributable to variation in whether and how transfusion clinical staff receiving the feedback interventions initially understand, engage with and subsequently enact the interventions in day-to-day clinical practice .
Therefore, we aim to adopt a multidimensional framework to assess fidelity, by applying the US National Institute of Health Behaviour Change Consortium’s (BCC) fidelity framework and guidance . This framework proposes five fidelity dimensions: (1) Design: treatment differentiation, acceptability, comprehensive specification of intervention components, theoretical underpinning and causal assumptions; (2) Training: extent to which intervention providers are competent and adequately trained to deliver interventions; (3) Delivery: extent to which intervention content is delivered as intended during the intervention period; (4) Receipt: whether intervention recipients initially understand/ engage with the intervention; and (5) Enactment: extent to which recipients apply the intervention as intended in ‘real life’ or target settings (e.g. clinical practice). It is important to assess fidelity across all five dimensions, as lack of fidelity to just one could affect study outcomes and thus the internal validity of the trial .
Moreover, outcomes and enactment across participants may be facilitated or hindered by contextual factors external to the intervention (e.g. organisational norms, policy, available resources) . In AFFINITIE, external factors, such as the publication of new transfusion clinical guidelines , may impact on responses to the interventions and observed outcomes. Some factors may influence intervention and control arms equally, while others may interact with or duplicate active components in the interventions. Therefore, to facilitate interpretation of fidelity and outcome data, a secondary objective of the AFFINITIE process evaluation is to explore potential contextual factors influencing fidelity of receipt and enactment.
The aims of the study are to conduct a multidimensional assessment of fidelity for two enhanced feedback interventions and assess the extent to which fidelity contributes to observed outcomes (i.e. proportion of unnecessary transfusions). The specific research questions map onto the BCC fidelity dimensions:
To what extent: (i) are the two enhanced feedback interventions clearly specified a priori, and (ii) is it possible to differentiate between the components of the two enhanced interventions and standard practice comparators (design)?
To what extent have the feedback interventions been delivered as intended by intervention providers (training/delivery)?
To what extent were the feedback interventions received and enacted as intended by transfusion clinical staff in UK hospitals (receipt/enactment)?
What are the contextual influences that might have influenced clinical staffs’ responses to the feedback interventions?
To what extent do measures of fidelity mediate observed outcomes?
Overall process evaluation design
Two linked, mixed-method fidelity assessments will be conducted in parallel to the two linked cluster-RCTs. Table 1 outlines how each sub-dimension of the five BCC fidelity dimensions  will be addressed in AFFINITIE. A summary of how each fidelity dimension will be investigated will be presented in turn.
Fidelity of design
Considerations regarding fidelity of design fall broadly into two categories: (1) trial design and (2) intervention design . The internal validity of any study will be impacted by whether the selected trial design can adequately test proposed hypotheses. In AFFINITIE, this has been considered as part of Workstream 2 (Fig. 1), via the selection of a 2×2, factorial, cluster-RCT design, and will be discussed in a separate trial protocol.
Fidelity of intervention design concerns how comprehensively interventions are specified a priori, whether intervention components adequately reflect underlying theory, and intervention acceptability . Such considerations have been addressed as part of intervention development in Workstream 1 (Fig. 1). Intervention delivery parameters (i.e. dose/ duration/ number of contacts) and content (i.e. component BCTs) have been specified a priori using BCT labels and definitions from an established BCT taxonomy . The BCTs in each intervention were then mapped onto control theory . Throughout the intervention development process, multidisciplinary experts (behaviour change scientists, transfusion clinical staff, patient representatives) have iteratively reviewed the interventions to consider whether the interventions have clinical face validity and reflect their proposed theory- and evidence-base [3, 8]. Pilot work has been conducted to assess the feasibility and acceptability of the interventions using think-aloud protocols  and semi-structured interviews .
An additional fidelity of design consideration is treatment differentiation, defined as the degree to which treatments (i.e. interventions), or two or more trial arms differ as intended along critical dimensions (e.g. content) . Treatment differentiation is of particular importance to the AFFINITIE pragmatic cluster-RCTs, which aim to evaluate the effectiveness of two feedback interventions (individually and combined), against current standard NCA feedback practice (Fig. 2). The ability to meaningfully conduct such evaluations depends on a minimum degree of differentiation between the interventions and standard practice comparators.
Treatment differentiation of the interventions as designed will be assessed by conducting comparisons between the content of (1) intervention 1 ‘enhanced content’ vs current standard practice (i.e. clinically led feedback reports); (2) intervention 2 ‘enhanced follow-on support’ vs current standard practice (i.e. no formalised follow-on support); and (3) intervention 1 ‘enhanced content’ vs intervention 2 ‘enhanced follow-on support.’ The intended content of each intervention and comparator will be coded by two independent coders into component BCTs using a BCT taxonomy . BCTs will be compared across both interventions and standard practice comparators in terms of BCT frequency, mode of delivery, behavioural specificity—including relation to target behaviours and enactment instructions. BCTs that are identified at least once in either intervention or comparator will be categorised as ‘fully convergent’ (i.e. present in both the intervention and comparator with similar frequency, mode of delivery, extent of behavioural specificity and enactment instructions); ‘partially convergent’ (i.e. present in both the intervention and comparator, but with differing frequencies, modes of delivery, extent of behavioural specificity and enactment instructions), or ‘unique’ (i.e. present only in the intervention or comparator). For each of the three comparisons, the percentage of BCTs in each of these categories will be noted. A higher proportion of ‘fully’ or ‘partially convergent’ BCTs indicates lower treatment differentiation. Inter-rater coding reliability will be assessed using Cohen’s kappa, with a value of k = 0.75 taken to represent high agreement .
Fidelity of training
The BCC guidance advocates standardisation of training between intervention providers and checking providers acquire pre-specified competence indicators prior to intervention delivery .
The intervention providers for intervention 1 are the NCA-assembled audit-writing groups. There will be two writing groups: an enhanced writing group, tasked with drafting feedback reports using the enhancement guidance manual; and a standard writing group, tasked with drafting feedback reports as per usual NCA practice. The enhanced writing group will be trained via a full-day workshop, where intervention developers will deliver a presentation explaining each proposed enhancement, and facilitate wider discussion of prototype enhanced feedback reports and how these may be adapted to write feedback reports for the current audit topic. Initial skill acquisition will be assessed by conducting a content analysis of the first draft enhanced feedback report(s). Two independent raters will assess whether the five proposed enhancements and theoretically consistent BCTs are ‘present’, ‘absent but should be present’, or ‘not applicable’ in the draft report, and inter-rater agreement assessed using Cohen’s kappa. The percentage of applicable intervention components present will be calculated.
The intervention providers for intervention 2 are responsible for developing the web-based toolkit (i.e. health psychologists, web designers) and providing the telephone support co-intervention (i.e. health psychologists). Intervention providers will be trained to deliver telephone support by an intervention developer over three training sessions using a standardised telephone support manual and practice role plays. This manual specifies the intended content and format of telephone support calls in terms of component BCTs. Intervention providers’ skill acquisition will be assessed prior to intervention delivery via audio-recorded, simulated role plays of telephone support sessions. An independent rater will content analyse role play transcripts and score the proportion of manual-specified BCTs delivered as intended.
Fidelity of delivery
Assessing fidelity of delivery during the intervention period concerns whether interventions were delivered as intended (i.e. adherence), ongoing treatment competency (i.e. whether providers maintain skills acquired during training) and monitoring for potential loss of treatment differentiation between trial arms, due to delivery of additional, unspecified intervention components and/or potential contamination [11–13].
For intervention 1, in the intervention arm, ongoing treatment competency will be encouraged by intervention developers providing on-going support to the audit-writing group during the drafting of feedback reports. Intervention developers will iteratively review drafts of reports and provide feedback on how to further incorporate delivery of proposed enhancements and theoretically consistent BCTs. Subsequent fidelity of delivery (i.e. adherence) will be assessed by conducting a content analysis of the final template enhanced feedback report(s). The template report(s) will be coded by two independent raters using a BCT taxonomy  to verify whether the five manual-specified enhancements and theoretically consistent BCTs are either ‘present’, ‘absent but should be present’ or ‘not applicable’. Inter-rater reliability will be assessed using Cohen’s kappa. Any additional, non-manual specified BCTs present in the final template feedback report(s) will be documented. The template feedback reports for the standard/comparator arm will be equivalently content analysed to monitor for any contamination and loss of treatment differentiation. To monitor delivery of feedback reports by NHSBT’s NCA, the webpage for every hospital in each trial arm will be examined to verify that the correct feedback reports were uploaded according to cluster allocation (i.e. enhanced vs standard).
For intervention 2, fidelity of delivery for the final version of the toolkit will be examined by conducting a content analysis of the toolkit to assess whether intended and theoretically consistent BCTs are present, absent but should be present or not applicable. The NCA webpage for all participating hospitals will be examined to verify that a link to the toolkit is present for hospitals allocated to receive intervention 2 and absent for control hospitals. For telephone support, fidelity of delivery will be promoted through use of a manual and flow chart to prompt delivery of intended BCTs. A log of all delivered telephone support calls will be maintained and reviewed to monitor whether all participating hospitals allocated to receive intervention 2 received at least one initial telephone support call. All telephone support sessions will be audio-recorded. Adherence and ongoing treatment competency will be assessed by transcribing a randomly selected sub-sample of sessions (n = 18 in total; i.e. 9 per trial arm receiving intervention 2; 6 per intervention provider). Transcripts will be coded into component BCTs to monitor whether intervention providers continue to competently deliver BCTs according to manual specification. Both content analyses of the toolkit and telephone support transcripts will be conducted by two independent raters and inter-rater reliability assessed using Cohen’s kappa.
Fidelity of delivery will be quantified for both interventions by assessing the proportion of manual-specified content (i.e. % proposed enhancements/BCTs) delivered as intended in the feedback reports/toolkit/telephone support sessions, and also the proportion of hospitals that received feedback reports, links to the toolkit and/or telephone support, according to trial allocation. Consensus in the literature suggests that 80 to 100% adherence to intervention specifications represents ‘high’ fidelity of delivery, 51 to 79% represents ‘moderate’ fidelity, and <50% or less represents ‘low’ fidelity [12, 21].
Fidelity of receipt, enactment and contextual influences
Receipt and enactment will be concurrently investigated alongside contextual influences. Recommended strategies for assessing receipt and enactment include objective verification of intervention implementation where possible or recipient self-report [11–13]. Receipt, enactment and context will thus be assessed using a mixed methods approach, combining objective web-analytics data, questionnaires and semi-structured interviews.
Web-analytics data to assess engagement and enactment
For intervention 1, the NCA webpage has been programmed to record the number of times each feedback report was downloaded by each hospital throughout the intervention period. For intervention 2, the toolkit has been programmed to record usage patterns during the intervention period, including the number and duration of logins, page views, and indicators of completion of tools (i.e. adding/deleting characters). Patterns of engagement throughout the intervention period will be examined for both interventions and represented visually using graphs.
Questionnaires and semi-structured interviews
Questionnaires will assess receipt, enactment and contextual influences across all participating hospitals. Approximately 3 months following intervention delivery, an invitation to complete a web-based questionnaire will be emailed to the NCA-listed clinical contact for the audit at each participating site (i.e. nominated member of the hospital transfusion team, typically transfusion practitioner, leading audit data collection and response). To also collect responses from the range of clinical staff potentially involved in applying findings from the audit, the initial contact will be asked to forward the questionnaire link and invitation email to at least two other clinical staffs involved in transfusion—ideally a further member of the hospital transfusion team and one representative from the clinical specialty being audited (e.g. anaesthetists/ surgeons for trial 1, which audits surgical patient blood management). Respondent demographic characteristics, including role/clinical specialty, will also be recorded. The response rate will be calculated as the percentage of participating sites (n = 155 Trial 1; n = 167 trial 2) for which at least one questionnaire response is received.
Semi-structured interviews will be conducted in a sub-sample of hospitals to enable more in-depth exploration of receipt, enactment and contextual influences. Interviews lasting up to 1 h will be conducted via telephone approximately 3 months following intervention delivery, with up to three clinical staff members in three randomly selected hospitals per trial arm (i.e. 3 hospitals × 4 trial arms = 12 hospitals; 12 hospitals × 3 clinical staff = 36 participants; 24 hospitals and 72 participants across both trials). Clinical staff eligible to participate will include two members of the hospital transfusion team and a clinical staff member from the relevant specialty/topic being audited.
Questionnaire items and interview topic guides will be based on the Theoretical Domains Framework (TDF) [22, 23] to enable systematic identification of barriers/enablers to intervention recipients understanding, engaging with and/or enacting feedback interventions as intended. The TDF synthesises 128 constructs from 33 distinct behaviour change theories into 12 theoretical domains. The TDF has been applied to investigate barriers/enablers to behaviour change across a range of clinical contexts, including transfusion [24–26]. It has also been applied as part of intervention development in AFFINITIE (Workstream 1; Fig. 1) to identify barriers/enablers to how hospitals currently respond to transfusion A&F delivered by NHSBT’s NCA . The target behaviours of interest are the extent to which clinical staff involved in transfusion decision making for elective surgical (trial 1) or haematology patients (trial 2), initially understand and engage with the feedback materials delivered by the NCA and subsequently act on recommendations provided in the reports. Questionnaire and interview topic guides will include at least one question related to each of the 12 theoretical domains. Additional file 1 provides sample questions for each domain.
Items related to fidelity of receipt will explore the extent to which participants initially engaged with and understood the intervention. TDF domains of relevance to receipt might thus include ‘knowledge’ (e.g.what the key findings and recommendations from the feedback reports were; whether participants understood how to use the tools in the toolkit), and ‘memory, attention, decision making’ (e.g. how readily participants were able to extract key information for their hospital from the feedback materials). Items related to enactment will explore how participants have subsequently acted upon the feedback. TDF domains of relevance to enactment might thus include ‘motivation and goals’ and ‘behavioural regulation’ (e.g. whether they have set goals or action plans in light of feedback; If not, why not? Have they encountered any difficulties when trying to change practice in light of feedback? If so, how have they overcome these?), or ‘nature of the behaviour’ (e.g. how has your response to this audit differed from what you normally do when receiving blood transfusion feedback from the NCA?). The TDF will also allow for potential contextual factors influencing intervention enactment to be investigated, for instance, via the domains ‘social Influences’ (e.g. How receptive have your clinical colleagues been to making changes in response to the audit findings?) and ‘environmental context and resources’ (e.g. To what extent does your team/site have the necessary resources to change practice in light of feedback? If not, what additional resources would be required?) The questionnaires and topic guides will be piloted and reviewed by two haematologists to ensure clinical face validity.
Questionnaire responses and web-analytics data will be summarised descriptively. Transcripts of audio-recorded interviews will be analysed using content analysis following a framework analysis approach [27, 28]. Using the TDF as a coding framework, participant responses within each transcript will be coded into TDF domains they are judged to represent. For example, ‘I do not feel confident that I will be able to act on any of the recommendations provided in the feedback reports’, would be coded as TDF domain ‘beliefs about capabilities’. A 20% sub-sample of transcripts will be double-coded to assess inter-rater reliability using Cohen’s kappa. Responses coded into each domain will subsequently compared across transcripts, sorting responses expressing similar views into groups. A belief statement will be inductively generated for each cluster of grouped responses, summarising the shared view across participants as to the role that domain plays in hindering or enabling intervention enactment. Generated belief statements will then be reviewed to identify those belief statements representing the likely most important barriers/enablers based on three criteria: (1) belief statement frequency across participants; (2) presence of conflicting beliefs across participants; and (3) expressed importance by participants .
Association with outcomes
Systematic reviews demonstrate that few fidelity assessments examine associations between fidelity and outcomes . After quantitative process evaluation data (i.e. extent of fidelity of delivery, receipt/enactment questionnaire responses; web-analytics data) and outcome data (i.e. proportion of unnecessary transfusions) have been analysed, we will explore whether fidelity mediates the impact of the randomised interventions on trial outcomes using causal mediation analyses .Full details will be included in the statistical analysis plan for the trials.
In line with recommendations for mixed methods research, qualitative (e.g. semi-structured interviews) and quantitative (e.g. questionnaires, web-analytics) findings will be integrated following a triangulation protocol (i.e. methodological triangulation) [31–33]. To examine consistencies in key fidelity findings across the two audit topics being evaluated, findings will also be triangulated across trials 1 and 2 (i.e. data source triangulation) .For each key finding, data from each data set will be tabulated in a ‘convergence coding matrix’. The extent of similarity of findings across data sets will be compared and categorised as representing either full agreement (data convergence), partial agreement (complementarity between data), conflicting findings (discord) or silence (finding identified in only one data source and no additional sources) [31, 32, 34]. Triangulated findings will be presented schematically to illustrate the relationship between fidelity dimensions and outcomes.
At the time of submission of this protocol (November 2016), the two enhanced A&F interventions had been delivered for both trials. Fidelity data on training, delivery, receipt and enactment had been collected for trial 1. Only data on fidelity of training and delivery had been collected for trial 2. Outcome data collection had commenced for trial 1 but not trial 2. No outcome or fidelity data from either trial were cleaned or analysed prior to submission.
This protocol describes planned strategies for assessing intervention fidelity as part of the process evaluation of two linked cluster RCTs in the AFFINITIE programme. A strength of the present process evaluation is the multidimensional approach to fidelity data collection. The planned methods and five fidelity dimensions assessed are also in line with the four process evaluation functions proposed by the recently published Medical Research Council guidance for designing and conducting process evaluations of complex interventions : (1) ensuring interventions’ core components, theoretical underpinning and key assumptions are clearly articulated (i.e. Design); (2) examining extent of implementation during the intervention period (i.e. training/delivery); (3) examining contextual factors affecting intervention delivery and functioning; and (4) investigating mechanisms of impact through which observed outcomes are achieved, including extent of engagement by intervention recipients (i.e. receipt/enactment; mediation analyses).
Results from the fidelity assessment will contribute to an in depth investigation of how the interventions ‘worked’ (or not), which will contribute to interpretation of observed trial outcomes. There are a number of possible scenarios:
If the trial outcome identifies significant reductions in the proportion of unnecessary transfusions, and the proposed fidelity assessments identify the interventions were implemented with high fidelity, with minimum contextual barriers, and that extent of fidelity mediated observed outcomes, then it may be more confidently inferred that observed outcomes are attributable to the enhanced A&F interventions.
Alternatively, if the interventions are not effective and there is evidence that fidelity was poor or inconsistent, it would be premature to dismiss the interventions as ineffective; given they were not implemented as intended and thus not evaluated. It may also be the case that poor fidelity is associated with delivery or enactment of unplanned BCTs that may have negatively impacted outcomes.
Conversely, the trial may identify significant reductions in unnecessary transfusions, despite poor fidelity. If so, delivery of unplanned BCTs or contextual factors may have interacted with intervention delivery to influence practice (e.g. transfusion clinical staff in intervention hospitals being more motivated to change than those in the control hospitals as a result of more contact with intervention providers). Alternatively, the interventions may have operated through mechanisms and pathways other than those investigated in the fidelity assessment.
Lastly, no significant changes in unnecessary transfusions may be observed, despite good fidelity. If so, it could be inferred that the interventions are not effective and should not be scaled up into routine feedback practice. Alternatively, the process evaluation may identify unexpected factors that contributed to the observed changes in clinical practice across all participating hospitals, including contamination between trial arms, loss of treatment differentiation and contextual factors (i.e. new national transfusion guidelines).
Any such scenarios would provide valuable findings that would contribute to the scientific knowledge, evidence and theory regarding what makes for more effective A&F. Furthermore, an ongoing debate in the fidelity literature is that of ‘strict fidelity’ vs ‘adaptation’ (to key contextual factors), and the unresolved question of ‘how much’ fidelity is necessary or beneficial [12, 36]. For pragmatic trials such as the AFFINITIE cluster-RCTs, where variation and localised tailoring are likely inevitable, uniform or strict 100% fidelity may not be feasible to achieve nor necessarily desirable. General consensus in the literature is that 80 to 100% fidelity is considered ‘high’ and ‘good’ , yet there is insufficient evidence to support this. The proposed moderator/mediator analyses may identify that improved outcomes are achieved, for instance, with a ‘moderate’ degree of fidelity (e.g. 60%); thus implying a permissible degree of loss of intervention fidelity or adaptation. Such findings would be of interest to researchers developing future A&F interventions, as well as to policy makers and service commissioners, by providing feasibility data that may inform how and to what extent the interventions may be generalised in new settings or implemented on a larger scale, while maintaining similar benefits. This is the focus of Workstream 4 of the AFFINITIE programme (Fig. 1), which aims to explore lessons learned regarding A&F in the context of blood transfusion that may be generalised to A&F in other clinical contexts.
There are a number of limitations and challenges to the proposed process evaluation. The trials in AFFINITIE are pragmatic evaluations, embedded in the existing NHSBT NCA A&F programme. The trials and process evaluation are thus restricted by what is feasible to deliver and evaluate in this context. One notable challenge, highlighted in the intervention development process, is in describing ‘standard practice.’ A content analysis of existing NCA feedback reports and interviews with clinical transfusion staff highlighted significant variability in the content and structure of existing feedback reports and how hospitals currently respond to transfusion A&F. The lack of clarity as to what constitutes standard practice makes it difficult to establish at what point practice in the intervention arms differs from ‘usual practice,’ and whether or not practice has changed as a result of the interventions delivered.
An additional limitation, and potential source of contamination, is that members of the intervention development team are involved in conducting the process evaluation. This increases risk of biasing analysis of process evaluation data. The MRC process evaluation guidance  highlights the importance of being reflective of the relationship and boundaries between intervention developers, implementers and evaluators and recommends seeking occasional critical review from independent peers. The AFFINITIE Programme Steering Group provides oversight to the conduct of all elements of the programme and includes an independent chair and expert members who are familiar with the models and methods being used in AFFINITIE.
It has been argued that if process evaluations are to be funded and conducted, it is important that the protocol and findings from the process evaluation are transparently reported and published, as is standard practice for outcome evaluations [35, 37]. This is particularly important for fidelity investigations. While the BCC fidelity framework provides a synthesised conceptualisation and definition of different fidelity dimensions, the authors themselves acknowledge that there is little standardised guidance on how to operationalise each dimension, and a consequent need for further empirical examples of how each dimension has been quantified and interpreted [14, 38]. It is thus hoped that publishing the fidelity protocol and findings for the AFFINITIE process evaluation will provide a methodological contribution to the fidelity literature that may inform future fidelity assessments in other contexts.
We have presented the protocol for a multidimensional assessment of intervention fidelity as part of the process evaluation of two enhanced A&F interventions to reduce unnecessary blood transfusions. It is anticipated that findings from the process evaluation will provide information critical to the interpretation of the trial outcomes. This protocol also contributes to the advancement of the process evaluation methods by illustrating the application of a comprehensive, multidimensional framework for assessing intervention fidelity, which may inform the design of future fidelity assessments.
Audit and feedback
Behaviour Change Consortium
Behaviour Change Technique
National Comparative Audit
National Health Service
National Health Service Blood and Transplant
National Institute for Health and Care Excellence
Randomised controlled trial
Theoretical Domains Framework
Carson JL, Carless PA, Hebert PC. Transfusion thresholds and other strategies for guiding allogeneic red blood cell transfusion. 2012. The Cochrane Library.
Stainsby D, Jones H, Asher D, Atterbury C, Boncinelli A, Brant L, et al. Serious hazards of transfusion: a decade of hemovigilance in the UK. Transfus Med Rev. 2006;20(4):273–82.
Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O'Brien MA, Johansen M, Grimshaw J, Oxman AD. Cochrane Database Syst Rev. 2012;13(6):CD000259. doi: 10.1002/14651858.CD000259.pub3. Review.
NHS Blood and Transplant National Comparative Audit. [Available from: http://hospital.blood.co.uk/safe_use/clinical_audit/National_Comparative/index.asp. Accessed 8 Dec 2016.
Estcourt L, Stanworth S, Doree C, Hopewell S, Murphy MF, Tinmouth A, et al. Prophylactic platelet transfusion for prevention of bleeding in patients with haematological disorders after chemotherapy and stem cell transplantation. 2012. The Cochrane Library.
Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ, et al. No more ‘business as usual’with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci. 2014;9(1):1.
Gardner B, Whittington C, McAteer J, Eccles MP, Michie S. Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Soc Sci Med. 2010;70(10):1618–25.
Carver CS, Scheier MF. On the self-regulation of behavior. Cambridge: Cambridge University Press; 2001.
Gould NJ, Lorencatto F, Stanworth SJ, Michie S, Prior ME, Glidewell L, et al. Application of theory to enhance audit and feedback interventions to increase the uptake of evidence-based transfusion practice: an intervention development protocol. Implement Sci. 2014;9(1):1.
Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med. 2013;46(1):81–95.
Bellg AJ, Borrelli B, Resnick B, Hecht J, Minicucci DS, Ory M, et al. Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH Behavior Change Consortium. Health Psychol. 2004;23(5):443.
Borrelli B. The assessment, monitoring, and enhancement of treatment fidelity in public health clinical trials. J Public Health Dent. 2011;71(s1):S52–63.
Borrelli B, Sepinwall D, Ernst D, Bellg AJ, Czajkowski S, Breger R, DeFrancesco C, Levesque C, Sharp DL, Ogedegbe G, Resnick B, Orwig D. J Consult Clin Psychol. 2005;73(5):852–60.
Resnick B, Bellg AJ, Borrelli B, De Francesco C, Breger R, Hecht J, et al. Examples of implementation and evaluation of treatment fidelity in the BCC studies: where we are and where we need to go. Ann Behav Med. 2005;29(2):46–54.
Schober I, Sharpe H, Schmidt U. The reporting of fidelity measures in primary prevention programmes for eating disorders in schools. Eur Eat Disord Rev. 2013;21(5):374–81.
Yardley L, Spring B, Riper H, Morrison L, Crane D, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. 2016;51(5):833–42.
Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials. 2013;14(1):1.
Padhi S, Kemmis-Betty S, Rajesh S, Hill J, Murphy MF. Blood transfusion: summary of NICE guidance. BMJ. 2015;351:h5832.
Fonteyn ME, Kuipers B, Grobe SJ. A description of think aloud method and protocol analysis. Qual Health Res. 1993;3(4):430–41.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;1:159-74.
Noell GH, Gresham FM, Gansle KA. Does treatment integrity matter? A preliminary investigation of instructional implementation and mathematics performance. J Behav Educ. 2002;11(1):51–67.
Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7(1):1.
Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care. 2005;14(1):26–33.
Francis JJ, O’Connor D, Curran J. Theories of behaviour change synthesised into a set of theoretical groupings: introducing a thematic series on the theoretical domains framework. Implement Sci. 2012;7(1):1.
Francis JJ, Stockton C, Eccles MP, Johnston M, Cuthbertson BH, Grimshaw JM, et al. Evidence‐based selection of theories for designing behaviour change interventions: using methods based on theoretical construct domains to understand clinicians’ blood transfusion behaviour. Br J Health Psychol. 2009;14(4):625–46.
Islam R, Tinmouth AT, Francis JJ, Brehaut JC, Born J, Stockton C, et al. A cross-country comparison of intensive care physicians’ beliefs about their transfusion behaviour: a qualitative study using the theoretical domains framework. Implement Sci. 2012;7(1):1.
Patey AM, Islam R, Francis JJ, Bryson GL, Grimshaw JM. Anesthesiologists’ and surgeons’ perceptions about routine pre-operative testing in low-risk patients: application of the Theoretical Domains Framework (TDF) to identify factors that influence physicians’ decisions to order pre-operative tests. Implement Sci. 2012;7(1):1.
McSherry LA, Dombrowski SU, Francis JJ, Murphy J, Martin CM, O’Leary JJ, et al. ‘It’sa can of worms’: understanding primary care practitioners’ behaviours in relation to HPV using the theoretical domains framework. Implementation. Science. 2012;7(1):1.
Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2003;18(2):237–56.
Emsley R, Dunn G, White IR. Mediation and moderation of treatment effects in randomised controlled trials of complex interventions. Stat Methods Med Res. 2010;19(3):237–70.
Tonkin-Crine S, Anthierens S, Hood K, Yardley L, Cals JW, Francis NA, et al. Discrepancies between qualitative and quantitative evaluation of randomised controlled trial results: achieving clarity through mixed methods triangulation. Implement Sci. 2016;11(1):1.
O’Cathain A, Murphy E, Nicholl J. Three techniques for integrating data in mixed methods studies. BMJ. 2010;341:c4587.
Hopf Y, Francis J, Helms P, Haughney J, Bond C. Core requirements for successful data linkage: an example of a triangulation method. BMJ Open. 2016;6(10):e011879.
Farmer T, Robinson K, Elliott SJ, Eyles J. Developing and implementing a triangulation protocol for qualitative health research. Qual Health Res. 2006;16(3):377–94.
Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.
Leventhal H, Friedman MA. Does establishing fidelity of treatment help in understanding treatment efficacy? Comment on Bellg et al.(2004). 2004
Ellard DR, Taylor SJ, Parsons S, Thorogood M. The OPERA trial: a protocol for the process evaluation of a randomised trial of an exercise intervention for older people in residential and nursing accommodation. Trials. 2011;12(1):1.
Graham L, Wright J, Walwyn R, Russell AM, Bryant L, Farrin A, et al. Measurement of adherence in a randomised controlled trial of a complex intervention: supported self-management for adults with learning disability and type 2 diabetes. BMC Med Res Methodol. 2016;16(1):132.
This paper summarises independent research funded by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research Programme (Grant Reference Number RP-PG-1210-12010). The views expressed are those of the authors and do not necessarily those of the NHS, the NIHR or the Department of Health. All authors’ contributions to this study are funded by this grant. Jeremy Grimshaw holds a Canada Research Chair in Health Knowledge Transfer and Uptake.
Availability of data and material
FL and JJF led the drafting of the manuscript and the development of the study rationale, design and methods. All authors contributed to the overall study rationale, design and methods development. All authors contributed to the drafting of the manuscript and approved the final version.
Robbie Foy, Liz Glidewell and Susan Michie are associate editors of Implementation Science. Jeremy Grimshaw and Fabiana Lorencatto are on the editorial board of Implementation Science. All decisions regarding this manuscript were handled by other editors.
Consent for publication
Ethics approval and consent to participate
This process evaluation has been reviewed and approved by a NHS Research Ethics Committee (ref: 14/EM/1295).
About this article
Cite this article
Lorencatto, F., Gould, N.J., McIntyre, S.A. et al. A multidimensional approach to assessing intervention fidelity in a process evaluation of audit and feedback interventions to reduce unnecessary blood transfusions: a study protocol. Implementation Sci 11, 163 (2016) doi:10.1186/s13012-016-0528-x
- Process evaluation
- Audit and feedback
- Research protocol
- Blood transfusion
- Randomised controlled trial