Skip to main content
Download PDF
Download ePub

Audit and feedback to reduce unwarranted clinical variation at scale: a realist study of implementation strategy mechanisms

Implementation Science volume 18, Article number: 71 (2023) Cite this article

Abstract

Background

Unwarranted clinical variation in hospital care includes the underuse, overuse, or misuse of services. Audit and feedback is a common strategy to reduce unwarranted variation, but its effectiveness varies widely across contexts. We aimed to identify implementation strategies, mechanisms, and contextual circumstances contributing to the impact of audit and feedback on unwarranted clinical variation.

Methods

Realist study examining a state-wide value-based healthcare program implemented between 2017 and 2021 in New South Wales, Australia. Three initiatives within the program included audit and feedback to reduce unwarranted variation in inpatient care for different conditions. Multiple data sources were used to formulate the initial audit and feedback program theory: a systematic review, realist review, program document review, and informal discussions with key program stakeholders. Semi-structured interviews were then conducted with 56 participants to refute, refine, or confirm the initial program theories. Data were analysed retroductively using a context-mechanism-outcome framework for 11 transcripts which were coded into the audit and feedback program theory. The program theory was validated with three expert panels: senior health leaders (n = 19), Agency for Clinical Innovation (n = 11), and Ministry of Health (n = 21) staff.

Results

The program’s audit and feedback implementation strategy operated through eight mechanistic processes. The strategy worked well when clinicians (1) felt ownership and buy-in, (2) could make sense of the information provided, (3) were motivated by social influence, and (4) accepted responsibility and accountability for proposed changes. The success of the strategy was constrained when the audit process led to (5) rationalising current practice instead of creating a learning opportunity, (6) perceptions of unfairness and concerns about data integrity, 7) development of improvement plans that were not followed, and (8) perceived intrusions on professional autonomy.

Conclusions

Audit and feedback strategies may help reduce unwarranted clinical variation in care where there is engagement between auditors and local clinicians, meaningful audit indicators, clear improvement plans, and respect for clinical expertise. We contribute theoretical development for audit and feedback by proposing a Model for Audit and Feedback Implementation at Scale. Recommendations include limiting the number of audit indicators, involving clinical staff and local leaders in feedback, and providing opportunities for reflection.

Peer Review reports

Background

Addressing unwarranted clinical variation in hospital care remains a key challenge to health system improvement. Despite all the improvements to health outcomes from modern medicine, care is not always provided in line with clinical practice guidelines [1, 2]. The underuse of effective services withholds potential beneficial outcomes from patients [3, 4]. Overuse or relying on insufficient and outdated evidence can waste valuable resources [5,6,7,8]. Misuse still befalls some patients who experience iatrogenic harm or adverse events [9]. These variations in care are considered unwarranted when they differ disproportionately from available evidence or the informed choices of patients [10, 11].

Clinical variation may occur due to a multitude of complex and interacting reasons. Warranted variation refers to situations where multiple effective options exist, and choice depends on patient preference. Sometimes there is genuine uncertainty among clinicians [12,13,14,15,16] in different contextual circumstances [17, 18] due to the unavailability of objective criteria to define appropriate care [19, 20]. However, when these situations arise in practice, medical opinion can tend to influence treatment choice [21,22,23]. Unwarranted variation is a value judgement about whether this variation is appropriate. Decisions to deviate from guideline recommendations can be influenced by clinician beliefs [24], preferences, or financial incentives [25, 26]. The evidence is well established that healthcare overuse and underuse may compromise patient care, increase inefficiency, and contribute to healthcare inequality [27]. Therefore, opportunities to change practice in a way that meets patients’ needs, according to clinical practice guidelines, are garnering the attention of health system decision-makers.

Audit and feedback is a common strategy used to reduce unwarranted clinical variation [28]. Providing healthcare professionals with performance feedback relative to specific target indicators may prompt modifications to their practice. Small improvements in provider compliance from audit and feedback have been demonstrated (median 4.3%); however, the range of effects between studies is substantial (− 9% to 70%) [29]. This wide variability indicates that not all audit and feedback strategies are optimally designed, operationalised, and components adequately specified. Many have not considered the previously hypothesised mechanisms for how audit and feedback works or can be bundled with other co-interventions [30, 31]. Furthermore, the success of audit and feedback strategies can be influenced by local contextual circumstances, in addition to the individual strategy components themselves [32]. Brown et al., developed the Clinical Performance Feedback Intervention Theory, which suggests that effective feedback is cyclical and sequential; becoming less effective if any one process within the cycle fails [32]. According to this theory and supporting empirical evidence, feedback interventions are more effective when they provide training and support regarding feedback [33], where there are health professionals with skills and experience in quality improvement, and the clinical topic under focus is targeted [33, 34]. Greater understanding of how and why audit and feedback work (or does not work) in different circumstances, is required to design better strategies.

Implementation context

In Australia, the New South Wales (NSW) Health system has invested in several programs to operationalise the concept of “value-based healthcare”. Value in health is defined as the net improvement of welfare across individuals within a society [35]. A value-based healthcare system is thus characterised by the optimisation of the welfare derived from health services including common or shared goals for equity and fairness, considering that resources used to achieve benefits in one manner are the same resources that can no longer be used for other purposes. Value-based healthcare is operationalised in NSW Health through several state-wide programs [36]. Leading Better Value Care (LBVC) is one flagship program administered by the NSW Ministry of Health, in partnership with the Agency for Clinical Innovation, Clinical Excellence Commission, and Cancer Institute NSW, to implement models of care for specific chronic conditions state-wide at scale. Three of the first eight LBVC initiatives focussed on reducing unwarranted variation in inpatient hospital care for patients with chronic obstructive pulmonary disease (COPD), chronic heart failure (CHF), and diabetes mellitus. These patient cohorts were selected due to high and persisting admission, readmission and complication rates, including small area and facility-level variation [37, 38]. Solutions included new or improved models of care, clinical audits, improved documentation, and patient coding processes [36]. A state-wide audit and feedback strategy was undertaken by the Agency for Clinical Innovation, delivered at the level of each hospital using medical record review and group feedback sessions to identify areas of practice to target with a quality improvement plan. The program was implemented across more than 100 facilities in the 2017–2018 financial year and presents a unique natural experiment to examine how and why the audit and feedback implementation strategy impacted unwarranted variation in care (Table 1). The initial COPD, CHF, and diabetes mellitus program logic are provided in Additional files 1 and 2.

Table 1 Description of the three LBVC initiatives and audit and feedback implementation strategy targeting inpatient variation in care
Full size table

Aim

The aim of this study was to identify how, why, and in what contexts the audit and feedback strategy contributed to the implementation of a state-wide value-based healthcare program to reduce unwarranted variation in care at scale. The main objectives were to:

  1. 1.

    Identify and articulate the audit and feedback implementation strategies used to reduce unwarranted variation in care for patients with CHF, COPD, and diabetes.

  2. 2.

    Determine the mechanisms (underlying social, cultural, structural, individual, and relational processes or events [39, 40]) by which the strategies operated to produce desired or undesired effects.

  3. 3.

    Investigate the impact of different contextual circumstances on the relationship between these implementation mechanisms and outcomes of interest.

Methods

Study design and rationale

A realist study was conducted to investigate how the LBVC audit and feedback strategy contributed to the implementation of a state-wide value-based healthcare program to reduce unwarranted variation in care. Realist study designs are theory-driven evaluations, based on a realist philosophy of science [41, 42]. They focus on generating plausible explanations for how and why programs work under different circumstances, referred to as “program theories” [43,44,45]. Therefore, the social responses to programs are considered the primary mechanisms of change and focus of inquiry, rather than the programs themselves [39, 46, 47]. This approach is well suited to address our objective to determine the mechanisms by which audit and feedback operate to produce desired or undesired effects. A greater understanding of these explanations may enable the replication of success, and avoid unintended outcomes when implementing healthcare improvement programs at scale [48,49,50].

The study was conducted and reported according to our published protocol [51] and the RAMESES II (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) reporting standards for realist studies [52] (Additional file 3). Ethical approval was provided by Macquarie University (Project ID 23816) and Hunter New England Human Research Ethics Committees (Project ID 2020/ETH02186). Three stages of research took place to posit, test, and refine program theories for the audit and feedback implementation strategy to reduce unwarranted variation in care (Fig. 1) [53]. In stage 1, initial program theories were identified for how the audit and feedback strategy was implemented to reduce unwarranted variation in care; in stage 2, these hypothesised program theories were then tested and refined; and in stage 3, the program theories were translated into generalisable implementation models. Multiple data sources were used across different time points to ensure the information best placed to inform the study was captured. Study protocol adaptations are described in Additional file 4.

Fig. 1
figure 1

Realist study stages. CMO = context–mechanism–outcome.  Adapted from Pawson et al. [42] and Sarkies et al. [51, 54]

Full size image

Setting

This realist study took place in NSW, Australia, examining the implementation of the LBVC program at scale between 2017 and 2021. In Australia, state and territory governments are responsible for healthcare planning and delivery by public hospitals. NSW Health provides universal access to health care services for Australia’s most populous state (~ 8.15 million people in 2022), which is operated by more than 130,000 staff spread across 234 public hospitals and multi-purpose facilities, over a geographical area of more than 800,000 km2 [36]. In relation to clinical care and quality improvement during the COVID-19 pandemic, throughout 2020 to 2021 NSW experienced among the lowest rates of COVID-19 internationally. In 2020, there were 4,782 confirmed cases of COVID-19 in NSW. In each month during the first half of 2021, there were few to no community-acquired cases. After the emergence of the Delta variant in June and Omicron in November 2021, there were a total of 280,601 confirmed cases in the second half of 2021 [55].

Participant recruitment and sampling

A maximum diversity, purposive sampling approach was taken to obtain a variety of organisational and individual perspectives considered best placed to provide information on the implementation of the initiatives. NSW Ministry of Health, Agency for Clinical Innovation, and local hospital staff eligible for participation were identified initially by the project partners and invited by the partners via email. Further respondents well placed to discuss the proposed program theories were identified through snowballing and were approached by the investigators directly via email. Informed consent was recorded verbally at the commencement of each interview.

Data collection and analysis

Stage 1: identifying the initial program theory

We developed and undertook a Realist Dialogic Approach to identify the initial program theory for the audit and feedback strategy to reduce unwarranted variation in care [53]. A program theory explains how and why a program is expected to work and makes an explicit connection between the activities undertaken in the program and the expected outcomes from those activities. The Realist Dialogic Approach process followed four phases to (1) understand relevant theories, (2) review academic and grey literature, (3) conduct informal discussions with key stakeholders, and (4) undertake research-group conversations. First, several sources of academic and grey literature were reviewed to identify hypotheses that could inform the development of an initial program theory for how and why the program was expected to work. This included a systematic review of studies examining implementation determinants for chronic condition hospital avoidance programs (13 articles) [45] and realist review conceptualising contexts, mechanisms, and outcomes for implementing large-scale, multisite hospital improvement initiatives (n = 51 articles) [56] to identify initial program theory propositions. The systematic review and realist review formed an initial source of data to identify theory propositions that could then be situated within the LBVC program using other sources of data. Second, public documents pertaining to the LBVC program were reviewed to modify and situate these propositions within the LBVC program of interest. Once these theory propositions were made specific to the LBVC program, informal discussions with key program stakeholders (~ 16 stakeholders) were also used to map any differences between how the implementation was planned and how it was operationalised in practice. The initial program theory, in the form of context-mechanism-outcome configurations and interview question guides, is provided in Additional file 5.

Stage 2: testing the hypothesised program theory

Internal documents pertaining to the LBVC program (126 documents) were reviewed to further situate the initial program theories within the program of interest. Semi-structured realist interviews [57, 58] to refute, refine, or confirm the initial program theories were then conducted with 56 participants via videoconference or telephone at the participants’ preferred time and location. The interviews were 30–60 min in duration, audio recorded and transcribed verbatim. They were conducted by two experienced researchers (MNS, EF-A) using an interview guide that was pilot-tested prior to data collection. The interviews sought to build a nuanced understanding of how the audit and feedback strategy was used to implement the program. We invited the participants to clarify or modify our hypothesised descriptions of the program implementation from their experience [59]. Data sources are summarised in Table 2.

Table 2 Summary of data sources
Full size table

Retroductive analysis was undertaken by five investigators (MNS, EF-A, CP, NR, JCL) concurrently with data collection using NVivo 20. This analysis used both inductive and deductive logic to identify causal mechanisms behind patterns of change. Investigators cycled between inductive and deductive logic across the program documents and interview transcripts, incorporating their own insights [60]. Transcripts were read in full and coded line-by-line according to a context-mechanism-outcome configuration framework [61,62,63]. Data were coded when links between the (1) contextual circumstances required to trigger change, (2) mechanisms generating change processes, and (3) outcomes produced when mechanisms are triggered in certain contextual circumstances for audit and feedback were identified [42, 64]. The quotes were categorised as either supporting, refuting, or refining the context-mechanism-outcome configuration. Memos were used to record the decision-making process according to processes outlined by Gilmore et al. [60]. Approximately 20% of the coded transcripts were checked by a second investigator and any disagreements between the investigators were resolved by discussion. Once coded, investigators engaged in group work consensus-building meetings to finalise each program theory [65]. Of the 56 interview transcripts, 11 were coded into the audit and feedback program theory, as they included direct reference to the audit and feedback strategy whereas the remaining 45 focussed on other strategies to be published separately (Table 3).

Table 3 Demographics of participants coded into the audit and feedback program theory
Full size table

Stage 3: translating into generalisable theoretical models for implementation

The program theories were presented to three healthcare quality expert panels for validation. A group of 19 Australian healthcare quality experts was assembled in a breakout meeting during the International Society for Quality in Health Care (ISQua) International Conference in Brisbane, Australia in 2022. A recommended model for the implementation of audit and feedback to reduce unwarranted clinical variation was proposed to the group, and questions were asked to seek confirmation, disconfirmation, and other interpretations (see Table 4 for example). Field notes were taken and incorporated with cross-case comparisons to determine how the same mechanism might produce different outcomes depending on the contextual circumstances. This assisted the research team to uncover potentially generative causal patterns (e.g., conditional causality) for the audit and feedback strategy and regularities and patterns between the proposed mechanisms and outcomes in certain contexts. A similar process was carried out when presenting the audit and feedback program theory to NSW Ministry of Health (n = 21) and Agency for Clinical Innovation staff (n = 11) to translate into generalisable theoretical models for implementation. These validation groups included some of the original interview participants from stages 1 and 2, providing the opportunity for re-engagement and further feedback from key stakeholders with deep insights into the program of interest, which is an important aspect of realist methods.

Table 4 Stage 3 feedback for an implementation model
Full size table

Results

Stage 1 led to a description of the initial audit and feedback strategy utilised in the LBVC program. The audit questions and indicators were developed by the Agency for Clinical Innovation from a review of the academic and grey literature related to each of the three conditions, assisted by a clinical expert reference group. This process did not necessarily always utilise the existing state-wide clinical networks. Criteria for feasibility, practicality, and the value of each audit indicator were considered in relation to the burden it might place on hospital clinicians. The indicators then underwent several rounds of refinement both internal and external to the Agency for Clinical Innovation before finalisation. Audits were then pilot-tested prior to their use. An audit team visited each participating hospital, which included two people from the Agency for Clinical Innovation (with variable clinical experience of the conditions) and two people nominated from the local hospital. Retrospective auditing of 40 case medical records took place without necessarily standardising the timeframe of case hospital admissions or whether cases were randomly or purposively sampled. Audit data were returned to the Agency for Clinical Innovation for data cleaning and visualisation prior to a feedback session at the hospital, typically within four to six weeks. Feedback was not always delivered by someone with a clinical background. Hospital staff at the feedback meeting were asked to identify three components of care that were being performed well and three components that could be improved. Unspecified ongoing support was offered by the Agency for Clinical Innovation for local hospital improvement efforts and follow-up audits were organised for hospitals on a voluntary basis. The audit and feedback strategy was refined over time to reduce the number of indicators and data points, increase the involvement of clinical staff or people with a clinical background, and prepare hospitals for the feedback sessions, and provide opportunities for reflection post-feedback.

The audit and feedback implementation strategy is hypothesised to operate through eight mechanisms under different contextual circumstances, underpinned by regular measurement and feedback of both clinical processes and outcomes. The strategy worked well when clinicians (1) felt ownership and buy-in to the process, (2) could make sense of the information provided, (3) were motivated by social influence, and (4) accepted responsibility and accountability for the proposed changes. The success of the strategy was constrained by (5) rationalisation of the status quo, (6) perceptions of unfairness and concerns about data integrity, (7) development of tokenistic improvements plans, and (8) perceived threats to professional autonomy. We report the manifestation of each audit and feedback mechanism to pinpoint key factors that determine success or where the strategy might falter (Table 5). Example quotes used to refute, refine, or confirm the audit and feedback program theory are provided in Table 6.

Table 5 Context-mechanism-outcome configuration for the audit and feedback program theory
Full size table
Table 6 Example quotes used to refute, refine, or confirm the audit and feedback program theory
Full size table

Ownership and buy in

Ownership and buy-in underpin the acceptability of the audit and feedback process. Ownership of or buying into the audit indicators and processes was described across multiple data sources as a key driver of practice change. Audits had to adequately capture local workflows in order for clinicians to make meaning of the audit data and buy-in to the process. Local workflows were better captured when audits were conducted by the Agency for Clinical Innovation with the participation of respected local hospital clinicians, such as senior medical, nursing, or allied health. Measures that held local meaning were regarded highly by clinicians. Audits conducted by an external institution remained productive where no clinician reluctance to engage had already taken hold.

Sensemaking of information feedback

Audits need to make sense to clinicians to be successful. This is about more than understanding the indicators, but also encompasses the ability to incorporate audit data with existing and alternate sources of information. Clinicians engaged well with the audit process when local leaders acted as a conduit between the Agency for Clinical Innovation and the hospitals, by introducing the auditors to key clinical stakeholders. Early and close engagement between the auditors and clinicians created a willingness to receive feedback about their performance against audit indicators, which could then be integrated with localised knowledge to make sense of the implications for those receiving care. Combining the externally validated feedback with local, codified knowledge, supported the development of local business cases that clinicians could use to justify improvement plans to hospital decision-makers.

Motivation and social influence

Audit and feedback are based on the presumption of motivating individual and collective behaviour change. It was common for clinicians to attribute variation from audit indicators to factors outside their control, in other words, an externalised locus of control. Audit results from other hospitals within the same hospital network opened their eyes to the discrepancy between patient outcomes being achieved compared to elsewhere. This was thought to create an intrinsic motivation to change or maintain performance, especially when the information was considered externally validated and paired with other externally driven incentives.

Responsibility and accountability

Reducing unwarranted variation in care requires clinicians to take responsibility for the variation and accountability for improvement efforts designed to change practice. Clinicians assumed responsibility for audit indicators when feedback was reinforced at the point of care to passionate and influential colleagues who had the power to effect practice change. Under these circumstances, the audit and feedback cycle was established as an ongoing process and could be used to garner hospital decision-maker support for local improvement plans.

Rationalisation of the status quo

For audit and feedback to be successful, it needed to overcome the proclivity for clinicians to rationalise the status quo when presented with performance feedback. Audit indicators were considered an impost where they lacked meaning and accuracy. Collecting too many indicators across too diffuse a sample of frustrated and disengaged stakeholders. When decisions on what will be measured come from elsewhere, local hospital staff were less likely to approve and support them. A limited partnership with local hospital clinicians contributed to this perception of too many audit indicators being assessed across the wrong patient cohorts, uncertainty around the evidence to support audit indicators, and selection of indicators where system-level barriers existed beyond clinicians’ control. For example, cynicism was thought to develop when system and organisational resourcing gaps were communicated to medical consultants who, in turn, felt these gaps were beyond their remit to address. Under these circumstances, clinicians tended to dismiss the audits and rationalise the status quo, leading to a general disengagement from the process and pursuit of their own local quality improvement priorities.

Perceptions of unfairness and concerns about data integrity

Concern about the integrity of the audits and a perception that the process was unfair, or setting clinicians up to fail, permeated some hospitals. Some perceived the variation at their hospital site to be warranted rather than unwarranted, or at least out of their control when considering issues at the level of the organisation or health system that they could not change on the ground. These hospitals were thought to be characterised by audits which did not capture local workflows, recognise system-level barriers, or the uniqueness of local settings and processes. Furthermore, those hospitals with immature systems for communication between executive and frontline staff were unable to manage expectations regarding the Agency for Clinical Innovation’s role in the audit and feedback process. Under these circumstances, clinicians perceived the audit as an unfair and unachievable process that set them up to fail. Results were misinterpreted as a performance management exercise rather than a learning opportunity and led to a defence of current practice instead of focussing on where care could be improved.

Improvement plans that are not followed

Improvement plans were expected to be developed by the local hospital staff after the audit results were fed back. However, in some cases, these were tokenistic, underdeveloped, or not implemented. Feedback did not provide a meaningful foundation for developing a quality improvement plan when it was delivered by someone external to the organisation from the Agency for Clinical Innovation, especially if they did not have a clinical background. Insufficient time for interpretation of the audit results and an unclear outline of the implementation support being offered by the Agency for Clinical Innovation further constrained the development and implementation of quality improvement plans at a local level, as well as the coordination of efforts at scale to implement changes system-wide. In response, some hospitals continued to focus on alternative quality improvement priorities that did not always align with the aims of the LBVC program.

Perceptions of threats to professional autonomy

Clinicians operate with a high degree of professional autonomy, which was thought to contribute to their resistance to any rigidity in how the audit indicators were measured and applied. Resistance to change or clinical inertia was apparent when the audit indicators developed by the Agency for Clinical Innovation did not appear to adequately capture local workflows or allow for adaptation. Frustration that clinician expertise was not being respected emphasised this perception of threatened professional autonomy, particularly when feedback was perceived as an external directive to mandate change. These issues manifested when rigid criteria were applied to the measurement of audit indicators rather than considering the indicators as broad principles of care that could be tailored to local settings. Using non-medical clinicians to provide audit feedback to medical staff further reinforced this perceived threat to professional autonomy.

We propose a Model for Audit and Feedback Implementation at Scale. This model maps the cyclical chain of events underlying the effect on reducing unwarranted clinical variation in inpatient hospital care (Fig. 2) and those underlying unsuccessful reduction of unwarranted clinical variation (Fig. 3). We hypothesise that audit and feedback works through one of four mechanisms (1) triggering ownership and buy in to the improvement process; (2) helping making sense of clinical performance; (3) creating social influence in the form of peer competition; and (4) assuming responsibility for the audit measures. These cyclical and sequential processes are theorised to build a case for change, leading to the development and implementation of improvement plans, which in turn reduce unwarranted variation in care. Contextual influences are represented as yellow ellipses identifying points in the chain of events. Unsuccessful audit and feedback lead to unintended consequences through one of four mechanisms (1) undermining trust in the results; (2) perception of audit as unfair and unachievable; (3) feedback not providing a foundation for quality improvement; and (4) feedback perceived as directive without respect for clinical expertise. These cyclical and sequential processes are theorised to lead clinicians to feel measures do not capture their work, leading to the defence of current practice and pursuit of local priorities, which in turn does not reduce unwarranted variation in care. Contextual influences are represented as yellow ellipses identifying points in the chain of events.

Fig. 2
figure 2

Model for successful audit and feedback implementation at scale

Full size image
Fig. 3
figure 3

Model for unsuccessful audit and feedback implementation at scale

Full size image

Discussion

Our study demonstrates that the success of audit and feedback implementation strategies for reducing unwarranted variation in care is contingent upon a variety of contextual circumstances. The program theories describe a series of potential mechanistic causal pathways, which are considered the social responses to audit and feedback. It is these responses that determine the strategy’s success. Different actions taken by various actors can trigger these social responses.

Meaningful partnerships between external auditors and local clinicians appear pivotal in enabling successful audit and feedback cycles. The opportunity for preparation prior to receiving feedback as well as time for post feedback reflection seems to provide a conducive environment for change. Generating sufficient change valence was thought to motivate clinicians, e.g. the more clinicians value the change, the more resolve they will feel to implement the change. Respecting and working in collaboration with clinicians’ professional duty to improve care for their patients more often saw local support for changes in line with the LBVC program. However, there were no magic bullets. The mechanistic causal pathways identified were conditional on various contextual factors. In other words, the causal link between audit and feedback and reduced variation occurs via an underlying generative process existing at certain points in time and space [66]. These casual associations are produced by mechanistic processes, which are themselves important to study.

Implementing an audit and feedback strategy at scale, across an entire health system, is materially different to scaling up a strategy that has already been implemented in other hospital sites. Concurrently delivering an audit and feedback strategy to change processes and systems of care across multiple hospitals requires a macro-view of implementation, by coordinating efforts so that the outcome produced is more than the sum of its parts. This requires a degree of flexibility in how strategies are delivered to balance both fidelity to the intended intervention and adaptation to local contextual needs. Motivating clinicians by comparing their audit results with similar peer hospitals can be achieved by including normative measures for comparison within written reports [67]. For example, in a large audit and feedback trial, Weissman et al. reported that audit recipients that were provided with comparisons to the top 10% of peers improved processes of care more than those who were only compared to the median peer performance [68]. However, feedback needs to describe both desirable and achievable processes and behavioural changes [69], as recipients must believe they have control over that behaviour and commit to change to avoid the risk of recipients rationalising the status quo [70]. Furthermore, our findings reinforce that the amount of feedback in terms of change targets and peer comparisons should be limited to avoid excessive cognitive load and mental effort required to process the feedback, so recipients can more easily make sense of the information provided [71]. The audit and feedback strategy implemented within the LBVC program was not initially designed in a way that could easily enable clinicians to compare audit results data between hospitals from different hospital networks because the sampling method and timeframe was not standardised. This made it difficult for implementors to coordinate efforts at scale across the state-wide health system.

Providing feedback on components of care that align with the existing goals and priorities of individual clinicians and organisations could be achieved by conducting a local needs assessment before providing feedback. That local needs assessment might facilitate the engagement and attention of recipients and maintain their self-efficacy and control of the process [71]. However, this does not mean that clinician values and norms cannot be challenged. Foy et al. previously demonstrated that clinical practice recommendations which were considered incompatible with clinician values and norms can produce greater behaviour changes than those considered compatible [72]. This is likely due to the greater potential for improvements and the likelihood that there might only be limited room for improvement for recommendations already compatible with clinician norms.

Our findings that local leaders could provide a beneficial conduit between the external auditor and hospitals aligns with previous research supporting feedback as more effective when delivered by senior colleagues [73]. Social pressures have been shown to influence intentions to change clinical practice when driven by respected local hospital clinicians [73]. In circumstances without the brokerage of local clinical leaders, it is possible that the feedback could have been perceived as an external directive to mandate change. Feedback is considered less effective when delivered by a regulatory agency, as it can be construed as punitive rather than supportive [69, 74]. Therefore, extra efforts might need to be made by external audit and feedback providers to ensure psychological safety, target behaviours considered important to the recipient, and provide reassuring messages to avoid invoking a defensive reaction to the feedback [71].

Our study findings broadly align with the “high-confidence hypotheses” put forward by Brown et al.’s Clinical Performance Feedback Intervention Theory (CP-FIT) [32]. However, we have identified where these findings can be used to refine this existing theory and contribute to the rich agenda focusing on how to optimise the effects of audit and feedback interventions. Our audit and feedback program theories for large-scale system change challenge three of the CP-FIT hypotheses: specificity, exclusions, and delivery to a group. The specificity hypothesis posits that feedback should be provided based on individual health professional performance rather than at the team or organisation level [75]. Our study identified mechanisms by which audit and feedback could be provided in group settings and produce hospital-wide quality improvement plans. Changing systems of care requires a focus on multilevel, multifactorial, and multidimensional processes that often are beyond the level of an individual clinician. This also appears to contradict the delivery to a group hypothesis that feedback should be delivered to groups of recipients [76]. We posit that feedback should be provided at the level where the behaviour change is required to achieve the desired outcome (e.g. individual, team, organisational in combination). Levesque and Sutherland previously outlined eight levers for change enabled by performance information that account for these different levels of the system where behaviour change is desired [77], and provide a potential framework for integrating future audit and feedback strategies with large system value-based healthcare programs. According to our findings, the exclusions hypothesis that recipients be allowed to exclude patients they consider inappropriate from the audit and feedback process [78] potentially risks gaming of the audit and feedback process. In our study, retrospective audits of medical records without standardising the timeframe of hospital admissions or sampling approaches reportedly led to high levels of heterogeneity in the cases selected and limited the ability to support state-wide implementation at scale by comparing audit results across hospitals. We posit that a standardised timeframe and sampling approach be used to ensure audits most closely reflect real-world clinical practice and ensure external generalisability of audit results.

Strengths and limitations

Our realist study integrated context within hypothesised causal models. In developing theory for implementation strategies in healthcare, it is not uncommon to divide contextual factors, theoretical mechanisms, and implementation outcomes, and study these factors in isolation [79, 80]. This can lead to competing theories of change in different contextual circumstances [54]. However, it is more likely that they are alternate theories of change which could occur in any setting depending on interactions with these contextual factors. This concept of latent or unrealised mechanisms in realist research paradigms allowed the elucidation of “dormant” change mechanisms which could be triggered by certain contextual circumstances [81].

The three LBVC initiatives targeting inpatient variation in care were complex interventions that included more than just audit and feedback. It is possible that our focus on the audit and feedback strategy within these initiatives did not sufficiently capture complex interactions between the audit and feedback strategy and other approaches, such as defining best practice management, capability building to support best practice management, and systematising local processes to detect and address unwarranted clinical variation. The research team was not involved in developing or delivering the audit and feedback strategy, limiting our ability to report the strategy according to the AACTT framework [82]. The strategy was also modified between sites and over time, further limiting our ability to provide specific descriptions according to the AACTT framework. Despite a large amount of qualitative data collected, relatively few interview participants provided information that could be coded to the audit and feedback program theory. However, realist hypotheses are not reliant on theoretical saturation obtained in a pre-specified number of qualitative interviews but through a focus on relevance, credibility, and rigour [57]. Furthermore, the limited availability of quantitative data outlined in our study protocol constrained the ability to triangulate qualitative and quantitative data to provide a more robust program theory for audit and feedback in large-system value-based healthcare initiatives.

Conclusion

Our realist study identified eight mechanisms by which audit and feedback implementation strategies may impact unwarranted clinical variation in hospital care. Ownership and buy-in were more likely to be achieved when audits were conducted in collaboration with respected local hospital clinicians to better capture local workflows. Furthermore, where these local leaders facilitated engagement between the external auditors and local clinicians, performance on audit indicators was better able to be integrated with localised knowledge to make sense of the practice change implications. Comparisons with peer equivalent hospitals motivated a sense of positive social influence if clinicians considered the audit indicators and processes of care to be within their responsibility and locus of control. Where there was no meaningful partnership between auditors and local clinicians, too many audit indicators were assessed that lacked the necessary meaning and accuracy to be actionable, leading to clinicians rationalising existing care processes rather than focussing on potential areas for quality improvement. This lack of partnership can create a perception of unfairness when local workflows are not adequately accounted for in the audits. Improvement plans were underdeveloped or not implemented when sufficient time to interpret audit results and a clear indication of the level of ongoing implementation support was not provided. Finally, where clinical expertise was not respected because of rigid audit indicator criteria, the process was considered a threat by some to professional autonomy.

Recommendations for future audit and feedback strategies include limiting the number of audit indicators and data points, ensuring the involvement of clinical staff and local leaders in delivering feedback, preparing hospital staff for the feedback sessions, and providing opportunities for reflection post-feedback. Future research could determine how to optimise benefits and avoid unintended outcomes by focussing on head-to-head comparisons of audit and feedback strategies, incorporating these elements with embedded process evaluations to determine the interaction effect of the hypothesised contextual factors in our program theories.

Availability of data and materials

The datasets generated and analysed during the current study are not publicly available due to potential concerns regarding the identifiability of participants but may be available from the corresponding author on reasonable request.

Abbreviations

NSW:

New South Wales

LBVC:

Leading Better Value Care

COPD:

Chronic obstructive pulmonary disease

CHF:

Chronic heart failure

BGL:

Blood glucose level

ACI:

Agency for Clinical Innovation

GP:

General practitioner

CNC:

Clinical nurse specialist

References

  1. Morgan S, Cunningham C, Hanley G, Mooney D. The British Columbia medical and hospital atlas: a companion to the British Columbia Rx atlas, 2nd edition. British Columbia: Centre for Health Services and Policy Research (CHSPR); 2009.

  2. Dartmouth Medical School. The Dartmouth atlas of health care. Chicago: American Hospital Association; 1996.

  3. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348(26):2635–45. https://doi.org/10.1056/NEJMsa022615.

    Article  PubMed  Google Scholar 

  4. Runciman WB, Hunt TD, Hannaford NA, Hibbert PD, Westbrook JI, Coiera EW, et al. CareTrack: assessing the appropriateness of health care delivery in Australia. Med J Aust. 2012;197(2):100–5. https://doi.org/10.5694/mja12.10510.

    Article  PubMed  Google Scholar 

  5. Breton ER, Fuemmeler BF, Abroms LC. Weight loss-there is an app for that! But does it adhere to evidence-informed practices? Transl Behav Med. 2011;1(4):523–9. https://doi.org/10.1007/s13142-011-0076-5.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Haines TP, Bowles K-A, Mitchell D, O’Brien L, Markham D, Plumb S, et al. Impact of disinvestment from weekend allied health services across acute medical and surgical wards: 2 stepped-wedge cluster randomised controlled trials. PLoS Med. 2017;14(10): e1002412. https://doi.org/10.1371/journal.pmed.1002412.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Kelly P, Baker G, McAdam C, Milton K, Richards J, Foster C, et al. Critique of “The physical activity myth” paper: discussion of flawed logic and inappropriate use of evidence. Br J Sports Med. 2016;50(20):1230–1. https://doi.org/10.1136/bjsports-2015-095120.

    Article  PubMed  Google Scholar 

  8. Saini V, Brownlee S, Elshaug AG, Glasziou P, Heath I. Addressing overuse and underuse around the world. Lancet. 2017;390(10090):105–7. https://doi.org/10.1016/S0140-6736(16)32573-9.

    Article  PubMed  Google Scholar 

  9. Sarkies MN, Bowles KA, Skinner EH, Haas R, Mitchell D, O’Brien L, et al. Do daily ward interviews improve measurement of hospital quality and safety indicators? A prospective observational study. J Eval Clin Pract. 2016;22(5):792–8. https://doi.org/10.1111/jep.12543.

    Article  PubMed  Google Scholar 

  10. Sutherland K, Levesque J-F. Unwarranted clinical variation in health care: definitions and proposal of an analytic framework. J Eval Clin Pract. 2019;26. https://doi.org/10.1111/jep.13181.

  11. Wennberg JE. Unwarranted variations in healthcare delivery: implications for academic medical centres. BMJ. 2002;325(7370):961–4. https://doi.org/10.1136/bmj.325.7370.961.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Birkmeyer JD, Reames BN, McCulloch P, Carr AJ, Campbell WB, Wennberg JE. Understanding of regional variation in the use of surgery. Lancet. 2013;382(9898):1121–9. https://doi.org/10.1016/S0140-6736(13)61215-5.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Lutfey KE, Link CL, Marceau LD, Grant RW, Adams A, Arber S, et al. Diagnostic certainty as a source of medical practice variation in coronary heart disease: results from a cross-national experiment of clinical decision making. Med Decis Making. 2009;29(5):606–18. https://doi.org/10.1177/0272989X09331811.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Mayer M, Naylor J, Harris I, Badge H, Adie S, Mills K, et al. Evidence base and practice variation in acute care processes for knee and hip arthroplasty surgeries. PLoS ONE. 2017;12(7): e0180090. https://doi.org/10.1371/journal.pone.0180090.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  15. Offerhaus PM, Geerts C, de Jonge A, Hukkelhoven CWPM, Twisk JWR, Lagro-Janssen ALM. Variation in referrals to secondary obstetrician-led care among primary midwifery care practices in the Netherlands: a nationwide cohort study. BMC Pregnancy Childbirth. 2015;15(1):42. https://doi.org/10.1186/s12884-015-0471-x.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Wennberg J, Gittelsohn. Small area variations in health care delivery. Science. 1973;182(4117):1102–8. https://doi.org/10.1126/science.182.4117.1102.

  17. de Jong JD, Westert GP, Lagoe R, Groenewegen PP. Variation in hospital length of stay: do physicians adapt their length of stay decisions to what is usual in the hospital where they work? Health Serv Res. 2006;41(2):374–94. https://doi.org/10.1111/j.1475-6773.2005.00486.x.

  18. Moen EL, Bynum JP, Austin AM, Skinner JS, Chakraborti G, O’Malley AJ. Assessing variation in implantable cardioverter defibrillator therapy guideline adherence with physician and hospital patient-sharing networks. Med Care. 2018;56(4):350–7. https://doi.org/10.1097/MLR.0000000000000883.

    Article  PubMed  PubMed Central  Google Scholar 

  19. McAlister FA, Lin M, Bakal J, Dean S. Frequency of low-value care in Alberta, Canada: a retrospective cohort study. BMJ Qual Saf. 2018;27(5):340–6. https://doi.org/10.1136/bmjqs-2017-006778.

    Article  PubMed  Google Scholar 

  20. Segal JB, Bridges JFP, Chang H-Y, Chang E, Nassery N, Weiner J, et al. Identifying possible indicators of systematic overuse of health care procedures with claims data. Med Care. 2014;52(2):157–63. https://doi.org/10.1097/MLR.0000000000000052.

    Article  PubMed  Google Scholar 

  21. Wennberg JE, Freeman JL, Culp WJ. Are hospital services rationed in New Haven or over-utilised in Boston? Lancet. 1987;1(8543):1185–9. https://doi.org/10.1016/s0140-6736(87)92152-0.

    Article  PubMed  CAS  Google Scholar 

  22. Wennberg JE, Freeman JL, Shelton RM, Bubolz TA. Hospital use and mortality among Medicare beneficiaries in Boston and New Haven. N Engl J Med. 1989;321(17):1168–73. https://doi.org/10.1056/NEJM198910263211706.

    Article  PubMed  CAS  Google Scholar 

  23. Fisher ES, Wennberg JE, Stukel TA, Sharp SM. Hospital readmission rates for cohorts of Medicare beneficiaries in Boston and New Haven. N Engl J Med. 1994;331(15):989–95. https://doi.org/10.1056/nejm199410133311506.

    Article  PubMed  CAS  Google Scholar 

  24. Cutler D, Skinner JS, Stern AD, Wennberg D. Physician beliefs and patient preferences: a new look at regional variation in health care spendingf. Am Econ J Econ Policy. 2019;11(1):192–221. https://doi.org/10.1257/pol.20150421.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Elshaug AG, Watt AM, Mundy L, Willis CD. Over 150 potentially low-value health care practices: an Australian study. Med J Aust. 2012;197(10):556–60. https://doi.org/10.5694/mja12.11083.

    Article  PubMed  Google Scholar 

  26. Saini V, Garcia-Armesto S, Klemperer D, Paris V, Elshaug AG, Brownlee S, et al. Drivers of poor medical care. Lancet. 2017;390(10090):178–90. https://doi.org/10.1016/S0140-6736(16)30947-3.

    Article  PubMed  Google Scholar 

  27. Braithwaite J, Glasziou P, Westbrook J. The three numbers you need to know about healthcare: the 60–30-10 Challenge. BMC Med. 2020;18(1):102. https://doi.org/10.1186/s12916-020-01563-4.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Gauld R, Horwitt J, Williams S, Cohen AB. What Strategies Do US Hospitals Employ to Reduce Unwarranted Clinical Practice Variations? Am J Med Qual. 2011;26(2):120–6. https://doi.org/10.1177/1062860610375107.

    Article  PubMed  Google Scholar 

  29. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012(6):CD000259. https://doi.org/10.1002/14651858.CD000259.pub3.

  30. Colquhoun HL, Brehaut JC, Sales A, Ivers N, Grimshaw J, Michie S, et al. A systematic review of the use of theory in randomized controlled trials of audit and feedback. Implement Sci. 2013;8(1):66. https://doi.org/10.1186/1748-5908-8-66.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Sales A, Smith J, Curran G, Kochevar L. Models, strategies, and tools. Theory in implementing evidence-based findings into health care practice. J Gen Intern Med. 2006;21 Suppl 2(S2):S43–9. https://doi.org/10.1111/j.1525-1497.2006.00362.x.

  32. Brown B, Gude WT, Blakeman T, van der Veer SN, Ivers N, Francis JJ, et al. Clinical Performance Feedback Intervention Theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implement Sci. 2019;14(1):40. https://doi.org/10.1186/s13012-019-0883-5.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Morrell C, Harvey G, Kitson A. Practitioner based quality improvement: a review of the Royal College of Nursing’s dynamic standard setting system. Qual Health Care. 1997;6(1):29–34. https://doi.org/10.1136/qshc.6.1.29.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  34. Grant AM, Guthrie B, Dreischulte T. Developing a complex intervention to improve prescribing safety in primary care: mixed methods feasibility and optimisation pilot study. BMJ Open. 2014;4(1): e004153. https://doi.org/10.1136/bmjopen-2013-004153.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Drummond M, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. Fourth edition. ed. Oxford; New York: Oxford University Press; 2015. xiii, 445.

  36. Koff E, Lyons N. Implementing value-based health care at scale: the NSW experience. Med J Aust. 2020;212(3):104-6.e1. https://doi.org/10.5694/mja2.50470.

    Article  PubMed  Google Scholar 

  37. Health Economics and Evaluation Team. Reducing unwarrented clinical variation in chronic obstructive pulmonary disease and chronic heart failure: monitoring and evaluation plan. Sydney: NSW Agency for Clinical Innovation; 2017. Available from: https://aci.health.nsw.gov.au/statewide-programs/lbvc/chronic-heart-failure.

  38. Health Economics and Evaluation Team. Inpatient Management of Diabetes Mellitus: Monitoring and evaluation plan. Australia: NSW Agency for Clinical Innovation; 2017. Available from: https://aci.health.nsw.gov.au/statewide-programs/lbvc/inpatient-management-of-diabetes-mellitus.

  39. de Souza DE. Elaborating the Context-Mechanism-Outcome configuration (CMOc) in realist evaluation: A critical realist perspective. Evaluation. 2013;19(2):141–54. https://doi.org/10.1177/1356389013485194.

    Article  Google Scholar 

  40. Lewis CC, Boyd MR, Walsh-Bailey C, Lyon AR, Beidas R, Mittman B, et al. A systematic review of empirical studies examining mechanisms of implementation in health. Implement Sci. 2020;15(1):21. https://doi.org/10.1186/s13012-020-00983-3.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Pawson R. Evidence-based policy. Thousand Oaks: SAGE Publications; 2006. p. 208.

    Book  Google Scholar 

  42. Pawson R, Tilley N. Realistic evaluation. Thousand Oaks: SAGE Publications; 1997. p. 256.

    Google Scholar 

  43. Funnell S, Rogers P. Purposeful program theory: effective use of theories of change and logic models. San Francisco: Wiley; 2011.

    Google Scholar 

  44. Jagosh J. Retroductive theorizing in Pawson and Tilley’s applied scientific realism. J Crit Realism. 2020;19:1–10. https://doi.org/10.1080/14767430.2020.1723301.

    Article  Google Scholar 

  45. Sarkies M, Long J, Pomare C, Wu W, Clay-Williams R, Nguyen H, et al. Avoiding unnecessary hospitalisation for patients with chronic conditions: a systematic review of implementation determinants for hospital avoidance programmes. Implement Sci. 2020;15. https://doi.org/10.1186/s13012-020-01049-0.

  46. Jackson S, Kolla G. A new realistic evaluation analysis method: linked coding of context, mechanism and outcome relationships. Am J Eval. 2012;33:339–49. https://doi.org/10.1177/1098214012440030.

    Article  Google Scholar 

  47. Wong G, Pawson R, Owen L. Policy guidance on threats to legislative interventions in public health: A realist synthesis. BMC Public Health. 2011;11:222. https://doi.org/10.1186/1471-2458-11-222.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Lacouture A, Breton E, Guichard A, Ridde V. The concept of mechanism from a realist approach: A scoping review to facilitate its operationalization in public health program evaluation. Implement Sci. 2015;10. https://doi.org/10.1186/s13012-015-0345-7.

  49. Sarkies M, Robinson S, Ludwick T, Braithwaite J, Nilsen P, Aarons G, et al. Understanding implementation science from the standpoint of health organisation and management: an interdisciplinary exploration of selected theories, models and frameworks. J Health Organ Manag. 2021. https://doi.org/10.1108/JHOM-02-2021-0056.

  50. Braithwaite J, Churruca K, Long J, Ellis L, Herkes J. When complexity science meets implementation science: A theoretical and empirical analysis of systems change. BMC Med. 2018;16. https://doi.org/10.1186/s12916-018-1057-z.

  51. Sarkies M, Francis-Auton E, Long J, Partington A, Pomare C, Nguyen H, et al. Implementing large-system, value-based healthcare initiatives: A realist study protocol for seven natural experiments. BMJ Open. 2020;10: e044049. https://doi.org/10.1136/bmjopen-2020-044049.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Wong G, Westhorp G, Manzano A, Greenhalgh J, Jagosh J, Greenhalgh T. RAMESES II reporting standards for realist evaluations. BMC Med. 2016;14:1–18. https://doi.org/10.1186/s12916-016-0643-1.

    Article  CAS  Google Scholar 

  53. Francis-Auton E, Sarkies MN, Pomare C, Long JC, Hardwick R, Nguyen HM, et al. Real Talk: A Realist Dialogic Approach in a Realist Evaluation. Int J Qual Methods. 2022;21:16094069221120748. https://doi.org/10.1177/16094069221120748.

    Article  Google Scholar 

  54. Sarkies M, Francis-Auton E, Long J, Pomare C, Hardwick R, Braithwaite J. Making implementation science more real. BMC Med Res Methodol. 2022;22. https://doi.org/10.1186/s12874-022-01661-2.

  55. Bureau of Health Information. New South Wales and the COVID-19 pandemic from 2020 to 2022. Sydney: BHI; 2023. Available from: https://www.bhi.nsw.gov.au/BHI_reports/healthcare_in_focus/new_south_wales_and_the_covid-19_pandemic_from_2020_to_2022.

  56. Long JC, Sarkies MN, Auton EF, Nguyen HM, Pomare C, Hardwick R, et al. Conceptualising contexts, mechanisms and outcomes for implementing large-scale, multisite hospital improvement initiatives: a realist synthesis. BMJ Open. 2022;12(5): e058158. https://doi.org/10.1136/bmjopen-2021-058158.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Manzano A. The craft of interviewing in realist evaluation. Evaluation. 2016;22. https://doi.org/10.1177/1356389016638615.

  58. Mukumbang F, Marchal B, Van Belle S, Van Wyk B. Using the realist interview approach to maintain theoretical awareness in realist studies. Qual Res. 2019;20:31. https://doi.org/10.1177/1468794119881985.

    Article  Google Scholar 

  59. Mukumbang FC, Marchal B, Van Belle S, van Wyk B. Using the realist interview approach to maintain theoretical awareness in realist studies. Qual Res. 2020;20(4):485–515. https://doi.org/10.1177/1468794119881985.

    Article  Google Scholar 

  60. Gilmore B, McAuliffe E, Power J, Vallières F. Data analysis and synthesis within a realist evaluation: toward more transparent methodological approaches. Int J Qual Methods. 2019;18:1–11. https://doi.org/10.1177/1609406919859754.

    Article  Google Scholar 

  61. Hewitt G, Sims S, Harris R. The realist approach to evaluation research: an introduction. Int J Ther Rehabil. 2012;19:250–60. https://doi.org/10.12968/ijtr.2012.19.5.250.

    Article  Google Scholar 

  62. Mark M, Henry G, Julnes G. A realist theory of evaluation practice. N Dir Eval. 2004;1998:3–32. https://doi.org/10.1002/ev.1098.

    Article  Google Scholar 

  63. Salter KL, Kothari A. Using realist evaluation to open the black box of knowledge translation: a state-of-the-art review. Implement Sci. 2014;9:115. https://doi.org/10.1186/s13012-014-0115-y.

    Article  PubMed  PubMed Central  Google Scholar 

  64. Rogers P. Program Theory: Not Whether Programs Work but How They Work. 2002. 209–32.

  65. De Brún A, McAuliffe E. Identifying the context, mechanisms and outcomes underlying collective leadership in teams: building a realist programme theory. BMC Health Serv Res. 2020;20. https://doi.org/10.1186/s12913-020-05129-1.

  66. Blossfeld HP. Causation as a Generative Process. The Elaboration of an Idea for the Social Sciences and an Application to an Analysis of an Interdependent Dynamic Social System. In: Engelhardt H, Kohler HP, Fürnkranz-Prskawetz A, editors. Causal Analysis in Population Studies. The Springer Series on Demographic Methods and Population Analysis, vol 23. Dordrecht: Springer; 2009. https://doi.org/10.1007/978-1-4020-9967-0_5.

  67. Kiefe CI, Allison JJ, Williams OD, Person SD, Weaver MT, Weissman NW. Improving quality improvement using achievable benchmarks for physician feedback: a randomized controlled trial. JAMA. 2001;285(22):2871–9. https://doi.org/10.1001/jama.285.22.2871.

    Article  PubMed  CAS  Google Scholar 

  68. Weissman NW, Allison JJ, Kiefe CI, Farmer RM, Weaver MT, Williams OD, et al. Achievable benchmarks of care: the ABCs of benchmarking. J Eval Clin Pract. 1999;5(3):269–81. https://doi.org/10.1046/j.1365-2753.1999.00203.x.

    Article  PubMed  CAS  Google Scholar 

  69. Hysong SJ, Best RG, Pugh JA. Audit and feedback and clinical practice guideline adherence: Making feedback actionable. Implement Sci. 2006;1(1):9. https://doi.org/10.1186/1748-5908-1-9.

    Article  PubMed  PubMed Central  Google Scholar 

  70. Locke EA, Latham GP. Building a practically useful theory of goal setting and task motivation. A 35-year odyssey. Am Psychol. 2002;57(9):705–17. https://doi.org/10.1037//0003-066x.57.9.705.

  71. Colquhoun HL, Carroll K, Eva KW, Grimshaw JM, Ivers N, Michie S, et al. Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses. Implement Sci. 2017;12(1):117. https://doi.org/10.1186/s13012-017-0646-0.

    Article  PubMed  PubMed Central  Google Scholar 

  72. Foy R, MacLennan G, Grimshaw J, Penney G, Campbell M, Grol R. Attributes of clinical recommendations that influence change in practice following audit and feedback. J Clin Epidemiol. 2002;55(7):717–22. https://doi.org/10.1016/s0895-4356(02)00403-1.

    Article  PubMed  Google Scholar 

  73. Ajzen I. The theory of planned behavior. Organ Behav Hum Decis Process. 1991;50(2):179–211. https://doi.org/10.1016/0749-5978(91)90020-T.

    Article  Google Scholar 

  74. Kluger AN, Van Dijk D. Feedback, the various tasks of the doctor, and the feedforward alternative. Med Educ. 2010;44(12):1166–74. https://doi.org/10.1111/j.1365-2923.2010.03849.x.

    Article  PubMed  Google Scholar 

  75. Nessim C, Bensimon CM, Hales B, Laflamme C, Fenech D, Smith A. Surgical site infection prevention: a qualitative analysis of an individualized audit and feedback model. J Am Coll Surg. 2012;215(6):850–7. https://doi.org/10.1016/j.jamcollsurg.2012.08.007.

    Article  PubMed  Google Scholar 

  76. Hysong SJ, Knox MK, Haidet P. Examining clinical performance feedback in Patient-Aligned Care Teams. J Gen Intern Med. 2014;29 Suppl 2(Suppl 2):S667–74. https://doi.org/10.1007/s11606-013-2707-7.

  77. Levesque J-F, Sutherland K. What role does performance information play in securing improvement in healthcare? a conceptual framework for levers of change. BMJ Open. 2017;7(8): e014825. https://doi.org/10.1136/bmjopen-2016-014825.

    Article  PubMed  PubMed Central  Google Scholar 

  78. Chadwick LM, MacPhail A, Ibrahim JE, McAuliffe L, Koch S, Wells Y. Senior staff perspectives of a quality indicator program in public sector residential aged care services: a qualitative cross-sectional study in Victoria. Australia Aust Health Rev. 2016;40(1):54–62. https://doi.org/10.1071/ah14197.

    Article  PubMed  Google Scholar 

  79. May CR, Johnson M, Finch T. Implementation, context and complexity. Implement Sci. 2016;11(1):141. https://doi.org/10.1186/s13012-016-0506-3.

    Article  PubMed  PubMed Central  Google Scholar 

  80. Wensing M, Grol R. Knowledge translation in health: how implementation science could contribute more. BMC Med. 2019;17(1):88. https://doi.org/10.1186/s12916-019-1322-9.

    Article  PubMed  PubMed Central  Google Scholar 

  81. Jagosh J. Realist synthesis for public health: building an ontologically deep understanding of how programs work, for whom, and in which contexts. Annu Rev Public Health. 2019;40:361–72. https://doi.org/10.1146/annurev-publhealth-031816-044451.

    Article  PubMed  Google Scholar 

  82. Presseau J, McCleary N, Lorencatto F, Patey AM, Grimshaw JM, Francis JJ. Action, actor, context, target, time (AACTT): a framework for specifying behaviour. Implement Sci. 2019;14(1):102. https://doi.org/10.1186/s13012-019-0951-x.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank our participants and our partners at the NW Ministry of Health, Agency for Clinical Innovation, Bureau of Health Information, and Local Health Districts and Networks across the state of New South Wales.

Funding

This work was supported by the Medical Research Future Fund (MRFF) (APP1178554, CI Braithwaite). The funding arrangement ensured the funder has not and will not have any role in study design, collection, management, analysis and interpretation of data, drafting of manuscripts and decision to submit for publication.

Author information

Authors and Affiliations

  1. Australian Institute of Health Innovation, Macquarie University, Sydney, Australia

    Mitchell Sarkies, Emilie Francis-Auton, Janet Long, Natalie Roberts, Johanna Westbrook, Peter Hibbert & Jeffrey Braithwaite

  2. School of Health Sciences, University of Sydney, Sydney, Australia

    Mitchell Sarkies

  3. Centre for Primary Health Care and Equity, University of New South Wales, Kensington, NSW, Australia

    Jean-Frederic Levesque

  4. NSW Agency for Clinical Innovation, Sydney, Australia

    Jean-Frederic Levesque & Kim Sutherland

  5. Bureau of Health Information, St Leonards, NSW, Australia

    Diane E. Watson

  6. Peninsula Medical School, Faculty of Health, University of Plymouth, Plymouth, UK

    Rebecca Hardwick

  7. NSW Ministry of Health, Sydney, Australia

    Gary Disher

  8. Allied Health and Human Performance, IIMPACT in Health, University of South Australia, Adelaide, SA, Australia

    Peter Hibbert

Authors
  1. Mitchell Sarkies
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Emilie Francis-Auton
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Janet Long
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Natalie Roberts
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Johanna Westbrook
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jean-Frederic Levesque
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Diane E. Watson
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Rebecca Hardwick
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Kim Sutherland
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Gary Disher
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Peter Hibbert
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Jeffrey Braithwaite
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JB, JFL, JW, and DEW conceptualised the study. JB, MNS, EFA, JCL, and RH designed the data collection. MNS and EFA conducted the interviews and MNS, EFA, NR, and JCL undertook the analysis. All authors MNS, EFA, JCL, NR, JW, JFL DEW, RH, KS, GD, PH, and JB critically reviewed and commented on the results. MNS wrote the first draft of the paper which was reviewed by all. All authors approved the final version.

Authors’ information

Dr. Mitchell Sarkies is a Senior Lecturer and NHMRC Emerging Leadership Fellow at the Sydney School of Health Sciences. Dr Sarkies leads the Innovation and Methods Stream for the Sydney Health Partners Implementation Science Academy and is a member of the Academic Implementation Science Network, which supports advancements in the science of implementation in health.

Corresponding author

Correspondence to Mitchell Sarkies.

Ethics declarations

Ethics approval and consent to participate

Ethical approval was provided by Macquarie University (Project ID 23816) and Hunter New England Human Research Ethics Committees (Project ID 2020/ETH02186). Informed consent was recorded verbally at the commencement of each interview.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Audit and feedback program logic for CHF and COPD initiatives.

Additional file 2.

Audit and feedback program logic for Inpatient Management of Diabetes Mellitus Leading Better Value Care initiative.

Additional file 3.

RAMESES II reporting standards for realist evaluations.

Additional file 4.

Study protocol adaptations.

Additional file 5.

Initial audit and feedback program theory for CHF, COPD, and diabetes initiatives.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sarkies, M., Francis-Auton, E., Long, J. et al. Audit and feedback to reduce unwarranted clinical variation at scale: a realist study of implementation strategy mechanisms. Implementation Sci 18, 71 (2023). https://doi.org/10.1186/s13012-023-01324-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13012-023-01324-w

Keywords

Implementation Science

ISSN: 1748-5908