Table 6 Example quotes used to refute, refine, or confirm the audit and feedback program theory

1. Ownership and buy-in

I01 Implementer, CHF and diabetes, rural hospital:

“It was good to have the ACI team come but also had some local knowledge…I knew the local processes and I knew also, when to chime in and say, oh by the way there's no GP in this town… So, having that local knowledge, definitely did help. Because otherwise I would have been blind to, where to that where to find that information.”

I02 Implementer, ACI:

“… the rest were really accepting of the data that we presented. Because there were some of them were actually involved in the development of the audits… so if we went out to the sites where those clinicians were working, we had quite good buy in there.”

103 Implementer, Ministry of Health:

“When we went back to the executive group and said here's the data. This is what we're seeing…And I think…that was a bit of a wakeup call in terms of…the compliance that they needed to start working towards. Even though it was a paper chart, they needed to start completing it. So, I think then they had a…diabetes CNC, who then really drove a strong piece of work around compliance on it and educating the nursing staff on it. She did a bit of a improvement project just on the chart there alone. And then conducted her own audits and spot-checked files, whenever she walked into someone's room and started doing stuff like that.”

2. Sensemaking of information feedback

I04 Implementer, ACI:

“[rural hospital] were amazing. We did a lot of sites in [rural hospital]…they were always really keen to hear and reflect and to take it on board and even if they didn't necessarily agree with it…maybe we need to look into that. And they…had that real…learning culture that they wanted to get something out of the audit process.”

I05 Implementer, ACI:

“…that sort of turned the tables a little bit in that we could go there. Knowing often what the local context was, if not knowing some of the clinicians really well, and saying hey you guys didn't do so well with this, like say for instance, chest X ray. However, we do know that you're a site that doesn't have X ray coverage from 8 pm at night to late in the morning. So it's quite understandable that you've got a delay that might not be something that you can do something about, nor is that something we should jump up and down about given that you're quite good with the rest of your clinical diagnostic so…we can work with the clinicians to identify okay like it might be identified as unwarranted clinical variation, but is it an excellence based approach versus a perfectionist based approach, I guess, is kind of where that turned the table a little bit…”

I01 Implementer, CHF and diabetes, rural hospital:

“Yeah, that was really beneficial, and we use those numbers in the brief, and also the money it was going to save the hospital and patient time transport…so that was really beneficial…”

3. Motivation and social influence

I07 Clinician, rural hospital, COPD:

“Because Ministry of Health is telling the other side…this is our expectations. This is the benchmark we've set this quarter; this is the benchmark, we want you to make, they audited everyone the same, which I think was really important…but at least we're all getting the same audit done to us. So, that’s handy because it means that everyone's been treated the same…”

I04 Implementer, ACI:

“Having outcomes is really important and that's one thing I've really learnt. Being on the other side of the bed, is that how important data is and how it can really give some good evidence and some good weights to change. I think there's also a real, depending on the personality, but the scrutiny by peers and what peers are doing, also gives the potential rise to change in practice and improving can improve in practices.”

I07 Clinician, rural hospital, COPD:

“So that was a good tool for change like you had managers…sort of executives that couldn’t avoid…the fact that it was on the audit which is why it came back.”

4. Responsibility and accountability

I02 Implementer, ACI:

“…part of our role was to ensure that those correct stakeholders including managers and clinicians were attending…We looked at availability and ensured that they were all in the same room together…for example…one of the rural sites, I think they had two or three clinicians, but they were the right clinicians in the room. So that was part of our planning to ensure they were there.”

I02 Implementer, ACI:

“they did have accountability and were taking responsibility for the data, I think, where the gaps were was, they're doing this on top of their work like any other project. We're dedicated to it so, we're doing our best to keep them engaged, and we did a bit of the work but at the end of the day, they do own the project.”

I05 Implementer, ACI:

“Yes, some people did use it as a bit of an enabler to support a project or something that they wanted to work on like [rural hospital] has reported to us that they've used LBVC to deliver on some improvements in the service they provide because they provide services to CHF and COPD.”

5. Rationalisation of the status quo

I06 Clinician, COPD, metro hospital:

“it was more about, from my perspective, not having the confidence that we would make change from the audit rather than…not having the confidence in the audit process.”

I07 Clinician, COPD, rural hospital:

“…there was a little bit, from the doctors feeling that they questioned how accurate it was… So, I think initially then trusting results is a bit difficult or feeling a bit confused about results. I think it put everyone a little bit on a back foot. When we initially got it, we didn't know this was coming, we didn't understand what you were doing, and they think that they’ve just handed us results and you want us to do something about it. That's what everyone else on a bit of a back foot for us.”

I08 Clinician, rural hospital:

“I appreciate it wasn't gaps around the clinicians’ performance, often I was looking around system gaps in terms of the organizational resourcing. And those things are typically beyond the remit of the consultants to address those. And so therefore, I think that's where that level of cynicism grew. I felt that those the system gaps were being pointed at them.”

6. Perceptions of unfairness and concerns about data integrity

I01 Implementer, CHF and diabetes, rural hospital:

“They don't know who their exec. department are now, no one's a real, or who are you talking about. So there seems to be this communication level, and you got your talk about some of our senior clinicians on the floor didn’t know who you're talking about…then there was that not distrust, but that disgruntled…’oh well they don’t that care about us anyway because we never see them’, you know, so that's where I think there's a disconnect. They want us to do this, yet we don't know who you are. And you'd haven’t got a relationship with us. So that relationship was missing…from the managers’ perspective they said well, we don't want to be micromanagers, we want to trust that a clinician will do the right thing and so it's a fine line between how much do you micromanage. And how much do you trust that they're just doing it.”

I05 Implementer, ACI:

“Clinicians know exactly what, what it is but the system sometimes provides huge barriers to delivering that. So the focus was never about either can we change a system or how can we work around it. It was always about this the care in doing it. So it's kind of a bit of a setup to fail scenario.”

I04 Implementer, ACI:

“So it might be that they weren't documented as being offered smoking cessation support, because the staff knew that they'd done that 10 times before and it wasn't it wasn't going to happen again. So therefore, it wasn't documented, even though they were in the week prior…And the problem with a medical record is it cuts off when the patient is discharged. It doesn't necessarily capture the fact that a phone call was made a week later, or referrals and pulmonary rehab was made a month later, our specific questions were related to…a referral to pulmonary rehab made at that admission. And if it wasn't documented, it didn't happen. But in reality, a lot of times those patients were referred to pulmonary rehab. It just happened a week or two later. Is that acceptable practice? Potentially not, but that came down to the workings of the way that they conducted their service.”

7. Improvement plans that are not followed

I05 Implementer, ACI:

“At the time of the audit…there were a lot of variables and the process was…there would be analysis, and then a report would be sent to the site and feedback would be obtained. And then that data was presented to the broader team of clinicians, some of who may or may not have been aware of the auditing process, and then straight away they were asked to prioritize what activities they were going to do differently… But just asking people straight away there in the room to make a decision on what they're going to prioritize. I think was quite difficult. I don't think was particularly useful.”

I07 Clinician, COPD, rural hospital:

“Unfortunately, I think the biggest thing would be having like an audit team, someone that could sit there and say no, that is a player that, when I'm looking at that person is supposed to be with this team. That's the only way that I could see that maybe for hospitals such as mine, make that a little less impacting. Like, they tried to, they've got someone local, but that person probably wasn't the right person.”

I04 Implementer, ACI:

“The feeling was our [state government] election [was] soon and LBVC will just be replaced by something else I was actually surprised that it wasn't. But and you know we had a change in Minister as well so there was also that that thinking behind it. You know, I've only been on the management side of health since, 2017, but for those that have been in the system a lot longer. They’re used to…that sort of coming and going out of flavour so if there was that that sense that all the funding is only for a year. There's an election and it's probably going to be something else.”

8. Perceptions of threats to professional autonomy

I06 Clinician, COPD, metro hospital:

“I think that it was to do more with the hard matrix of it. It not being understood in our area, or our situation, our environment, our…understanding the medical team’s reasons for why they variate, why there is the variation. I don’t think that's understood why there is clinical variation. We know it exists, but why does it exist I just don't think that that is understood as much…You know, sometimes you've got to make it work for the patient…if your shoelace breaks you might use a string, but maybe that's not understood.”

I08 Clinician, rural hospital:

“There wasn't agreement with it, I suppose. Insulted, I mean, for me, insulted is a stronger term. Just thinking about the various feedbacks that I brought into. It may be but it also instantly led to that frustration because this audit that was done removed from them about the care that they provide, with no understanding. So, I suppose I’d lean more towards the frustration and then that automatically prep the barriers to saying this isn’t going to be relevant for us because they're not measuring us on our on the care that we provide.”

I07 Clinician, rural hospital, COPD:

“I would say a lot of that resistance, came from feeling like we were…caught unawares, by what was going on and not understanding to achieve. You felt very put on the spot. We were sat in a room, we sat through a PowerPoint presentation, they go okay here’s how you’re doing. And then, for a period of time feeling like I had no, no feedback or follow up that that I think definitely put my physician that I work with on the backfoot.”