Table 1 Evaluation tool criteria items organized by Brehaut and colleagues’ 15 suggestions for improved audit and feedback interventions [4]
15 Suggestions | Criteria Items | Response Scale |
|---|---|---|
Nature of the desired action | ||
1. Recommend actions that are consistent with established goals and priorities | 1.1 Is there any indication that the recipients set an internal goal for themselves (i.e. a specific, numerical target/threshold)? | Yes/ No/ Not Reported/ Unclear |
1.2 Was feedback presented in the context of an external, explicit priority? | Yes/ No/ Not Reported/ Unclear | |
1.3 If yes, at what level was the priority set? | National/Federal, Provincial/State, Municipal, Institutional, Departmental, Individual-healthcare providers, Individual- researchers, Other (Please specify), Not Reported, Unclear, N/A | |
1.4 Does the feedback directly address one or more of the goals or priorities? | Yes/ No/ Not Reported/ Unclear/ N/A | |
2. Recommend actions that can improve and are under the recipient’s control | 2.1 Was feedback on performance provided to allow current performance to be compared against previous performance? | Yes/ No/ Not Reported/ Unclear |
2.2 Does the feedback (or the description of the feedback) describe or show a discrepancy between recipient performance and the goal/ benchmark/ target/comparator? | Yes/ No/ Not Reported/ Unclear | |
2.3 Is it reasonable that the feedback recipient can be responsible for the change in behavior? | Yes/ No/ Not Reported/ Unclear | |
2.4 Does the feedback provide data on behaviors, outcomes, or both? | For each, answer: Yes/ No/ Not Reported/ Unclear | |
3. Recommend specific actions | 3.1 Did the feedback intervention incorporate suggested corrective actions to support plans for problem solving? For example: action plans, coping strategies, a menu of options, etc. | Yes/ No/ Not Reported/ Unclear |
Nature of the data available for feedback | ||
4. Provide multiple instances of feedback | 4.1 Was feedback (for a given behavior) provided more than once? | Yes/ No/ Not Reported/ Unclear/ Other |
4.2 Did recipients continue to receive feedback on their performance after the study was completed? | Yes/ No/ Not Reported/ Unclear | |
5. Provide feedback as soon as possible and at a frequency informed by the number of new patient cases | 5.1 What is the average age of the data (i.e. interval between the clinical encounter and delivery of the feedback)? | Describe (i.e. Days, Weeks, Months, Years, Not Reported, Unclear) |
5.2 If feedback was provided more than once, what was the time interval between the receipt of feedback reports? | Describe (i.e. Hours, Days, Weeks, Months, Years, Variable, Not Reported, Unclear, N/A) | |
5.3 Do the authors of the study provide a justification for the interval between feedback reports? | Yes/ No/ Unclear/ N/A | |
5.4 Was the justification for the interval between feedback reports related to the number of patient cases? | Yes/ No/ Unclear/ N/A | |
6. Provide individual rather than general data | 6.1 Was feedback given about the individual’s own performance? | Yes/ No/ Not Reported/ Unclear |
6.2 Was feedback about the performance of a group of which the recipient is a member? | Yes/ No/ Not Reported/ Unclear | |
6.3 What is the level of the group? | Describe: Unit, Department, Practice, Hospital, Region, Province, Other (please specify), Not Reported, N/A | |
6.4 Did feedback include patient- level data for the recipient’s own patients? | Yes/ No/ Not Reported/ Unclear | |
6.5 Did feedback include aggregated patient data involving recipient’s own patients? | Yes/ No/ Not Reported/ Unclear | |
6.6 Do the authors give a justification for the specificity of the feedback (or a reason for the level of data presented)? | Yes/ No/ Unclear | |
7. Choose comparators that reinforce desired behavior change | 7.1 Did the feedback provide any comparators? | Yes/ No/ Not Reported/ Unclear |
7.2 How many comparators were provided? | 1, 2, 3, More than 3, None, Not Reported, Unclear, Other (describe as needed) | |
7.3 Describe all that apply (e.g. Own previous performance, Other’s performance, Benchmark/ Standardized Guideline/Target, Other, Not Reported, Unclear) | Own previous performance, Other's performance, Benchmark/ Standardized Guideline/Target, Other, Not Reported, Unclear | |
7.4 Does feedback include one or more aspirational comparators (as opposed to average performance comparators)? | Yes/ No/ Not Reported/ Unclear/ Other | |
7.5 Do the authors provide a justification for which comparators were used? | Yes/ No/ Unclear/ N/A | |
Feedback display | ||
8. Closely link the visual display and summary message | 8.1 Are the visual display and the summary message presented in visual proximity of each other? | Yes/ No/ Not Reported/ Unclear/ N/A |
9. Provide feedback in more than one way | 9.1 Was feedback provided in more than one way? | Yes/ No/ Not Reported/ Unclear |
9.2 Does the feedback intervention include: 1) Verbal interaction; 2) Text; 3) Numerical information; 4) Graphs or tables; 5) A summary message; 6) Other important elements (please specify)? | Provide an answer for each item: Yes/ No/ Not Reported/ Unclear | |
10. Minimize extraneous cognitive load for feedback recipients | 10.1 Was the feedback intervention pilot-tested? | Yes (with target population)/ Yes (with non-target population)/ No/ Not Reported, Unclear |
10.2 How long is the feedback report? | Describe | |
10.3 How many behaviors does the feedback address? | 1, 2, 3, 4, 5–9, more than 9, Not Reported, Unclear, Other (describe as needed), N/A | |
10.4 How many clinical variables were fed back to the recipients? | Describe/ Not Reported/ Unclear | |
10.5 How many graphs or tables are used? | Describe/ Not Reported/ Unclear | |
10.6 Did the feedback include any graphical elements that lend themselves to misinterpretation? (Ex. Pie charts, 3D graphs, etc.) | Yes/ No/ Not Reported/ Unclear/ N/A | |
Delivering the feedback intervention | ||
11. Address barriers to feedback use | 11.1 Were potential drivers and barriers to recipients engaging with the feedback component of the intervention assessed? | Yes/ No/ Not Reported/ Unclear |
11.2 Was the assessment informed by theory? | Yes/ No/ Not Reported/ Unclear/ N/A | |
11.3 Was there an assessment of whether the recipients engaged with the feedback? | Yes/ No/ Not Reported/ Unclear/ Other | |
11.4 Was the assessment informed by theory? | Yes/ No/ Not Reported/ Unclear/ N/A | |
12. Provide short, actionable messages followed by optional detail | 12.1 Are the summary messages actionable (or described as actionable)? | Yes/ No/ Not Reported/ Unclear/ N/A |
12.2 Is there additional, more detailed feedback provided alongside the summary message? | Yes/ No/ Not Reported/ Unclear | |
13. Address credibility of the information | 13.1 Does the feedback (or the description of the feedback) indicate who is providing the feedback data? | Yes/ No/ Not Reported/ Unclear |
13.2 Does the feedback intervention indicate the source of comparators? | Yes/ No/ Not Reported/ Unclear/ Other | |
13.3 Was the feedback intervention delivered by a supervisor or close colleague? | Yes/ No/ Not Reported/ Unclear | |
13.4 Was the feedback intervention supported by a relevant professional organization? | Yes/ No/ Not Reported/ Unclear | |
14. Prevent defensive reactions to feedback | 14.1 Did the feedback intervention include reassurance that the intervention would not trigger punitive measures? | Yes/ No/ Not Reported/ Unclear/ Other |
15. Construct feedback through social interaction | 15.1 Did development of the feedback intervention involve members of the target group? | Yes/ No/ Not Reported/ Unclear |
15.2 Was the feedback explicitly designed to be received and discussed in a social context? | Yes/ No/ Not Reported/ Unclear/ Other | |
15.3 If feedback was provided more than once, how often was feedback received and discussed in a social context? | Every time feedback was provided, Only once, Variably, Not Reported, Unclear, Other (please specify), N/A | |
15.4 If feedback was received and discussed in a social context, was the feedback discussion facilitated by a facilitator? | Yes/ No/ Not Reported/ Unclear/ N/A | |
15.5 Did the feedback intervention involve engaging in self- assessment around target behaviors prior to receiving feedback? | Yes/ No/ Not Reported/ Unclear | |
15.6 Did the investigators actively seek feedback from the recipients on the feedback? | Yes/ No/ Not Reported/ Unclear | |