Peer chart audits: A tool to meet Accreditation Council on Graduate Medical Education (ACGME) competency in practice-based learning and improvement
Implementation Science volume 2, Article number: 24 (2007)
The Accreditation Council on Graduate Medical Education (ACGME) supports chart audit as a method to track competency in Practice-Based Learning and Improvement. We examined whether peer chart audits performed by internal medicine residents were associated with improved documentation of foot care in patients with diabetes mellitus.
A retrospective electronic chart review was performed on 347 patients with diabetes mellitus cared for by internal medicine residents in a university-based continuity clinic from May 2003 to September 2004. Residents abstracted information pertaining to documentation of foot examinations (neurological, vascular, and skin) from the charts of patients followed by their physician peers. No formal feedback or education was provided.
Significant improvement in the documentation of foot exams was observed over the course of the study. The percentage of patients receiving neurological, vascular, and skin exams increased by 20% (from 13% to 33%) (p = 0.001), 26% (from 45% to 71%) (p < 0.001), and 18% (51%–72%) (p = 0.005), respectively. Similarly, the proportion of patients receiving a well-documented exam which includes all three components – neurological, vascular and skin foot exam – increased over time (6% to 24%, p < 0.001).
Peer chart audits performed by residents in the absence of formal feedback were associated with improved documentation of the foot exam in patients with diabetes mellitus. Although this study suggests that peer chart audits may be an effective tool to improve practice-based learning and documentation of foot care in diabetic patients, evaluating the actual performance of clinical care was beyond the scope of this study and would be better addressed by a randomized controlled trial.
The Accreditation Council on Graduate Medical Education (ACGME) mandates Practice-Based Learning and Improvement as a core competency area for residents in training. To fulfill this competency, residents are expected to : 1) analyze practice experience and perform Practice-Based Learning and Improvement activities using a systematic methodology, 2) locate appraise and assimilate evidence from scientific studies related to their patients' health problems, 3) obtain and use information about their own population of patients and the larger population from which their patients are drawn, 4) apply knowledge of study designs and statistical methods to appraisal of clinical studies and other information on diagnostics and 5) use information technology to manage information and access on-line information . Continuous Quality Improvement, also called Performance Improvement (PI) projects help to meet this requirement. The improvement activities must relate to the core competencies, involve residents and faculty and produce measurable improvements in patient care or residency education .
A chart audit is one quality performance measurement technique which can be used to evaluate residents' competence in Practice-Based Learning and Improvement [3, 4]. By itself, chart audit merely measures improvement in performance not competence. A recent pilot study found that self audits led to meaningful physician behavior changes , while a Cochrane Collaboration systematic review documented the effectiveness of trained abstractors performing clinical audit with feedback to monitor and improve physician performance [6, 7]. While improvements might be due to increased competence in the specific activity of practice-based learning, increased performance could be due to other forms of learning and behaviors as well.
To date there are still few studies evaluating the effectiveness of peer chart audits performed by residents: most studies conducted to date have evaluated self-audits or external audits, and most combined chart audit with formal feedback or an educational intervention [8–11]. Audit-feedback generally involves external audit and relies heavily on the feedback activity for its effectiveness in changing clinical practice. Therefore, the audit-and-feedback strategy fails to recognize that the audit activity itself may have educational value. Little is known about the effectiveness and feasibility of chart audits to meet the ACGME requirements. In addition, the peer chart process itself, in the absence of a formal educational intervention or feedback, has not been studied as a quality improvement technique. We hypothesized that the peer chart audit process itself, without formal educational intervention or feedback, would be associated with improved documentation of foot care.
The study took place in the three general internal medicine primary care continuity clinics at the Brody School of Medicine at East Carolina University. The Institutional Review Board required written informed consent be obtained from the residents. All patient identifiers were removed at the completion of each audit.
Adult patients with diabetes mellitus were identified by searching the electronic medical records (Logician®, Medicalogic, GE Medical Systems Information Technologies, Hillsboro, Oregon, USA). Only patients with ICD-9 codes 250.XX in their problem list and receiving continuity care by residents in the categorical and combined internal medicine programs were included.
The chart audits occurred for one-week intervals during continuity clinic conference time. All residents who were present in the clinic during that week participated. Personnel in Medical Records selected the charts of patients who were followed by the residents. The charts were subsequently assigned to the residents. Residents could not audit charts of their own patients, and patient lists were reviewed manually to ascertain that no patient's chart was used more than once per audit.
Audit one was performed in June 2003. Residents were allowed to abstract information dating back for one year prior to June 2003. Audit two occurred in September 2003 for patients seen between July 2003 and September 2003. Audit three was performed in May 2004 for patients seen between October 2003 and May 2004. For audits two and three, the residents were assigned specific visit dates that would encompass visits made after the previous audit to better determine the impact of the audit itself on documenation of care. Charts for repeat audits were selected based on whether the patient had a visit within the time periods above, with no exclusion or inclusion based on whether they had been audited before. No formal feedback was provided to residents between audits. Residents were not informed of the audit until the time of the audit. General Internal Medicine faculty members were aware of the results of the audits, but did not provide formal feedback to residents.
We developed the audit form based on the Diabetes Quality Improvement Project (DQIP) guidelines  (see description below) and discussions among general medicine faculty. The form was reviewed and revised for clarity based on consensus, but was not formally piloted. Using the electronic medical record, each resident used the form to review two to five charts during each audit phase. All visits were reviewed to identify the following three domains: (1) history and review of systems, including any mention of the foot or foot problems; (2) foot examination, including performance of the exam and presence of abnormalities; and (3) interventions. An intervention was considered to be present when patients received recommendations for foot care (e.g., prescription for shoes) or were referred for podiatric care or vascular evaluation. The analyses reported here assessed improvements in resident performance related to documentation of the foot examination.
Documentation of the foot exam is described in the Diabetes Quality Improvement Project (DQIP) guidelines . The quality of care standard defined by the DQIP is the percentage of patients receiving a well-documented foot exam. The DQIP foot exam items have been previously validated as predictors for ulceration. The components of a well-documented foot exam include neurological (sensate or vibratory testing with the Semmes-Weinstein monofilament or fork test), vascular (pedal pulses), and skin findings .
Standard descriptive statistics were used and data were analyzed using SPSS® (Chicago, IL). Audits were compared with the chi-square test for trend. The Mantel-Hantzel odds ratio was calculated to quantify the likelihood of interventions between patients with and without abnormalities. The unit of analysis was the patient.
Residents audited 347 electronic records. Patients had an average of 3.8 (SD 2.5) visits per year during the period of the chart reviews. We observed no increase in documentation of aspects of the history or review of systems related to the feet between audit one (range, 14% to 51%), audit two (range, 15% to 45%) and audit three (range, 11% to 59%) (all p > 0.05). Over time, residents showed improved documentation of the foot exam. Documentation of the neurological exam by the monofilament or fork test (p = 0.001), the vascular exam by assessment of pedal pulses (p < 0.001), and the skin exam (p = 0.005) improved (Figure 1). Documentation of all three exams – neurological, vascular, skin – increased from 6% to 24% (p < 0.001) (Figure 1).
Among audits, we observed no differences in the documented prevalence of foot abnormalities overall, 38% (all p > 0.11), or the frequency of interventions overall, 25% (all p > 0.10). (Table 1). During all three audits, patients with any foot abnormalities received more interventions for foot care as compared to patients without foot abnormalities, [audit one (46% vs. 15%, P = 0.001), audit two (37% vs. 20%, P = 0.02), and audit three (39% vs. 12%, P = 0.002)], data not shown. The odds ratio for any intervention was 3.47 (95% CI 2.09 to 5.75, P < 0.001) for patients with foot abnormalities, as compared to patients without foot abnormalities.
This study addressed whether peer chart audit performed by residents, without formal feedback, is associated with improved standards of care for the foot exam in patients with diabetes mellitus. Follow-up chart audit results were associated with a fourfold increase in the number of well-documented foot exams. Although the magnitude of improvement in documentation is statistically significant, the current study was not designed to address what care was actually delivered pre- and post-intervention.
The positive educational impact of the peer chart audits is highlighted by the absence of an extensive instructional component about diabetic foot care. We do not feel that a one-time, half-hour discussion regarding foot care would have had much impact, as past studies with even more extensive physician education have been mixed in terms of demonstrating improved outcomes .
The impact of peer involvement may be an important factor contributing to our findings. Studies show that peer coaching, for example, contributes to physicians' professional development of both the learner and the mentor by encouraging reflection time and learning . We suspect that faculty and residents informally engaged in discussions during the process and learned that the foot exam is an important and reliable indicator of care.
We did not see any change in the history or review of systems; other studies have found these items inconsistently asked and documented . This finding may be further explained by the fact that the foot examination is often emphasized as the measure of quality.
Although it is well known that routine visits for patients with diabetes should include advice that they examine their feet daily and obtain an annual foot exam by their provider, studies found that the single most important item of the exam – the neurological exam- was performed in only one third of patients [17, 18]. Our findings are consistent with other studies demonstrating less than optimal foot exams and poor adherence to diabetes guidelines [19, 20]. For example, in a study by Greenfield et al., the prevalence of foot checks was 61.8% by general internists and 49.6% by endocrinologists .
Overall, the data support chart audits as a useful tool for teaching Practice-Based Learning and Improvement. Another study showed that a quality improvement curriculum can produce creative projects that address the core competencies . We also incorporated additional ACGME core competencies including effective patient care, application of medical knowledge to patient care and systems-based practice. In our study we used an accepted standard of care to assess compliance and measure improvement of the foot exam. During the process we learned that implementation was feasible and did not require professional chart abstractors. However, it did require additional personnel, careful planning, and expertise in data management. These additional resources will have financial implications for residency program directors and department heads.
Our study has some limitations. Improvements in foot exam documentation might not reflect changes in practice; we were not able to directly measure practices. Observed improvements might be due to factors other than the peer chart audit activity. For example, the observed changes may have been due to the Hawthorne effect, in which subjects of a study modify their behavior because they are participating in a study . Also, because a variety of other conferences and teaching activities occur elsewhere in our curriculum, it is difficult to control for learning that may have taken place in other forums. However, to our knowledge, no other structured program was implemented at the same time as our chart review. Evidence to more definitively link the peer chart audit activity to observed changes in documentation (and clinical practice) will require a stronger evaluation design such as a randomized controlled trial. Follow-up studies might include a control group of residents, informed of the measurement process but not actually participating in the chart audit process, in order to link the audits to observed improvements.
A peer chart audit performed by residents, in the absence of formal educational interventions or feedback, was associated with improved documentation of the foot exam in patients with diabetes mellitus. Our conclusions are limited by our study design, and the results observed might be due to other factors rather than the repeated peer reviews. Yet this study demonstrates the feasibility of the peer chart audit method and suggests that an educational tool allowing residents to review the charts of their peers may serve as a reminder of standards of care, and may heighten awareness of the need for quality improvement efforts. The peer chart audit method supports the ACGME recommendations of performance improvement processes by internal medicine residency programs and warrants further evaluation and refinement to support expanded use.
Accreditation Council on Graduate Medical Education. Last Accessed April 2006, [http://www.acgme.org]
Djuricich AM, Ciccarelli M, Swigonski NL: A continuous quality improvement curriculum for residents: addressing core competency, improving systems. Acad Med. 2004, 79: S65-7. 10.1097/00001888-200410001-00020.
Paukert JL, Chumley-Jones HS, Littlefield JH: Do peer chart audits improve residents' performance in providing preventive care?. Acad Med. 2003, 78: S39-41. 10.1097/00001888-200310001-00013.
Coleman MT, Nasraty S, Ostapchuk M, Wheeler S, Looney S, Rhodes S: Introducing Practice-Based Learning and Improvement ACGME core competencies into a family medicine residency curriculum. Jt Comm J Qual Saf. 2003, 29: 238-247.
Holmboe ES, Meehan TP, Lynn L, Doyle P, Sherwin T, Duffy FD: Promoting physicians' self-assessment and quality improvement: The ABIM Diabetes Practice Improvement Module. The Journal of Continuing Education in the Health Professions. 2006, 26: 109-118. 10.1002/chp.59.
Foy R, Eccles MP, Jamtvedt G, Young J, Grimshaw JM, Baker R: What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review. BMC Health Services Research. 2005, 5: 50-10.1186/1472-6963-5-50.
Jamtvedt G, Young JM, Kristoffersen DT, Thomson O'Brien MA, Oxman AD: Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2006, 000259-2
Holmboe E, Scranton R, Sumption K, Hawkins R: Effect of medical record audit and feedback on residents' compliance with preventive health care guidelines. Acad Med. 1998, 73: 901-903. 10.1097/00001888-199808000-00016.
Fihn SD, McDonell MB, Diehr P, Anderson SM, Bradley KM, Au DH, Spertus JA, Burman M, Reiber GE, Kiefe CI, Cody M, Sanders KM, Whooley MA, Rosenfeld K, Baczek LA, Sauvigne A: Effects of sustained audit/feedback on self-reported health status of primary care patients. Am J Med. 2004, 116: 241-248. 10.1016/j.amjmed.2003.10.026.
Kern DE, Harris WL, Boekeloo BO, Barker LR, Hogeland P: Use of an outpatient medical record audit to achieve educational objectives: changes in residents' performances over six years. J Gen Intern Med. 1990, 5: 218-224. 10.1007/BF02600538.
Kiefe CI, Allison JJ, Williams OD, Person SD, Weaver MT, Weissman NW: Improving quality improvement using achievable benchmarks for physician feedback: a randomized controlled trial. JAMA. 2001, 285: 2871-2879. 10.1001/jama.285.22.2871.
Fleming BB, Greenfield S, Engelgau MM, Pogach LM, Clauser SB, Parrott MA: The Diabetes Quality Improvement Project: moving science into health policy to gain an edge on the diabetes epidemic. Diabetes Care. 2001, 24: 1815-1820. 10.2337/diacare.24.10.1815.
Singh N, Armstrong DG, Lipsky BA: Preventing foot ulcers in patients with diabetes. JAMA. 2005, 293: 217-228. 10.1001/jama.293.2.217.
Renders CM, Valk GD, Griffinn S, Wagner EH, van Eijk JThM, Assendelft WJJ: Interventions to improve the management of diabetes mellitus in primary care, outpatient and community settings. Diabetes Care. 2001, 24 (10): 1821-1833. 10.2337/diacare.24.10.1821. Art. No.:CD001481. DOI: 10.1002/14651858.CD001481
Sekerka LE, Chao J: Peer coaching as a technique tofoster professional development in clinical ambulatory settings. Journal of Continuing Education in the Health Professions. 2005, 23: 30-37. 10.1002/chp.1340230106.
Sussman KE, Reiber G, Albert SF: The diabetic footproblem – a failed system of health care?. Diabetes Res Clin Pract. 1992, 17: 1-8. 10.1016/0168-8227(92)90037-R.
American Diabetes Association: Standards of medical care for patients with diabetes mellitus. Diabetes Care. 2003, 26: S33-50. 10.2337/diacare.26.7.2194.
American Diabetes Association: Standards of medical care for patients with diabetes mellitus. Diabetes Care. 2002, 25: 213-229. 10.2337/diacare.25.4.742.
Saaddine JB, Engelgau MM, Beckles GL, Gregg EW, Thompson TJ, Narayan KM: A diabetes report card for the United States: quality of care in the 1990s. Ann Intern Med. 2002, 136: 565-574.
De Berardis G, Pellegrini F, Franciosi M, Belfiglio M, Di NardoB, Greenfield S, Kaplan SH, Rossi MC, Sacco M, Tognoni G, Valentini M, Nicolucci A: Quality of care and outcomes in type 2 diabetic patients: a comparison between general practice and diabetes clinics. Diabetes Care. 2004, 27: 398-406. 10.2337/diacare.27.2.398.
Greenfield S, Rogers W, Mangotich M, Carney MF, Tarlov AR: Outcomes of patients with hypertension and non-insulindependent diabetes mellitus treated by different systems and specialties. Results from the medical outcomes study. JAMA. 1995, 274: 1436-1444. 10.1001/jama.274.18.1436.
Carraccio C, Englander R: Evaluating competence using a portfolio: a literature review and web-based application to the ACGME competencies. Teach Learn Med. 2004, 16: 381-387. 10.1207/s15328015tlm1604_13.
Renders CM, Valk GD, Griffin S, Wagner EH, van Eijk JThM, Assendelft WJJ: Interventions to improve the management of diabetes mellitus in primary care, outpatient and community settings. Diabetes Care. 2001, 24: 1821-1833. 10.2337/diacare.24.10.1821.
We thank Ms. Christine Ransdell for assistance during data collection and Dr. Bruce Johnson for reviewing the manuscript. This study was presented in part at the Southern Society of General Internal Medicine 2004 meeting in New Orleans, LA, in February, 2004, and at the Association of Program Directors, Spring Meeting in 2004.
The author(s) declare that they have no competing interests.
All authors contributed equally to the work. LS conceived the study, participated in the design and coordination and helped draft the manuscript. SK conceived the study and participated in the design and coordination. SP conceived the study and participated in the design and coordination. GT participated in the design and coordination and helped to perform the statistical analysis. CE participated in the design and coordination of the study, helped to draft the manuscript and performed the statistical analysis. All authors read and approved the final manuscript.
Lisa J Staton, Suzanne M Kraemer, Sangnya Patel, Gregg M Talente and Carlos A Estrada contributed equally to this work.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
Staton, L.J., Kraemer, S.M., Patel, S. et al. Peer chart audits: A tool to meet Accreditation Council on Graduate Medical Education (ACGME) competency in practice-based learning and improvement. Implementation Sci 2, 24 (2007). https://doi.org/10.1186/1748-5908-2-24