Sources of non-compliance with clinical practice guidelines in trauma triage: a decision science study
© Mohan et al.; licensee BioMed Central Ltd. 2012
Received: 19 April 2012
Accepted: 22 October 2012
Published: 25 October 2012
United States trauma system guidelines specify when to triage patients to specialty centers. Nonetheless, many eligible patients are not transferred as per guidelines. One possible reason is emergency physician decision-making. The objective of the study was to characterize sensory and decisional determinants of emergency physician trauma triage decision-making.
We conducted a decision science study using a signal detection theory-informed approach to analyze physician responses to a web-based survey of 30 clinical vignettes of trauma cases. We recruited a national convenience sample of emergency medicine physicians who worked at hospitals without level I/II trauma center certification. Using trauma triage guidelines as our reference standard, we estimated physicians’ perceptual sensitivity (ability to discriminate between patients who did and did not meet guidelines for transfer) and decisional threshold (tolerance for false positive or false negative decisions).
We recruited 280 physicians: 210 logged in to the website (response rate 74%) and 168 (80%) completed the survey. The regression coefficient on American College of Surgeons – Committee on Trauma (ACS-COT) guidelines for transfer (perceptual sensitivity) was 0.77 (p<0.01, 95% CI 0.68 – 0.87) indicating that the probability of transfer weakly increased as the ACS-COT guidelines would recommend transfer. The intercept (decision threshold) was 1.45 (p<0.01, 95% CI 1.27 – 1.63), indicating that participants had a conservative threshold for transfer, erring on the side of not transferring patients. There was significant between-physician variability in perceptual sensitivity and decisional thresholds. No physician demographic characteristics correlated with perceptual sensitivity, but men and physicians working at non-trauma centers without a trauma-center affiliation had higher decisional thresholds.
On a case vignette-based questionnaire, both sensory and decisional elements in emergency physicians’ cognitive processes contributed to the under-triage of trauma patients.
KeywordsPhysician decision making Clinical guidelines Compliance Trauma Triage Heuristics Perceptual sensitivity Signal detection theory
Trauma affects one out of five Americans, requiring the expenditure of $400 billion in direct medical costs each year. Regionalization—tiered levels of care that distribute the sickest patients to the highest-intensity hospitals (trauma centers)—reduces mortality and morbidity [1–6]. Well-established clinical practice guidelines in trauma specify when to triage patients to specialty trauma centers . Despite concerted efforts to address known barriers to compliance (e.g., lack of familiarity with the guidelines), between 30% to 70% of all patients who meet criteria for transfer to a trauma center remain at non-trauma centers (under-triage) [8–11]. At the physician level, existing quality improvement efforts have focused on improving knowledge, modifying attitudes, and removing structural and economic barriers to transfer [12, 13]. The extent to which cognitive aspects of physician decision-making contribute to under-triage is unknown.
Based on cognitive theory for discrimination tasks, inappropriate disposition decisions (transfer/not transfer) may result from perceptual sensitivity (the ability to discriminate between patients who do and do not meet clinical practice guidelines for transfer) and/or decisional thresholds (the tendency to err on the side of false positive or false negative decisions) . These two dimensions of decision making have different determinants. Perceptual sensitivity reflects both physicians’ knowledge of the clinical practice guidelines as well as intuitive judgments (heuristics) about which patients meet those guidelines. Decisional thresholds reflect variables such as attitudes towards the guidelines, incentives, and organizational norms.
Signal detection theory, a basic decision science method, allows the measurement of these two dimensions of decision making by parsing performance into sensory and decisional components . We used signal detection theory to analyze emergency physician responses to a case vignette-based questionnaire. The objective of this study was to assess whether decision making in trauma triage primarily reflected physicians’ perceptual sensitivity or their decisional thresholds. Correctly attributing the source of non-compliance with clinical practice guidelines would allow us to design more effective quality improvement interventions.
Triage of trauma patients can occur either in the field or after an emergency physician has evaluated the patient in the Emergency Department (ED) of a non-trauma center. The American College of Surgeons – Committee on Trauma (ACS-COT) publishes guidelines specifying injuries that warrant transfer to a trauma center. They recommend the transfer of all patients with moderate to severe injuries, defined as either an injury considered to be ‘life-threatening or critical’ or an Injury Severity Score >15. They recommend that emergency physicians make their transfer decision based on information obtained from a history, physical exam, and chest and pelvis x-ray. We conducted a web-based cross-sectional survey among a convenience sample of United States (US) physicians who worked in the EDs of non-trauma centers, and used responses to clinical vignettes, based on case histories, to quantify physicians’ perceptual sensitivity and decisional threshold for trauma triage.
Development of questionnaire
2x2 table used to categorize identification decisions
Meets reference standard
Does not meet the reference standard
Decision to transfer
True positive (Hit)
False positive (False alarm)
Decision not to transfer
False negative (Miss)
True negative (Correct reject)
Clinical vignette pre-testing
To ensure that our vignettes and procedures were clear and without leading information, we conducted one round of pretests using cognitive interviews with University of Pittsburgh fellows and faculty who were board-eligible or certified in EM (n = 9). After revising our materials, we conducted additional cognitive interviews with emergency physicians who practiced in non-trauma centers in western Pennsylvania (n = 17). As a result of this second round of testing, we shifted from forced-choice options to free-response text boxes to avoid the priming that fixed options suggested. We also reduced the set of vignettes to 30, after finding that some physicians could not stay fully engaged while evaluating all 50.
We recruited physicians at a national meeting of the American College of Emergency Physicians (ACEP) in Fall 2010, using a booth in the Exhibition Hall. We designed the study to capture a moderate effect size of 0.15 for the association between clinical practice guidelines and transfer decisions in a regression model with two independent variables (α = 0.05, 1-β=0.80) . Anticipating a 25% response rate, we recruited 280 physicians. Physicians eligible for participation had completed residency and cared for adult patients in the ED of either non-trauma centers or Level III/IV trauma centers in the United States. Those who completed the questionnaire received a $100 payment.
The University of Pittsburgh Institutional Review Board study reviewed and approved the study.
Participants completed the web-based protocol at their convenience. After logging in, they read a brief passage describing the task, completed a demographic survey, and evaluated the vignettes.
The demographic survey included questions about age, gender, race, educational background (board certification, Advanced Trauma Life Support (ATLS) certification, years since completing residency), and practice environment (hospital trauma designation, affiliation with a Level I/II trauma center, affiliation with an EM residency program, size of the community served).
The evaluation portion had five blocks of six randomly ordered case vignettes, with each block containing two cases with moderate-severe injury severity/blunt mechanism, one with moderate-severe injury/penetrating mechanism, two with minor injury/blunt mechanism, and one with minor injury/penetrating mechanism. Physicians used a free-response text box to answer the question, ‘What would you do to manage the patient?’ The instructions asked them to provide information about treatment, interventions, and disposition. They had two weeks to complete the survey.
Descriptive summary of responses
We coded each free-text response to identify management decisions related to: disposition, consults, imaging, medications, laboratory studies, procedures, and resuscitation efforts. To assess inter-rater reliability, two coders evaluated a random sample of 10% of the management decisions. We found very high agreement for all six types of management decisions (kappa 0.81–0.94) . We used logistic regression models, clustered by physicians, to estimate the predicted probability that physicians would make types of management decisions for minor and moderate to severe cases.
We then scored each response dichotomously as compliant or non-compliant with ACS-COT guidelines. For patients with moderate to severe injuries, we defined compliance as: transferring the patient to a trauma center without obtaining additional testing; transferring the patient after performing additional testing contingent on not delaying the transfer; or consulting trauma surgery (assuming that the surgeon would ensure the appropriate triage). For patients with minor injuries, we defined compliance as: not transferring the patient to a trauma center; transferring a patient to a trauma center contingent on identifying further injuries; or consulting trauma surgery.
We analyzed performance in two ways. First, we calculated adherence to standard ACS-COT triage benchmarks. The ACS-COT defines under-triage as the proportion of patients with moderate-severe injuries who are not transferred. It defines over-triage as the proportion of transferred patients who have minor injuries . We summarized rates of under- and over-triage for each physician.
where Y is the disposition decision and X is the reference standard . The fit of logistic models approximates signal detection models, but with a scale π/√3 times larger than the Gaussian distribution used in signal detection theory .
We fit a model for each physician, predicting the log-odds of his or her disposition decisions (dependent variable) using an intercept and a regression weight on the ACS-COT guidelines for transfer (independent variable). The model’s intercept predicts disposition decisions in the absence of any information, hence provides an estimate for the decisional threshold . The regression weight on the ACS-COT guideline variable represents perceptual sensitivity, showing the degree of reliance on guidelines (implicitly or explicitly) when making disposition decisions . We scaled our estimates by 1.8 to approximate the standard deviation units of signal detection values.
A perceptual sensitivity estimate greater than zero indicated a positive association between the guidelines and decisions to transfer patients. A negative decisional threshold suggested a tendency to transfer trauma patients (i.e., effectively preferring false positives to false negatives), and a positive value suggested a tendency not to transfer. As accuracy approached the limit (15 of 15 true positive or true negative cases transferred appropriately), the estimation procedure failed. We corrected for perfect scores by converting proportions of 0 and 1 to 1/2N and 1-1/2N (where N equaled the total number of trials) respectively .
Applying the ACS-COT benchmarks for triage resulted in the following relationships.
In subsequent analyses, we used a random effects logistic regression model to estimate the perceptual sensitivity and decisional threshold for the entire cohort, adding mechanism of injury (blunt versus penetrating) as an additional predictor of physician disposition decisions. We also used Student’s t-test and Spearman correlations as appropriate to estimate the influence of physician characteristics (age, gender, race, board certification, ATLS certification, years since completing residency), and practice environment (hospital trauma designation, affiliation with a Level I/II trauma center, affiliation with an EM residency program, size of the community served) on perceptual sensitivity and decisional thresholds, and tested these characteristics in a multivariable linear regression model.
Demographic characteristics of physicians
Age, mean (SD)
Caucasian or white
African American or black
Hispanic or Latino
Primary specialty, n(%)
Years since completing residency, mean (SD)
ATLS certified, n(%)
Years since ATLS certification, mean (SD)
Designation of hospital where physician works, n(%)
Non-trauma center (non-TC)
Affiliation of the hospital with a TC (if working at a non-TC), n(%)
Affiliation of the hospital with an EM residency program, n(%)
Size of community served by hospital, n(%)
250,000 – 1 million
200,000 – 249,999
2,500 – 199,999
Descriptive summary of responses
Predicted probability of management decisions made for patients with minor and moderate to severe injuries
Patients with minor injuries
Patients with moderate-severe injuries
The median under-triage rate among physicians was 80% (Inter-Quartile Range [IQR] 40–97%). In other words, physicians expeditiously transferred 20% of patients meeting the guidelines for transfer. The most common reason for under-triage was delaying transfer to obtain additional imaging. The median over-triage rate was 29% (IQR 0–47%). In other words, 71% of patients transferred met the reference standard. Sixteen participants (9%) did not transfer any patients, but consulted specialists at their own institutions for further management of all cases.
Including mechanism of injury in the analysis revealed that cases with penetrating (versus blunt) injuries were more likely to be transferred (OR 1.73, p<0.01, 95% CI 1.46–2.03), even though they were no more (or less) likely to meet ACS-COT guidelines in the sample vignettes.
Physician characteristics (e.g., gender, training) did not influence perceptual sensitivity. Men had higher decisional thresholds than women (p = 0.01). Physicians with board certification in Emergency Medicine compared to physicians with board certification in Internal Medicine or Family Practice (p = 0.03), and physicians working at non-trauma centers without an established affiliation with a trauma center compared to physicians working at non-trauma centers with an affiliation (p<0.01) had higher decisional thresholds. Two of these associations (gender and affiliation of their hospital with a trauma center) remained significant in the multivariable linear regression model.
In a vignette-based study with 168 emergency physicians, we used signal detection theory to quantify the perceptual sensitivity and decisional threshold of physicians making trauma triage decisions. We found that cognitive processes in physician decision-making may contribute to persistent rates of under-triage in trauma triage. When responding to the cases, many physicians in the study had low perceptual sensitivity, making triage decisions only weakly related (if at all) to the ACS-COT standard. Additionally, most had a high decisional threshold for transfer, systematically erring on the side of not transferring patients to regional trauma centers.
Many epidemiological studies have shown that patients with moderate to severe injuries are routinely under-triaged [8–11]. However, those studies have typically focused on patient-level determinants of variability in triage decisions. Chang et al. have described age as a determinant of triage decisions . Macias et al. posit that age, co-morbidities, and severity of injury influence the triage of patients with traumatic spinal cord injuries to trauma centers . We used signal detection theory, a basic decision science method, to determine how physician cognitive processes might influence rates of under-triage. Although rarely used for this purpose, signal detection theory has the advantage that it distinguishes between sensory and decisional components of decision making. For example, by using signal detection theory to analyze nursing risk assessments for patients admitted to hospital, Thompson et al. have shown that time–pressure and clinical experience influence variability in decision making through different mechanisms. We hypothesized that signal detection theory would allow us to identify reasons for non-compliance with clinical practice guidelines. Better understanding of the cognitive processes responsible for under-triage would provide valuable information for the design of future quality improvement interventions .
We presented physicians with a series of case vignettes that required a range of triage decision-making. We found that when answering the case vignettes most physicians demonstrated limited perceptual sensitivity, as shown in Figure 4. The ACS-COT uses ATLS, an educational program that operationalizes the clinical guidelines, as one of its primary tools to standardize the treatment of trauma patients. However, although most physicians in our sample had received ATLS certification, the regression weight on the guidelines in our models suggested that triage decisions on this questionnaire corresponded only weakly with the ACS-COT criteria for the transfer of patients.
Patterns in the responses suggested potential explanations for the limited perceptual sensitivity demonstrated by these physicians. For example, as shown in Table 3, among one-half of the under-triaged patients, physicians correctly identified the patient as meeting the standard for transfer. However, they delayed transport to obtain additional diagnostic imaging. Acquisition of this imaging suggested triage decision-making consistent with the disjunction effect. As described by Shafir and Tversky, people have difficulty making decisions in the context of complex uncertainty. One manifestation of that difficulty is the pursuit of non-instrumental information, which appears relevant but if available would not impact decision making [23, 24]. In effect, the information from imaging acquired for patients with moderate to severe injuries would have had negative value, as it reduced the chances of successful treatment.
Similarly, the greater likelihood of transferring patients with penetrating (rather than blunt) injury suggested reliance on the representativeness heuristic. Formally equivalent problems should provoke equivalent judgments. In other words, people should estimate the probability of x belonging to set A in the same way that they estimate the probability of y belonging to set A. However, Kahneman and Tversky have shown that when people rely on the representativeness heuristic to make judgments, they substitute the perceived similarity of x or y to other objects in set A for that probability estimate . In this context, cases with blunt injuries seemed less likely to need tertiary center care as otherwise equivalent cases with penetrating injuries. This hypothesis would also explain patterns of patient-level variability in triage. For example, if physicians judge severe trauma as the purview of young men, they may systematically under-triage elderly patients, as described by Chang et al. and Macias et al. [8, 10].
Figure 4 also demonstrates that the group of physicians who responded to the questionnaire had a high threshold for transfer. To target physicians’ decisional thresholds, the ACS-COT has relied on trauma systems, voluntary networks of local community hospitals that associate with high-volume trauma centers . Through outreach and accreditation programs, it has urged physicians at non-trauma centers to have a low decisional threshold in order to minimize under-triage. In other words, it advocates transferring as many patients as necessary to ensure the capture of all those with moderate to severe injuries, even at the cost of increasing over-triage . Despite these efforts, our sample of physicians preferred to err on the side of minimizing over-triage. Free-text comments by participants suggested that their decisional thresholds reflected conscious negative attitudes towards transferring patients, invoking issues such as resentment towards guidelines in general and distaste at relinquishing control over patient care. The finding of lower decisional thresholds among physicians at hospitals with an established trauma center affiliation suggested that organizational norms or incentives may play roles as well. For example, hospitals without these affiliations may encourage their physicians to avoid transferring patients out of network to prevent the loss of revenue.
Current quality improvement efforts in trauma assume that the same barriers to compliance with clinical practice guidelines affect all physicians equally. However, the individual performance differences revealed in Figure 5 suggest the need for a more nuanced approach. For example, some physicians had a high threshold for transferring patients, and an above average ability to discriminate between patients with minor and moderate-severe injuries. These doctors seemed to at least partially compensate for their apparent unwillingness to transfer patients through skill at distinguishing between those who really did ‘need’ transfer and those who did not. Other physicians had a threshold for transferring patients set around zero and a lower than average perceptual sensitivity. These doctors seemed to at least partially compensate for their apparent inability to distinguish between those who did and did not ‘need’ transfer by being more willing to transfer everyone.
Physicians with decisional thresholds biased away from transfer and with above-average perceptual sensitivity would most likely benefit from an intervention that recalibrated their decisional threshold, perhaps by addressing their concerns about transferring patients or creating financial incentives for the appropriate transfer of patients. In contrast, exposing the second type of physician with a decisional threshold already set around zero to that same intervention might have unwanted consequences. Specifically, increasing their willingness to transfer patients would increase over-triage, and could impose a burden on level I centers. An analysis of triage patterns in Pennsylvania by Mohan et al. demonstrates that simply shifting decisional thresholds to achieve ACS-COT targets for triage would result in a five-fold increase in transfers to trauma centers . Moreover, non-trauma centers would lose an important source of revenue and the opportunity to provide care for patients in their community. Instead, these physicians might benefit from a strategy that modified their heuristics, perhaps through training with stimuli like the vignettes used in this study .
Our study had several limitations. First, we used vignettes to measure physicians’ perceptual sensitivity and decisional threshold, which did not replicate the time or organizational pressures of clinical decision-making, and with less information than may be available in real-life. Evidence suggesting that case vignettes can predict physician practice patterns comes primarily from the outpatient setting. For example, Peabody et al. demonstrated that physicians’ management of ‘paper patients’ with common clinical conditions, like lower back pain, corresponded to their management of real patients [27, 28]. Our specific vignettes have the content validity that comes from basing them on the case histories of patients treated at the University of Pittsburgh Medical Center. Additionally, the vignettes were extensively pretested. However, we have no knowledge of whether physician decision-making in response to a case vignette-based questionnaire corresponds to actual practice patterns. An alternative approach, and a possibility for future research, might be using an administrative dataset to calculate physicians’ perceptual sensitivity and decisional threshold based on actual triage decisions.
Second, we used the ACS-COT guidelines as our reference standard for transfer, despite flaws that might affect acceptance by physicians: all the stakeholders did not participate in their design ; the definition of over-triage creates an association between prevalence of injury and performance , and the benchmarks may lack feasibility . Yet, robust observational data supports their overall validity. MacKenzie et al. among others, have shown that patients meeting ACS-COT criteria for transfer have better outcomes when treated at trauma centers . Moreover, their widespread dissemination makes them the de facto standard for decision making among physicians, regardless of their potential failings.
Third, we used a non-representative sample of cases on our questionnaire to allow the use of signal detection theory. We did systematically vary the complexity of the cases to elicit the range of decisions that physicians would perform in practice. However, we used a much higher proportion of cases with moderate to severe injuries than physicians would see in practice. We speculate that the absence of non-trauma cases made the triage task easier than the one faced by physicians in their EDs. However, the higher base rate of severe injuries may have altered customary practice patterns. Birnbaum has shown that the predicted effect of the base rate on signal detection estimates depends on the theory of judgment used . We therefore have no clear feeling for how this bias would have affected the patterns observed here. As a test of possible learning effects and base rate influence on the parameter estimates, we compared performance on the first half of the study relative to the second half and found no differences.
Finally, we recruited physicians at a national meeting of emergency medicine physicians, which might limit the generalizability of our observations. However, because physicians attending academic meetings likely have greater knowledge of current clinical practice guidelines, we assume that any bias introduced by our sampling frame would be in the direction of better practice.
Under-triage persists despite comprehensive efforts by governmental and professional organizations to standardize trauma practice. In responses to a case vignette-based questionnaire, we found that under-triage can result from cognitive processes used in physician decision-making. Further research is required to determine how cognitive aspects of physician decision-making affects the triage of patients in real practice, as well as how best to intervene.
We thank many University of Pittsburgh faculty and staff for their contributions: Jason Sperry for his help reviewing the case vignettes; Donald Yealy for his help recruiting physician subjects and his contributions to the instrument design; Christina Steele for her help recruiting physician subjects; Anupama Vemuganti, Jinghua Ou and Timothy Van Dyke for their help developing the website. We also thank the many physicians who participated in this study.
This work was supported by grant 1KL2RR024154 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH) and NIH Roadmap for Medical Research (DM). Its contents are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH. Information on NCRR is available at http://www.ncrr.nih.gov/. Information on Re-engineering the Clinical Research Enterprise can be obtained from http://nihroadmap.nih.gov/clinicalresearch/overview-translational.asp. This work was also funded in part by a seed grant from the Department of Critical Care Medicine at the University of Pittsburgh (DM). The funding agencies had no role in the design and conduct of the study; collection, management, analysis and interpretation of the data; or preparation, review or approval of the manuscript.
- MacKenzie EJ, Rivara FP, Jurkovich GJ: A national evaluation of the effect of trauma-center care on mortality. NEJM. 2006, 354 (4): 366-378. 10.1056/NEJMsa052049.View ArticlePubMed
- Celso B, Tepas J, Langland-Orban B: A systematic review and meta-analysis comparing outcome of severely injured patients treated in trauma centers following the establishment of trauma systems. J Trauma. 2006, 60 (2): 371-378. 10.1097/01.ta.0000197916.99629.eb.View ArticlePubMed
- Demetriades D, Martin M, Salim A: The effect of trauma center designation and trauma voume on outcome in specific severe injuries. Ann Surg. 2005, 242 (4): 512-517.PubMed CentralPubMed
- Mann NC: Assessing the effectiveness and optimal structure of trauma systems: a consensus among experts. J Trauma. 1999, 47 (3S): S69-S74.View ArticlePubMed
- Mullins RJ, Mann NC: Population based research assessing the effectiveness of trauma systems. J Trauma. 1999, 47 (3S): S59-S66.View ArticlePubMed
- MacKenzie EJ, Rivara FP, Jurkovich GJ: The impact of trauma-center care on functional outcomes following major lower-limb trauma. J Bone Joint Surg Am. 2008, 90 (1): 101-109. 10.2106/JBJS.F.01225.View ArticlePubMed
- Committee on Trauma – American College of Surgeons: Resources for optimal care of the injured patient. 2006, American College of Surgeons, Chicago
- Macias CA, Rosengart MR, Puyana JC: The effects of trauma center care, admission volume, and surgical volume on paralysis after traumatic spinal cord injury. Ann Surg. 2009, 249 (1): 10-17. 10.1097/SLA.0b013e31818a1505.PubMed CentralView ArticlePubMed
- Nathens AB, Jurkovich GJ, MacKenzie EJ, Rivara FP: A resource-based assessment of trauma care in the United States. J Trauma. 2004, 56 (1): 173-178. 10.1097/01.TA.0000056159.65396.7C.View ArticlePubMed
- Chang DC, Bass RR, Cornwell EE, MacKenzie EJ: Undertriage of elderly trauma patients to state-designated trauma centers. Arch Surg. 2008, 143 (8): 776-781. 10.1001/archsurg.143.8.776.View ArticlePubMed
- Mohan D, Rosengart MR, Farris C: Are the American College of Surgeons’ guidelines for the transfer of trauma patients feasible?. Arch Surg. 2011, 146: 786-792. 10.1001/archsurg.2011.43.PubMed CentralView ArticlePubMed
- Cabana MD, Rand CS, Powe NR: Why don't physicians follow clinical practice guidelines?. JAMA. 1999, 282 (15): 1458-1465. 10.1001/jama.282.15.1458.View ArticlePubMed
- US Department of Health and Human Services: Model Trauma System Planning and Evaluation. https://www.socialtext.net/acs-demo-wiki/index.cgi?regional_trauma_systems_optimal_elements_integration_and_assessment_systems_consultation_guide Published February 2006. Accessed November 21, 2010
- McFall RM, Treat TA: Quantifying the information value of clinical assessments with signal detection theory. Annu Rev Psych. 1999, 50: 215-241. 10.1146/annurev.psych.50.1.215.View Article
- Swets JA, Dawes RM, Monahan J: Psychological science can improve diagnostic decisions. Psychol Sci Public Interest. 2000, 1 (1): 1-26. 10.1111/1529-1006.001.View ArticlePubMed
- Cohen J: Quantitative methods in psychology: a power primer. Psych Bull. 1992, 112 (1): 155-159.View Article
- McGinn T, Wyer PC, Newman TB: Tips for learners of evidence-based medicine: 3. Measures of observer variability (kappa statistic). CMAJ. 2004, 171: 1369-1373. 10.1503/cmaj.1031981.PubMed CentralView ArticlePubMed
- Yonelinas AP, Colleen CM: Receiver operating characteristics in recognition memory: a review. Psychol Bull. 2007, 133: 800-832.View ArticlePubMed
- Yeater EA, Treat TA, Viken RJ: Cognitive processes underlying women’s risk judgment: associations with genderual victimization history and rape myth acceptance. J Consult Clin Psych. 2010, 78 (3): 375-386.View Article
- DeCarlo TL: Signal detection theory and generalized linear models. Psychol Methods. 1998, 3 (2): 186-205.View Article
- Macmillan NA, Creelman CD: Detection Theory: A User's Guide. 2005, Lawrence Erlbaum Associates, New York
- Thompson C, Dalgleish L, Bucknell T: The effects of time pressure and experience on nurses' risk assessment decisions: a signal detection analysis. Nurs Res. 2008, 57 (5): 302-311. 10.1097/01.NNR.0000313504.37970.f9.View ArticlePubMed
- Shafir E, Tversky A: Thinking through uncertainty: nonconsequential reasoning and choice. Cogn Psychol. 1992, 24: 449-474. 10.1016/0010-0285(92)90015-T.View ArticlePubMed
- Bastardi A, Shafir E: On the pursuit and misuse of useless information. J Pers Soc Psychol. 1998, 75 (1): 19-32.View ArticlePubMed
- Kahneman D, Tversky A: Judgment under uncertainty: Heuristics and biases. Judgment under uncertainty: Heuristics and biases. Edited by: Daniel K, Paul S, Amos T. 2003, Cambridge University Press, New York, NY
- Wolfe JM, Horowitz TS, Van Wert MJ: Low target prevalence is a stubborn source of errors in visual search tasks. J Exp Psychol. 2007, 136: 623-638.View Article
- Shah R, Edgar DF, Evans BJW: A comparison of standardized patients, record abstraction and clinical vignettes for the purpose of measuring clinical practice. Ophthal Physiol Opt. 2010, 30: 209-224. 10.1111/j.1475-1313.2010.00713.x.View Article
- Peabody JW, Luck J, Glassman P: Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality. JAMA. 2000, 283 (13): 1715-1722. 10.1001/jama.283.13.1715.View ArticlePubMed
- Institute of Medicine: Clinical guidelines we can trust. 2011, National Academies Press, Washington D.C
- Birnbaum MH: Base rates in bayesian inference: signal detection analysis of the cab problem. Am J Psychol. 1983, 96: 85-94. 10.2307/1422211.View Article
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.