Detecting cheating in written medical examinations by statistical anal
http://www.100md.com
《英国医生杂志》
1 Department of Psychology, University College London, London WC1E 6BT, 2 Examinations Department, Royal College of Paediatrics and Child Health, London W1W 6DE
Correspondence to: I C McManus i.mcmanus@ucl.ac.uk
Objective To assess whether a computer program using a variant of Angoff's method can detect anomalous behaviour indicative of cheating in multiple choice medical examinations.
Design Statistical analysis of 11 examinations held by the Royal College of Paediatrics and Child Health.
Setting UK postgraduate medical examination.
Participants Examination candidates.
Main outcome measures Detection of anomalous candidate pairs by regression of similarity of correct answers in all possible pairs of candidates on the overall proportion of correct answers. Anomalous pairs were subsequently assessed in terms of examination centres and the seating plan of candidates, to assess adjacency.
Results The 11 examinations were taken by a total of 11 518 candidates, and Acinonyx examined 6 178 628 pairs of candidates. Two examinations showed no anomalies, and one examination found an anomaly resulting from a scanning error. The other eight examinations showed 13 anomalies compatible with cheating, and in each pair the two candidates had sat the examination at the same centre, and for six examinations with seating plans, the candidates in the anomalous pairs had been seated side by side. The raw probabilities of the anomalies varied from 3.9x10-11 to 9.3x10-30 (median = 1.1x10-17), with Bonferroni-corrected probabilities in the range 2.4x10-5 to 4.1x10-24 (median = 1.6x10-11). This suggests that one anomalous pair is found for every 1000 or so candidates taking this postgraduate examination.
Conclusions This statistical technique identified a small proportion of candidates who had very similar patterns of correctly answered questions. The likelihood is that one candidate has copied from the other, or that there was collusion, or that a technical error occurred in the exams department (as happened in a single case). Analysis of similarities can be used to identify cheating and as part of the quality assurance process of postgraduate medical examinations.
Cizek GJ. Cheating on tests: How to do it, detect it, and prevent it. Mahwah, NJ: Erlbaum, 1999.
Bay L. Factors associated with cheating among college students: a review. Res Higher Educ 1998;39: 235-74.
Wesolowsky GO. Detecting excessive similarity in answers on multiple choice exams. J Applied Statistics 2000;27: 909-21.
Sierles F, Hendrickx I, Circle S. Cheating in medical school. J Med Educ 1980;55: 124-5.
Stimmel B,.Yens D. Cheating by medical students on examinations. Am J Med 1982;73: 160-4.
Carlova J. How many doctors are cheating their way into practice? Med Econ 1984;61: 84-92.
Rozance CP. Cheating in medical schools: implications for students and patients. JAMA 1991;255: 2453-6.
Baldwin D-C Jr, Daugherty SR, Rowley BD, Schwarz MD. Cheating in medical school: a survey of second-year students at 31 schools. Acad Med 1996;71: 267-73.
Angoff WH. The development of statistical indices for detecting cheaters. J Am Stat Assoc 1974;69: 44-9.
Merwin SE, Moeller DW, Kenedy WE, Moeller MP. Application of the Supreme Court's Daubert criteria in radiation litigation. Health Physics 2001;81: 670-7.
Nizer L. My life in court. New York: Doubleday, 1961.
General Medical Council. Good medical practice. London: GMC, 2001.
General Medical Council Professional Conduct Committee. Professional conduct hearing: Purmessur, Shyam Nundun Singh, 6-9 October 2003. www.gmc-uk.org/probdocs/decisions/pcc/2003/PURMESSUR_20031009.htm (accessed 8 april 2005).
Stearns SA. Administrative ramifications of student cheating. J Assoc Communication Admin 1997;(2): 133-9.(I C McManus, professor of psychology and)
Correspondence to: I C McManus i.mcmanus@ucl.ac.uk
Objective To assess whether a computer program using a variant of Angoff's method can detect anomalous behaviour indicative of cheating in multiple choice medical examinations.
Design Statistical analysis of 11 examinations held by the Royal College of Paediatrics and Child Health.
Setting UK postgraduate medical examination.
Participants Examination candidates.
Main outcome measures Detection of anomalous candidate pairs by regression of similarity of correct answers in all possible pairs of candidates on the overall proportion of correct answers. Anomalous pairs were subsequently assessed in terms of examination centres and the seating plan of candidates, to assess adjacency.
Results The 11 examinations were taken by a total of 11 518 candidates, and Acinonyx examined 6 178 628 pairs of candidates. Two examinations showed no anomalies, and one examination found an anomaly resulting from a scanning error. The other eight examinations showed 13 anomalies compatible with cheating, and in each pair the two candidates had sat the examination at the same centre, and for six examinations with seating plans, the candidates in the anomalous pairs had been seated side by side. The raw probabilities of the anomalies varied from 3.9x10-11 to 9.3x10-30 (median = 1.1x10-17), with Bonferroni-corrected probabilities in the range 2.4x10-5 to 4.1x10-24 (median = 1.6x10-11). This suggests that one anomalous pair is found for every 1000 or so candidates taking this postgraduate examination.
Conclusions This statistical technique identified a small proportion of candidates who had very similar patterns of correctly answered questions. The likelihood is that one candidate has copied from the other, or that there was collusion, or that a technical error occurred in the exams department (as happened in a single case). Analysis of similarities can be used to identify cheating and as part of the quality assurance process of postgraduate medical examinations.
Cizek GJ. Cheating on tests: How to do it, detect it, and prevent it. Mahwah, NJ: Erlbaum, 1999.
Bay L. Factors associated with cheating among college students: a review. Res Higher Educ 1998;39: 235-74.
Wesolowsky GO. Detecting excessive similarity in answers on multiple choice exams. J Applied Statistics 2000;27: 909-21.
Sierles F, Hendrickx I, Circle S. Cheating in medical school. J Med Educ 1980;55: 124-5.
Stimmel B,.Yens D. Cheating by medical students on examinations. Am J Med 1982;73: 160-4.
Carlova J. How many doctors are cheating their way into practice? Med Econ 1984;61: 84-92.
Rozance CP. Cheating in medical schools: implications for students and patients. JAMA 1991;255: 2453-6.
Baldwin D-C Jr, Daugherty SR, Rowley BD, Schwarz MD. Cheating in medical school: a survey of second-year students at 31 schools. Acad Med 1996;71: 267-73.
Angoff WH. The development of statistical indices for detecting cheaters. J Am Stat Assoc 1974;69: 44-9.
Merwin SE, Moeller DW, Kenedy WE, Moeller MP. Application of the Supreme Court's Daubert criteria in radiation litigation. Health Physics 2001;81: 670-7.
Nizer L. My life in court. New York: Doubleday, 1961.
General Medical Council. Good medical practice. London: GMC, 2001.
General Medical Council Professional Conduct Committee. Professional conduct hearing: Purmessur, Shyam Nundun Singh, 6-9 October 2003. www.gmc-uk.org/probdocs/decisions/pcc/2003/PURMESSUR_20031009.htm (accessed 8 april 2005).
Stearns SA. Administrative ramifications of student cheating. J Assoc Communication Admin 1997;(2): 133-9.(I C McManus, professor of psychology and)