The United States Medical Licensing Examination (USMLE) is a three-step national competency test required for physicians before they can practice medicine. Step 1 is traditionally taken by second-year medical students, and tests basic physiologic mechanisms and principles. Step 2 emphasizes clinical diagnosis and disease pathogenesis, and is traditionally taken at the end of medical school. Step 3 tests first year residents in clinical management. Those seeking ECFMG certification can take Step 1 and Step 2 in any order, but must pass both in order to take Step 3. Since their introduction (Steps 1 and 2 in 1992 and Step 3 in 1994) each has been administered twice annually in a two-day format consisting of multiple-choice questions (MCQs), either "one best answer" or "extended matching."
Until last year the USMLE was only administered as a written exam, commonly referred to as pen-and-paper testing (PPT). In March 1999, the National Board of Medical Examiners (NBME) -- after years of study and development, accompanied by much controversy -- launched computerized-based testing (CBT). USMLE Step 1 was computerized in May, Step 2 in August, and Step 3 will be computer-administered in mid-November of 1999. By the end of November, a great many medical students and physicians-in-training will have experienced national standardized CBT.
As described on the USMLE Web site, computer-based testing (CBT) will introduce a number of administrative changes in addition to changes in test format. Each Step is now offered throughout the year rather than twice annually. Instead of two eight-hour testing days, Steps 1and 2 are each completed in a day; Step 3 remains a two-day exam.
CBT Steps 1 and 2 are taken in seven or eight blocks of 30-60 minutes each, using the same MCQ types as the written exam. Only the layout (not content) is adjusted for the computer. Step 3 will include both MCQs and one or more blocks of computer-based case simulations (CCS) taken during the afternoon of the second day. During the defined time examinees may answer questions and review/change answers in any order, but cannot return to the block after exiting or when the time allotment expires. Minimum break time for the entire exam (time between blocks and for lunch) is 45 minutes.
Sylvan Prometric, a division of Sylvan Learning Systems, provides scheduling and test centers for USMLE. After registering with USMLE, the examinee then schedules a test date at the most convenient center.
Why computer-based training?
Although it has been slow to gain acceptance within the American medical education system, CBT is not altogether new to other health professionals. For example, the National Council Licensure Examination for Registered Nurses (NCLEX-RN) has been computerized since 1994, and in China a computerized examination system has been used since 1989 to assess applicants for residencies.1
The necessity of computerizing the USMLE may have been controversial, but not the the need for increased examination security, improved test formats, more flexible scheduling for examinees, and a more efficient test administration process overall. The written exam was already mostly computerized, from automated scoring of bubble forms and computer analysis of the results, to electronic reporting of students' scores to their respective medical schools. Further computerizing its administration will eliminate time wasted in converting test answers to the electronic format. Eventually, score reports will be mailed to examinees and transmitted to medical schools within two weeks of the examination, rather than the months it takes now. This is particularly significant for Step 1 of the USMLE, since many schools require a passing grade for matriculation or promotion to the third year.
In spite of CBT's supposed benefits over PPT, however, controversy surrounds the possible differences in performance directly caused by the mode of administration. In addition to providing an excellent review of prior research in the field, Luecht et al. reported the results of a field study using statistically identical PPT and CBT versions of the basic science portion of the USMLE.2 The researchers employed a crossover experimental design, in which students took both the PPT test and CBT test. Comparisons of examinee performance showed significant correlation (r=0.82) between the two tests; furthermore, the difference between total-group performance (61.2 vs 61.5) was statistically insignificant. Thus, the authors concluded that computerization of the test alone did not appear to affect examinee performance.
Medical students vary in their experience using computers, and physicians vary even more. So it is noteworthy that Leucht also found that subjects' prior computer experience had little or no effect on CBT performance. To address disparate computing experiences, USMLE will insure that all examinees take a short pretest tutorial on test taking and computers. Also, the NBME makes sample multiple-choice questions in the CBT format on its Web site and on CD-ROM available to all applicants and medical institutions. Practice sessions are particularly emphasized for the last afternoon session in Step 3, using its Primum CCS software. Practice sessions are also offered at the testing centers prior to the actual exam, for an additional fee.
The NBME cannot provide a computer for every student taking the exam. Thus, the USMLE is now administered on an individual basis on one of several possible days, rather than having a single, two-day administration for all test takers. Furthermore, because test-takers can be individually timed, the CBT version of the USMLE is now administered in blocks (seven for Step 1, eight for Step 2) interspersed with break time, which the test-taker must budget him- or herself.
Unfortunately, as the test takers may take the exam at different times, this presents a new security issue. That is, the potential arises for examinees who have already taken the test to collude with those who have not. To address this, the NBME recently sent out a cautionary advisory via e-mail to several medical schools noting that it actively monitors medical student Web sites and listservs for evidence of irregular behavior and disclosure of exam content. At the same time, the CBT affords significant advantages in terms of test security; examination materials are encrypted to confound thieves and transmitted electronically, eliminating the possibility of stealing paper copies of exams. Exam conditions also can be better controlled and standardized at the testing centers. To insure that only medical students scheduled for the exam gain entrance, examinees are currently subject to identification measures; the testing centers are also equipped with on-site digital cameras, as well as audio and video surveillance. Furthermore, the exam questions (350 for Step 1, 400 for Step 2) are drawn from a large data pool, so that on any given day a number of different exams are administered.
These security issues are often overshadowed by the advantages that CBT brings to standardized exam formats. For example, computerization of the USMLE will allow the use of computerized adaptive testing (CAT) such as that used in the NCLEX-RN.3 CAT assesses the examinee's knowledge base by using statistical theory to adjust the content and difficulty of each question based on prior answers and overall exam performance. Full-scale implementation of CAT on the USMLE has not yet been formally announced.
Computer-based testing may also offer the potential for new question formats. With multiple-choice questions (MCQs), for example, the test-taker may be able to arrive at the correct answers by unintentional or subliminal hints offered by the answer choices themselves. Such cueing can make current exams less accurate tools for assessment. Cueing effects might be eliminated, and test accuracy improved, by the use of computerized long-menu questions (CLMs). Schuwirth et al.4 and Damjanov5 have discussed the use of CLMs, which present the test-taker with a long list of possible answers rather than four or five choices. Based on their studies, they suggest scores from tests using CLMs are more comparable to exams with open-ended questions rather than MCQ-based exams. In the future, the NBME might reduce cueing -- and perhaps even improve examinee assessment -- by using clinical and laboratory simulations, use of multimedia presentations, and "point and click" question formats.
Broadening the use of CBT to other exams for health professionals affords many practical advantages, such as the use of national or state-level standardized board examinations to more clearly identify areas in need of remedial education (rather than simply granting or withholding certification or licensure as is currently the practice). Technology can also allow this process to be nearly instantaneous, which is particularly important if a practicing physician has a critical deficiency in his/her knowledge base. Donald and Thompson6 discussed the success of one such system in a study of British general practitioner trainees. But this needs to be studied further, since Butzin et al.7 found that a group of pediatricians who had thought they had learned more using such a system to identify their weaknesses did not actually demonstrate a significant learning effect.
Thus, overall, what remains to be seen are the modifications the NBME chooses to be make to the USMLE in the near future, including CAT, new questions formats, and improved security measures. Such issues may be addressed in an October 1999 progress report by the NBME. The success or failure of the computer-based USMLE will also affect the design and administration of other examinations outside NBME's control, including the currently paper-based MCAT, as well as specialty certification examinations. Finally, as universal access to the Internet becomes commonplace and advanced security channels are developed, the USMLE may be able to avoid the added expense and security risk of involving a third-party administrator by communicating electronically with each examinee directly and at their convenience, at home or at work.
References- Zhang X, Cai F, Liu F, Bao X, Liu Y. A new and cheap medical examination system with artificial intelligence, abstracted. Proceedings of the Eighth World Congress on Medical Informatics 1995;8(pt 2):1702. (abstract) PubMed Link Back to text
- Luecht RM, Hadadi A, Swanson DB, Case SM.A comparative study of a comprehensive basic science test using paper-and-pencil and computerized formats. Acad Med 1998;73(October suppl):S51-3 PubMed Link Back to text
- Forker JE, McDonald ME. Methodologic trends in the healthcare professions: computer adaptive and computer simulation testing. Nurse Educ 1996;21:13-14 PubMed Link Back to text
- Schuwirth LW, van der Vlueten CP, Donkers HH. A closer look at cueing effects in multiple-choice questions. Med Educ 1996;30:44-49 PubMed Link Back to text
- Damjanov I, Fenderson BA, Veloski JJ, Rubin E. Testing of medical students with open-ended, uncued questions. Hum Pathol 1995 Apr;26:362-365. (abstract) PubMed Link Back to text
- Donald JB, Thomson D. Evaluation of a computerized assessment package for general practitioner trainees. Br J Gen Pract 1993;43:115-118. (abstract) PubMed Link Back to text
- Butzin DW, Friedman CP, Brownlee RC. A pilot study of microcomputer testing in paediatrics. Med Educ 1984;18:339-342. PubMed Link Back to text
