General Guidelines for Developing Multiple-Choice Items

adapted from a longer article by Mary E. Piontek, Best Practices for Designing and Grading Exams

Multiple-choice items have a number of advantages. First, multiple-choice items can measure various kinds of knowledge, including students' understanding of terminology, facts, principles, methods, and procedures, as well as their ability to apply, interpret, and justify. When carefully designed, multiple-choice items can assess higher-order thinking skills as shown in Example 1, (below) in which students are required to generalize, analyze, and make inferences about data in a medical patient case.

Multiple-choice items are less ambiguous than short-answer items, thereby providing a more focused assessment of student knowledge. Multiple-choice items are superior to true-false items in several ways: on true-false items, students can receive credit for knowing that a statement is incorrect, without knowing what is correct. Multiple-choice items offer greater reliability than true-false items as the opportunity for guessing is reduced with the larger number of options. Finally, an instructor can diagnose misunderstanding by analyzing the incorrect options chosen by students.

A disadvantage of multiple-choice items is that they require developing incorrect, yet plausible, options that can be difficult for the instructor to create. In addition, multiple-choice questions do not allow instructors to measure students' ability to organize and present ideas. Finally, because it is much easier to create multiple-choice items that test recall and recognition rather than higher order thinking, multiple-choice exams run the risk of not assessing the deep learning that many instructors consider important (Gronlund & Linn, 1990; McMillan, 2001).

-------------------------------
Example 1: A Series of Multiple-Choice Items That Assess Higher Order Thinking:

Patient WC was admitted for 3rd degree burns over 75% of his body. The attending physician asks you to start this patient on antibiotic therapy. Which one of the following is the best reason why WC would need
antibiotic prophylaxis?
a. His burn injuries have broken down the innate immunity that prevents microbial invasion.
b. His injuries have inhibited his cellular immunity.
c. His injuries have impaired antibody production.
d. His injuries have induced the bone marrow, thus activated immune system.

Two days later, WC's labs showed: WBC 18,000 cells/mm3; 75% neutrophils (20% band cells); 15% lymphocytes; 6% monocytes; 2% eosophils; and 2% basophils. Which one of the following best describes WC's lab results?
a. Leukocytosis with left shift
b. Normal neutrophil count with left shift
c. High eosinophil count in response to allergic reactions
d. High lymphocyte count due to activation of adaptive immunity
(Jeong Park, U-M College of Pharmacy, personal
communication, February 4, 2008)
------------------------------

Guidelines for developing multiple-choice items

There are nine primary guidelines for developing multiple-choice items (Gronlund & Linn, 1990; McMillan, 2001). Following these guidelines increases the validity and reliability of multiple-choice items that one might use for quizzes, homework assignments, and/or examinations.

The first four guidelines concern the item "stem," which poses the problem or question to which the choices refer.

1. Write the stem as a clearly described question, problem, or task.
2. Provide the information in the stem and keep the options as short as possible.
3. Include in the stem only the information needed to make the problem clear and specific.

The stem of the question should communicate the nature of the task to the students and present a clear problem or concept. The stem of the question should provide only information that is relevant to the problem or concept, and the options (distractors) should be succinct.

4. Avoid the use of negatives in the stem (use only when you are measuring whether the respondent knows the
exception to a rule or can detect errors).

You can word most concepts in positive terms and thus avoid the possibility that students will overlook terms of "no, not, or least" and choose an incorrect option not because they lack the knowledge of the concept but because they have misread the stated question. Italicizing, capitalizing, using bold-face, or underlying the negative term makes it less likely to be overlooked.

The remaining five guidelines concern the choices from which students select their answer.

5. Have ONLY one correct answer.

Make certain that the item has one correct answer. Multiple-choice items usually have at least three incorrect options (distractors).

6. Write the correct response with no irrelevant clues. A common mistake when designing multiple-choice questions is to write the correct option with more elaboration or detail, using more words, or using general terminology rather than technical terminology.

7. Write the distractors to be plausible yet clearly wrong. An important, and sometimes difficult to achieve, aspect of multiple-choice items is ensuring that the incorrect choices (distractors) appear to be possibly correct. Distractors are best created using common errors or misunderstandings about the concept being assessed, and making them homogeneous in content and parallel in form and grammar.

8. Avoid using "all of the above," "none of the above," or other special distractors (use only when an answer can
be classified as unequivocally correct or incorrect).

All of the above and none of the above are often added as answer options to multiple-choice items. This technique requires the student to read all of the options and might increase the difficulty of the items, but too often the use of these phrases is inappropriate. None of the above should be restricted to items of factual knowledge with absolute standards of correctness. It is inappropriate for questions where students are asked to select "the best" answer. All of the above is awkward in that many students will choose it if they can identify at least one of the other options as correct and therefore assume all of the choices are correct - thereby obtaining a correct answer based on partial knowledge of the concept/content (Gronlund & Linn, 1990).

9. Use each alternative as the correct answer about the same number of times.

Check to see whether option "a" is correct about the same number of times as option "b" or "c" or "d" across the instrument. It can be surprising to find that one has created an exam in which the choice "a" is correct 90% of the time. Students quickly find such patterns and increase their chances of "correct guessing" by selecting that answer option by default.

-----------------------------
Checklist for Writing Multiple-Choice Items
* Is the stem stated as clearly, directly, and simply as possible?
* Is the problem self-contained in the stem?
* Is the stem stated positively?
* Is there only one correct answer?
* Are all the alternatives parallel with respect to grammatical structure, length, and complexity?
* Are irrelevant clues avoided?
* Are the options short?
* Are complex options avoided?
* Are options placed in logical order?
* Are the distractors plausible to students who do not know the correct answer?
* Are correct answers spread equally among all the choices?
(McMillan, 2001, p. 150)
----------------------

References (for full article)

Brown, F. G. (1983). Principles of educational and psychological testings(3rd ed.). New York: Holt, Rinehart
and Winston.

Cashin, W. E. (1987). Improving essay tests. Idea Paper, No. 17. Manhattan, KS: Center for Faculty Evaluation and
Development, Kansas State University.

Critical thinking rubric. (2008). Dobson, NC: Surry Community College.

Grading systems. (1991, April). For Your Consideration, No. 10. Chapel Hill, NC: Center for Teaching and Learning,
University of North Carolina at Chapel Hill.

Gronlund, N. E., & Linn, R. L. (1990). Measurement and evaluation in teaching (6th ed.). New York: Macmillan
Publishing Company.

Halpern, D. H., & Hakel, M. D. (2003). Applying the science of learning to the university and beyond. Change,
35(4), 37-41.

Isaac, S., & Michael, W. B. (1990). Handbook in research and evaluation.San Diego, CA: EdITS Publishers.

McKeachie, W. J. , & Svinicki, M. D. (2006). Assessing, testing, and evaluating: Grading is not the most important
function. In McKeachie's teaching tips: Strategies,research, and theory for college and university teachers (12th ed., pp. 74-86). Boston: Houghton Mifflin Company.

McMillan, J. H. (2001). Classroom assessment:
Principles and practice for effective instruction. Boston:
Allyn and Bacon.

Seymour, E., & Hewitt, N. M. (1997). Talking about leaving: Why undergraduates leave the sciences. Boulder, CO: Westview Press.

Svinicki, M. D. (1998). Helping students understand grades. College Teaching, 46(3), 101-105.

Svinicki, M. D. (1999a). Evaluating and grading students. In Teachers and students: A sourcebook for UT-Austin faculty(pp. 1-14). Austin, TX: Center for Teaching Effectiveness, University of Texas at Austin.

Svinicki, M. D. (1999b). Some pertinent questions about grading. In Teachers and students: A sourcebook for UT-Austin faculty(pp. 1-2). Austin, TX: Center for Teaching Effectiveness, University of Texas at Austin.

Thorndike, R. M. (1997). Measurement and evaluation in psychology and education. Upper Saddle River, NJ: Prentice-Hall, Inc.

Wiggins, G. P. (1998). Educative assessment: Designing assessments to inform and improve student performance. San Francisco: Jossey-Bass Publishers.

Worthen, B. R., Borg, W. R., & White, K. R. (1993). Measurement and evaluation in the schools. New York:
Longman. Writing and grading essay questions.(1990, September). For Your Consideration, No. 7. Chapel Hill, NC: Center for Teaching and Learning, University of North Carolina at Chapel Hill.

Teaching evening classes and nontraditional students

Teaching Large Evening Classes

They've been working all day, they're tired, and there are scores of them-how do you reach them all?

By Oscar Wambuguh

High enrollments, conflicting student work schedules, and the sheer convenience of once-a-week classes are pushing many colleges to schedule evening courses. Held from 6 to 9 pm or 7 to 10 pm, these classes at my institution are typically packed, sometimes with more than 150 students in a large lecture theater. How can faculty effectively teach, control, or even simply keep awake the students in such classes, many of whom start their days very early in the morning with family responsibilities?

Evening classes bring special teaching challenges. Evening students tend to interact socially more than day students, necessitating frequent instructor interventions to maintain calm and order, and many come to class overwhelmed, hungry, and tired. In addition, there are the usual challenges: poor lighting conditions in many large lecture theaters; lack of, or poor-quality, audiovisual equipment; and distractions caused by student electronic devices such as cell phones, iPods, or laptops. Under such conditions, even the most ardent instructor can become frustrated.

In my large evening classes, I always have a mix of students, ranging from the highly motivated to the couldn't-care-less types. I teach environmental health classes focusing on attention-catching topics such as global warming and climate change; air pollution; epidemiological issues like SARS and Asian bird flu; toxicology, especially of heavy metals in our water and food; biodiversity loss; and food concerns. Over the years, to enhance student engagement and maintain interest, I have incorporated strategies such as combining audiovisuals (PowerPoint presentations, short video clips, and transparencies) and class discussions and activities. Doing so is challenging in large classes set in "lecture-only" theaters. However, I have found four strategies that maintain student engagement and interest throughout what might otherwise be a long evening.

PowerPoint Presentations

I start my lectures with a two- to three-minute video clip about current course-related issues. Then I lecture using PowerPoint slides for about an hour and a quarter. Although PowerPoint isn't exactly a brand-new technology, it works well for my classes. Many students are relatively fresh at the beginning of class and likely to remain attentive through the whole lecture. I use illustrations and pictures interspersed with text slides as much as possible. I time myself carefully and always stop after seventy-five minutes, after which students take a ten-minute break to recharge for the next session.

Intergroup Questions

Before the first class meeting of a semester, I organize students in groups of eight to ten and post group numbers and the names of group members on a Blackboard site, where students can access them. I end up with fifteen to sixteen groups for a class of 150 students. During the first meeting, as I call students' names, I ask them to move into their groups for a "get-to-know each- other" session and information exchange. To maintain regularity, I ask group members to stay with their groups at the same location in the lecture theater in future meeting sessions. The groups are responsible for generating five short-answer questions (typed, with answers) for each class meeting. Within their groups, students take turns developing questions each week and circulating them among other group members for feedback through Blackboard before class. Once we begin intergroup question time, two groups are chosen to answer two questions asked by each of the other groups.

Here's how it works. When the first session begins, groups 1 and 2 are ready to begin answering questions. The first question comes from group 3, which asks group 1 one of its five short-answer questions. Members take turns each week reading the questions aloud as everyone in the class listens. One member from group 1 (members take turns) responds briefly (taking no more than two minutes total) to the question asked. If I judge that a concept is not clear, I add to the answer given. Then group 3 asks group 2 its next question, after which group 4 asks groups 1 and 2 questions. The process continues for between forty five and sixty minutes until all groups have been reached. Satisfactory answers are given two points each, recorded next to the question asked. Throughout, I maintain order and monitor timing. At the end, I collect all the groups' short-answer questions for grading and record the points earned by the two chosen groups. At the next meeting session, groups 3 and 4 will be the chosen groups, and so on until the last group is reached.

This system works well in a number of ways. It encourages students to read course material ahead of time; helps the instructor cover course content and explain concepts and material that are not clear during the question-and answer session; keeps students motivated and energized by requiring individual responsibility, attention, and group commitment (no one wants to be blamed for letting the group down); encourages active participation among students, allowing them to develop and polish their oral communication skills; gives students a sense of owning the questions and the learning; and creates excitement and a sense of achievement, especially if group members answer their questions correctly.

Article Time

Another part of our evening class deals with current articles in the field. Each group will have had a week to choose and summarize an article from the media (newspapers, newsmagazines, science magazines, journals, or the Internet) dealing with a topic covered by the course. Summaries, which are usually about half-a-page long, must be typed and must include the names of all the students in the group. Group members take turns choosing and summarizing the weekly article on behalf of their group.

During article time, the member of each group who chose and summarized the article stands up and tells the class the title of the article and the reasons why he or she chose it and then reads aloud the typed summary. The groups report in order of their group number, taking about two to three minutes each for a total of forty-five to sixty minutes. Everyone, including the instructor, listens, and the class is free to comment briefly after each article has been read. At the end of the session, the instructor collects the fifteen summaries for grading. This happens each week. Reviewing each article and its summary is quick, taking an average of about ten minutes (a total of just two-and a- half hours a week for the ten instruction weeks of the quarter).

I have found this process beneficial in many ways. It encourages literacy about current events in the field among everyone present, including the instructor; improves student reading and analytical and comprehension skills by requiring students to summarize two-to-three page articles in about half a page; improves students' confidence in their oral and written communication skills while also enhancing those skills; helps keep students engaged and motivated at a late hour of the evening as topics spin from one area to the next; and allows students to appreciate the practicality, complexity, and interdisciplinarity of some of the material covered by the course.

Short Quiz

While it is debatable whether college-level students should be required to physically attend every meeting session, I have made it hard for students to miss class unless something unavoidable happens (like an illness, a baby sitter not showing up, a transportation problem, or a family emergency). In such cases, students must call my voice mail or send me an e-mail and present a note at the next class meeting explaining the reasons for their absence.

What motivates students to be sure to come to my class? During the last ten to fifteen minutes of each evening session, we have an optional extra-credit quiz worth ten to fifteen points. If a student misses class but leaves me a voice or e-mail message and gives me a written note, I prorate his or her points based on a simple ratio of the total points earned for all prior quizzes divided by the total expected, multiplied by the total points for the quiz missed. The quiz, which is multiple choice, is given on a projection screen (eliminating unnecessary copying). Students complete it silently on fifteen- question Scantron forms- machine-readable forms designed for multiple-choice tests-without using class notes or text. I advance the questions on the projection screen as students answer them. I use a $20 sensor that works with PowerPoint and Word, which allows me to advance the questions remotely from anywhere in the room.

All the points students accumulate over the weeks are put into a "point savings account" that students can see on Blackboard. The "fatter" the account gets, the more motivated students become to keep it high. The account can swell to an average of about eighty-five points for most students. But they understand that no matter how many points they "save," their total is prorated so that it does not offset more than 10 percent of the overall course points, usually eight hundred. For example, if a student has accumulated twenty five points before the first midterm, he or she can miss up to twenty-five questions on the exam, which usually has a total of a hundred questions. If by the second midterm, the student has a total point savings of fifty-five, those points can offset the total points missed on the two midterms. Similarly, if by the final, the student has accumulated an overall total savings of eighty-five points, he or she can offset the total points missed on all three exams (subject to the 10 percent cutoff).

The quiz is very popular and has obvious benefits. It allows students to monitor their understanding of the lecture session's material right away; it gives the instructor feedback about the quality of instruction and student understanding; it motivates (or forces) students to stay for the entire class, as the quiz is administered at the very end; it encourages students to pay attention to the material presented (because of the pressure of immediate assessment); it allows the instructor to monitor daily attendance; and it saves the instructor's time, because some of the quiz questions can be rephrased and used on midterms and the final.

In addition, the quizzes encourage students to stay on top of the course material, because they have to review an evening's material ahead of time so that they can fill the "holes" as we discuss the same material in class. This, many students believe, provides an extra edge as they are not encountering this material for the first time before the quiz. Perhaps most important, the points students earn on quizzes give them a sense of hope that they can miss some questions on the midterms or final without penalty. The quizzes also allow them to remain optimistic about the course even after a devastating performance on the first midterm exam. Students quickly forget the optional nature of the quizzes and soon start taking them as a matter of priority and survival (some students who have missed an entire session have shown up the last fifteen minutes of class so as not to miss this quiz!).

During quizzes, the instructor has to remain vigilant, moving around the room to discourage students who whisper to each other, exchange Scantron forms, or check their notes. The instructor must also be sure to collect and secure all Scantron forms before taking time to chat with students after class.

Over the years, I have had positive evaluations about the quizzes. Many students are pleased that they get immediate feedback about their comprehension of course material, allowing them to take advantage of the period in which they can drop the course without penalty. Do the quizzes contribute to grade inflation? Hardly, given the 10 percent cutoff for the number of quiz points counted toward the final course grade and the fact that the questions are not "giveaways"; they are real exam questions, requiring advance preparation, sustained attention, and understanding.

The quizzes do not help all students. I find that some routinely miss class despite the cost to their grades, while some perform consistently on the quizzes. Most students actually look forward to taking the quizzes. On the rare occasions that my schedule doesn't allow me to prepare a quiz by the end of a class meeting, I can see disappointment on most students' faces. They consider they have been "robbed" of the opportunity to accumulate point savings and improve their grades.

Today, with an increasing number of nontraditional students returning to school, we are challenged to develop creative and innovative ways to make our classes work for them and to make our course material relevant to their daily lives. Evening classes, which afford these students flexibility and convenience, are one way to meet their needs, and I hope my experience with evening classes will help others find ways to make them work for this diverse and dedicated group of students.