General Guidelines for Developing Multiple-Choice Items

adapted from a longer article by Mary E. Piontek, Best Practices for Designing and Grading Exams

Multiple-choice items have a number of advantages. First, multiple-choice items can measure various kinds of knowledge, including students' understanding of terminology, facts, principles, methods, and procedures, as well as their ability to apply, interpret, and justify. When carefully designed, multiple-choice items can assess higher-order thinking skills as shown in Example 1, (below) in which students are required to generalize, analyze, and make inferences about data in a medical patient case.

Multiple-choice items are less ambiguous than short-answer items, thereby providing a more focused assessment of student knowledge. Multiple-choice items are superior to true-false items in several ways: on true-false items, students can receive credit for knowing that a statement is incorrect, without knowing what is correct. Multiple-choice items offer greater reliability than true-false items as the opportunity for guessing is reduced with the larger number of options. Finally, an instructor can diagnose misunderstanding by analyzing the incorrect options chosen by students.

A disadvantage of multiple-choice items is that they require developing incorrect, yet plausible, options that can be difficult for the instructor to create. In addition, multiple-choice questions do not allow instructors to measure students' ability to organize and present ideas. Finally, because it is much easier to create multiple-choice items that test recall and recognition rather than higher order thinking, multiple-choice exams run the risk of not assessing the deep learning that many instructors consider important (Gronlund & Linn, 1990; McMillan, 2001).

-------------------------------
Example 1: A Series of Multiple-Choice Items That Assess Higher Order Thinking:

Patient WC was admitted for 3rd degree burns over 75% of his body. The attending physician asks you to start this patient on antibiotic therapy. Which one of the following is the best reason why WC would need
antibiotic prophylaxis?
a. His burn injuries have broken down the innate immunity that prevents microbial invasion.
b. His injuries have inhibited his cellular immunity.
c. His injuries have impaired antibody production.
d. His injuries have induced the bone marrow, thus activated immune system.

Two days later, WC's labs showed: WBC 18,000 cells/mm3; 75% neutrophils (20% band cells); 15% lymphocytes; 6% monocytes; 2% eosophils; and 2% basophils. Which one of the following best describes WC's lab results?
a. Leukocytosis with left shift
b. Normal neutrophil count with left shift
c. High eosinophil count in response to allergic reactions
d. High lymphocyte count due to activation of adaptive immunity
(Jeong Park, U-M College of Pharmacy, personal
communication, February 4, 2008)
------------------------------

Guidelines for developing multiple-choice items

There are nine primary guidelines for developing multiple-choice items (Gronlund & Linn, 1990; McMillan, 2001). Following these guidelines increases the validity and reliability of multiple-choice items that one might use for quizzes, homework assignments, and/or examinations.

The first four guidelines concern the item "stem," which poses the problem or question to which the choices refer.

1. Write the stem as a clearly described question, problem, or task.
2. Provide the information in the stem and keep the options as short as possible.
3. Include in the stem only the information needed to make the problem clear and specific.

The stem of the question should communicate the nature of the task to the students and present a clear problem or concept. The stem of the question should provide only information that is relevant to the problem or concept, and the options (distractors) should be succinct.

4. Avoid the use of negatives in the stem (use only when you are measuring whether the respondent knows the
exception to a rule or can detect errors).

You can word most concepts in positive terms and thus avoid the possibility that students will overlook terms of "no, not, or least" and choose an incorrect option not because they lack the knowledge of the concept but because they have misread the stated question. Italicizing, capitalizing, using bold-face, or underlying the negative term makes it less likely to be overlooked.

The remaining five guidelines concern the choices from which students select their answer.

5. Have ONLY one correct answer.

Make certain that the item has one correct answer. Multiple-choice items usually have at least three incorrect options (distractors).

6. Write the correct response with no irrelevant clues. A common mistake when designing multiple-choice questions is to write the correct option with more elaboration or detail, using more words, or using general terminology rather than technical terminology.

7. Write the distractors to be plausible yet clearly wrong. An important, and sometimes difficult to achieve, aspect of multiple-choice items is ensuring that the incorrect choices (distractors) appear to be possibly correct. Distractors are best created using common errors or misunderstandings about the concept being assessed, and making them homogeneous in content and parallel in form and grammar.

8. Avoid using "all of the above," "none of the above," or other special distractors (use only when an answer can
be classified as unequivocally correct or incorrect).

All of the above and none of the above are often added as answer options to multiple-choice items. This technique requires the student to read all of the options and might increase the difficulty of the items, but too often the use of these phrases is inappropriate. None of the above should be restricted to items of factual knowledge with absolute standards of correctness. It is inappropriate for questions where students are asked to select "the best" answer. All of the above is awkward in that many students will choose it if they can identify at least one of the other options as correct and therefore assume all of the choices are correct - thereby obtaining a correct answer based on partial knowledge of the concept/content (Gronlund & Linn, 1990).

9. Use each alternative as the correct answer about the same number of times.

Check to see whether option "a" is correct about the same number of times as option "b" or "c" or "d" across the instrument. It can be surprising to find that one has created an exam in which the choice "a" is correct 90% of the time. Students quickly find such patterns and increase their chances of "correct guessing" by selecting that answer option by default.

-----------------------------
Checklist for Writing Multiple-Choice Items
* Is the stem stated as clearly, directly, and simply as possible?
* Is the problem self-contained in the stem?
* Is the stem stated positively?
* Is there only one correct answer?
* Are all the alternatives parallel with respect to grammatical structure, length, and complexity?
* Are irrelevant clues avoided?
* Are the options short?
* Are complex options avoided?
* Are options placed in logical order?
* Are the distractors plausible to students who do not know the correct answer?
* Are correct answers spread equally among all the choices?
(McMillan, 2001, p. 150)
----------------------

References (for full article)

Brown, F. G. (1983). Principles of educational and psychological testings(3rd ed.). New York: Holt, Rinehart
and Winston.

Cashin, W. E. (1987). Improving essay tests. Idea Paper, No. 17. Manhattan, KS: Center for Faculty Evaluation and
Development, Kansas State University.

Critical thinking rubric. (2008). Dobson, NC: Surry Community College.

Grading systems. (1991, April). For Your Consideration, No. 10. Chapel Hill, NC: Center for Teaching and Learning,
University of North Carolina at Chapel Hill.

Gronlund, N. E., & Linn, R. L. (1990). Measurement and evaluation in teaching (6th ed.). New York: Macmillan
Publishing Company.

Halpern, D. H., & Hakel, M. D. (2003). Applying the science of learning to the university and beyond. Change,
35(4), 37-41.

Isaac, S., & Michael, W. B. (1990). Handbook in research and evaluation.San Diego, CA: EdITS Publishers.

McKeachie, W. J. , & Svinicki, M. D. (2006). Assessing, testing, and evaluating: Grading is not the most important
function. In McKeachie's teaching tips: Strategies,research, and theory for college and university teachers (12th ed., pp. 74-86). Boston: Houghton Mifflin Company.

McMillan, J. H. (2001). Classroom assessment:
Principles and practice for effective instruction. Boston:
Allyn and Bacon.

Seymour, E., & Hewitt, N. M. (1997). Talking about leaving: Why undergraduates leave the sciences. Boulder, CO: Westview Press.

Svinicki, M. D. (1998). Helping students understand grades. College Teaching, 46(3), 101-105.

Svinicki, M. D. (1999a). Evaluating and grading students. In Teachers and students: A sourcebook for UT-Austin faculty(pp. 1-14). Austin, TX: Center for Teaching Effectiveness, University of Texas at Austin.

Svinicki, M. D. (1999b). Some pertinent questions about grading. In Teachers and students: A sourcebook for UT-Austin faculty(pp. 1-2). Austin, TX: Center for Teaching Effectiveness, University of Texas at Austin.

Thorndike, R. M. (1997). Measurement and evaluation in psychology and education. Upper Saddle River, NJ: Prentice-Hall, Inc.

Wiggins, G. P. (1998). Educative assessment: Designing assessments to inform and improve student performance. San Francisco: Jossey-Bass Publishers.

Worthen, B. R., Borg, W. R., & White, K. R. (1993). Measurement and evaluation in the schools. New York:
Longman. Writing and grading essay questions.(1990, September). For Your Consideration, No. 7. Chapel Hill, NC: Center for Teaching and Learning, University of North Carolina at Chapel Hill.

Cooperative Learning Groups

From the Teaching Effectiveness Program, University of North Carolina at Chapel Hill

Many teachers will occasionally break their classes into small groups for discussions, but only a few use the technique as a fundamental teaching tool. A class can be divided into learning teams that are periodically given instructional tasks to complete, either in or out of class. Research has shown that, with careful planning, this technique increases the efficiency and effectiveness of learning.

Groups of six or seven work best because this size is small enough for everyone to participate in problem-solving or debate, yet large enough for a spectrum of views to be represented. To work successfully, groups require a wide variety of viewpoints and intellectual skills, so it is important to make them as heterogeneous as possible. The individual data cards that you collect on the first day of class can yield important information about your students' backgrounds and preparation and make it easier to create heterogeneous groups. A professor of political science who uses long-term groups in his class tries to insure that each team has someone with a math background and at least one political science major. He creates groups with maximum diversity with respect to major, gender, race, and other characteristics.

The tasks that you assign for group work should challenge students to analyze phenomena, solve problems, apply theories, exercise judgment, or perform some combination of these activities. Clearly-written instructions are vital to the success of this kind of exercise, which means that the teacher must analyze the task carefully and break it down into its component parts. During the exercise, the teacher moves from group to group, answering questions, clarifying instructions, giving advice, and observing the group process. Group exercises can be designed for 15 to 20-minute periods, and need not consume an entire period.

In a well-designed group activity, there should be little need for direct intervention by the teacher. It is true that many teachers are uncomfortable with the loss of direct control that accompanies small-group work, but remember that you still govern the process and outcome by the instructions you provide for the groups. Small groups can be used with a variety of other techniques, such as peer teaching, case studies, and simulations; imaginative teachers are discovering new ways to use the technique every day. At UNC, many English composition instructors have successfully structured their courses around cooperative learning groups, and teachers in some high schools are using the technique in math classes.

Audiobooks Aid Challenged Readers

I read this encouraging article in the April/May 2007 issue of AudioFile Magazine. I am an avid audiobook reader on my daily commute and I was gratified to see that educators are finding that the introduction of audiobooks to challenged readers has demonstrated significant improvements in oral fluency, vocabulary, and comprehension.

An excerpt from Listen! It's Good for Kids by Susie Wilde:

"Children today are bombarded with visual images and may not have many opportunities to stimulate their auditory imagination. During Dr. Teri Lesesne's 2006 presentation on audiobooks for the American Library Association, she noted several benefits audiobooks offer such young listeners. 'Audiobooks can help start the movie in the head', she says, 'and allow children to form their own visual images'.

They also serve as models for oral fluency, building both vocabulary and comprehension. 'Kids often lack verbal endurance because they don't read enough. Listening to audio helps develop verbal endurance,' says Lesesne.

Series are a wise way to start for children listening to their first novels, because exposure to one often leaves a newly minted reader asking for more. Choose series that have stood the test of time, such as Beverly Cleary's Henry books or Jeff Brown's Flat Stanley series. Familiarity with a series often gives readers courage to branch out to new stories and unfamiliar characters.

In classrooms across America, teachers worry about ways to develop cultural literacy. Audiobooks help bridge cultural gaps and educate children about their own history. Parents and teachers can use them as departure for discussion. A great example is Blues Journey by Walter Dean and Christopher Myers, a father and son who share a love of blues that they express in pictures and words. Live Oak Media has paired the book with a CD read by Richard Allen that accents the rhythms and allows plenty of room to introduce blues recordings or to talk about how poverty, chain gangs, and persecution gave voice to some of the greatest blues music."

The article goes on to discuss the impact of audiobooks on ESL students:

"Those teaching ESL find that audio helps students pick up cues about phrasing, pronunciation, and intonation. Listening also improves concentration in ADD and ADHD children and can be of significant help to any learner who processes information more easily through listening than reading print. Students with learning disabilities benefit from the support of multi-sensory experience that provides auditory cues to aid in decoding written words."

(Click Chart to Zoom In)

Using Class Time Well: Active Learning

Using Class Time Well: Active Learning

from the Center for Teaching Excellence, Kansas University

Teaching is developmental rather than directive or presentational.

Active learning involves the implementation of “learning experiences in which the students are thinking about the subject matter” (McKeachie, 2002). It is based on the premise that students must do more than just listen to fully comprehend new information. They must read, write, discuss, and problem solve. By employing active learning in your classes, you will increase the effectiveness of your teaching and your students’ learning.

Reading

One suggestion for encouraging active learning is the use of the Treasure Hunt technique (Magnan, 1990). This strategy is based on the premise that if you’ve assigned a reading, there must be something valuable for the students in it. Choose several pages or sections, and then ask students to find the most important point, idea, or argument and write it down, along with a sentence or two justifying their selection. If you choose assigned passages well, you can increase understanding and participation immediately.

Writing

John C. Bean suggests several ways to incorporate writing into a class, including:

*Writing at the beginning of class to probe a subject: Ask students to write short answers to a question that reviews previous material or stimulates interest in what’s coming.

*Writing during class to refocus a lagging discussion or cool off a heated one: When students run out of things to say, or when a discussion gets too hot, ask students to write for a few minutes.

*Writing at the end of class to sum up a lecture or discussion: Give students a few minutes to sum up the day’s lecture or discussion and to prepare questions to ask at the start of the next class.

Discussing

While the most common approach to encouraging active learning in the classroom is the use of discussion, not all discussions are created equal, and there are other methods in which to achieve the difficult task of drawing students into lectures, discussions, and readings. One suggestion for engaging students in active learning is using the Thumb’s Up technique (Ukens, 2000). To implement this technique, ask students to form groups of six to ten people, with each group sitting in a circle. Participants will discuss topics within their groups. To keep everyone involved, each person is to extend his or her fist toward the middle of the circle. Once discussion begins, each member is to share, in any order, one idea or piece of information on the topic. As each person shares, his or her thumb will go up. A person may not share again until all thumbs are up. Then, members can begin again and continue the process. After about five minutes, stop the discussion. If you wish, repeat with new topics.

Another method to employ asks the students to frame the discussion, or determine the direction of the discussion. Ask students to identify one question from their readings that they would like to have answered in class. Ask them to share their question with three peers, and then have the group pick one of the three questions to present to the instructor. Allow each group to ask its question.

Problem-Solving

Invite students to use Think-Pair-Share: To help them better understand a lecture, stop for a moment. Ask students to think about a question or problem that relates to the lecture material, turn to a peer, and explain their answer or solution. This is a great way to apply and reinforce key ideas.

Resources:

Bean, J.C. (1996). Engaging Ideas: The professor’s guide to integrating writing, critical thinking, and active learning in the classroom. San Francisco: Jossey-Bass.

McKeachie, W.J. (2002). McKeachie’s Teaching Tips: Strategies, Research, and Theory for College and University Teachers. 11th Ed. Boston: Houghton Mifflin Company.

Magnan, R., Ed. (1990). 147 Practical Tips for Teaching Professors. Madison: Magna.

Stocking, S.H. et al. (1998). More quick hits. Bloomington, IN: Indiana University Press