By Dr. Gary Deel, Ph.D., J.D.
Faculty Director, School of Business, American Public University

This is the second article in a three-part series about quantitative assessments for graduate students.

In the first part of this article, I discussed how and why quantitative assessments are often dismissed by graduate programs as insufficiently challenging. In this second part, I’ll articulate how law schools and state bar examiners create rigorous and challenging quantitative exam questions with the use of several key strategies.

Start a degree program at American Public University.

How Law Schools and State Bar Examiners Create Challenging Multiple-Choice Questions

In law schools and on state bar exams, multiple-choice questions are designed with several strategies in mind, in order to make them even more difficult than if the questions had been asked without answer prompts. First, each question is actually a series of two or more nested questions. For example, consider a law school question that asks, “If you pull out a gun and demand that an old lady give you her purse, have you committed a robbery or a theft?”

At first glance, this law school question might appear to have a simple answer. But the answer options will force more thought and analysis.

For instance, answer choices to this type of multiple-choice question might be:

A) Robbery, because threatening someone with a gun constitutes a robbery.

B) Robbery, because robbery is distinguished from theft by use of force.

C) Theft, because a theft can occur without knowledge by the victim.

D) Theft, because theft is distinguished from robbery by use of force.

E) None of the above are correct.

Notice that there are two answer choices for “robbery,” two answer choices for “theft” and a “none of the above” option. Consequently, law school students must not only decide whether the scenario is a theft or robbery, they must also decide the reason for the answer.

In addition, those law school students must keep in mind that it is possible that none of the answer options provided are correct. By writing a multiple-choice question this way, instructors are actually asking two questions at the same time and reducing the opportunity for a student to eliminate other answers through deductive reasoning.

Some Multiple-Choice Questions Are Designed to Require Students to Provide the Best Possible Correct Answer

A second strategy employed by law schools and state bar examiners is to require students to provide not just a correct answer, but the best correct answer. Review again the answer examples provided for the robbery question above.

First, any law school student or bar examinee worth his salt should know that the scenario provided is in fact a robbery, so we can rule out options C and D. Note, however, that the reason provided for C is actually a correct statement. Theft can in fact occur without the knowledge of the victim.

So half of answer C is correct. This choice is written this way as a “red herring” to cause students to question what they think they know about this subject in yet another level of challenge.

But now we are down to answers A, B or E. The problem with E, the “none of the above” option, is that it affords us no ability to deduce our answer. Option E allows for literally anything else in the known universe to be the correct answer.

This “none of the above” option effectively converts a multiple-choice question into an infinite choice question. The latter type of question is always more challenging than the former.

But we do know that if E is to be the correct answer, then both A and B must be incorrect answers. As a result, we must analyze and rule out those answers if we can.

For instance, Option A states that the answer to the question is “Robbery, because threatening someone with a gun constitutes a robbery.” This statement is factually true. So we have a correct first answer — the scenario provided is in fact a robbery — and it is followed by a factually true second statement.

Let’s look at option B. Option B correctly states that the scenario is a robbery, and the reason given is “because robbery is distinguished from theft by use of force.” This is also a factually true statement.

Now we have two correct answers with two factually true statements as supporting reasons. At this point, we may be able to rule out Option E, but between A and B, which is the best correct answer?

To work out the best possible answer, we have to look at the call of the question. What is the question really asking us? The question asks us to determine whether we’ve committed a robbery or a theft. So the best answer to the question would ostensibly be an answer that makes a distinction between robbery and theft, if that type of answer is available.

In option A, it’s true that threatening someone with a gun is a robbery. But that fact doesn’t really answer the question; it just reiterates the facts stated in the question.

As an analogy, let’s say that someone asked you if apples are red or blue, and you responded that “they’re red because they’re apples.” You may be correct but the reason doesn’t really support the answer you give.

By contrast, option B gives the correct answer and explains why the scenario is a robbery and not a theft. Therefore, while both A and B are factually correct statements, option B is the best correct answer in response to this question.

Obviously, multiple-choice questions that are strategically designed can be even more challenging than a free-form answer. For example, if an exam asked students whether the scenario was a robbery or theft and just left a blank space for students to write in their answer, students could simply write “robbery” and move on without any further critical thinking.

To be fair, “why” could also be included in the question’s wording. But then students might respond with a tangential answer such as “because it is violent” and force us as test assessors to deliberate over how many points such a response is worth.

By requiring students to consider both the answer and the reason for the answer, and by requiring that students scrutinize more than one factually correct option, the multiple-choice design demands a higher level of rigor.

How might such strategies be applied to a non-legal curriculum? In the last part of this article, I’ll demonstrate what a non-law exam question might look like using law school exam question strategies.

About the Author

Dr. Gary Deel is a Faculty Director with the School of Business at American Public University. He holds a J.D. in Law and a Ph.D. in Hospitality/Business Management. Gary teaches human resources and employment law classes for American Public University, the University of Central Florida, Colorado State University and others.