Make Decisions with Sequential Observations: A Simulation

Theodore W. Frick

Copyright 2005 (revised June 27, 2012)

This simulation illustrates how to make a specific decision when making sequential observations. It employs a method of reasoning that is based on Bayes' Theorem and Abraham Wald's Sequential Probability Ratio Test. These methods have been used in computer adaptive testing for determining mastery or nonmastery of educational objectives (e.g., see Frick, 1992; Welch & Frick, 1993), and more recently in computerized classification testing. These methods have also been used in quality control in manufacturing and other settings where sequential sampling is both practical and a more efficient way to reach a specific decision.

You can play with this simulation to see how decisions can be made with uncertainty, so you'll know the chances that you have made the right decision for a specific case. Feel free to experiment, to see what happens under different assumptions and outcomes of observations. You might be surprised to discover how few observations are needed to reach a decision under some conditions, and how many are needed for others.

Just to make this concrete, let's say you are an algebra teacher and you have a student named Sam. You want to know whether Sam can solve quadratic equations. If he can, then you don't need to teach him quadratic equations, and he could spend his time learning something new. Otherwise, you will need to teach him. So the decision has consequences for both Sam and you his teacher.

As another example, suppose you are a parent, you want to know whether your teenage daughter, Susie, can drive a car safely. If she can, you'll let her drive your car. Othewise, you won't. There are big consequences here. If Susie is not able to drive safely, she could kill or injure herself or others if she has an accident. It's your car and you need to decide if you will let her drive it by herself.

In both examples, you need to make a decision about a specific case and choose between 2 alternatives: whether or not to teach Sam quadratic equations, or to let Susie drive your car.

1. Enter labels for two mutually exclusive alternatives, one of which you need to choose--based on observations you will make.

Which of these is true? We assume that both alternatives cannot be true at the same time. For example, either Sam can solve quadratic equations, or he cannot. One or the other is true, but we don't know which is the case.

The goal of sequential sampling, in which you evaluate the result each observation, is to choose Alternative A or B as soon as possible by making the minimum number of observations necessary, while at the same time minimizing the chances that you have drawn the wrong conclusion.

If you are already highly certain about which of the alternatives is true, you have sufficient data to support your conclusion, and you have no reason to believe that things have changed, then you don't need to inquire further. This decision method assumes that you don't know for sure which alternative is true, or you suspect something may have changed and you need to investigate further.

2. What Is the Expected Success Rate under Each Alternative?

Let's say you are clearly willing to conclude that Sam can solve quadratic equations if he does so successfully at least 97 percent of the time.

Furthermore, you are clearly willing to conclude that Sam cannot quadratic equations correctly if he does so 80 percent of the time or less.

The gap between these two boundaries indicates where you are "sitting on the fence" and not willing to choose either alternative. This is what Abraham Wald called the "zone of indifference."

How do you choose values for Alternatives A and B? It depends on what you, the decision maker, are clearly willing to accept as a success rate rate for each alternative. As an example, for Alternative A above, 97 percent is the lower bound. Anything between a 97 and 100 percent success rate means that you will choose A (Sam can solve quadratic equations).

For Alternative B above, 80 percent is the upper bound for choosing it. Anything between a 0 and 80 percent success rate means that you will choose B (Sam cannot solve quadratic equations).

Any success rate between the two boundaries is your zone of indifference (81 to 96 percent). In this case, you are not willing to decide whether Sam can or cannot solve quadratic equations.

3. What Is the Expected Error Rate for a False Conclusion?

Let's say you are willing to be wrong 1 percent of the time about Sam (or any other student) by concluding that he can solve quadratic equations, when in fact he cannot.

The other kind of error is to conclude that Sam cannot solve quadratic equations correctly when in fact he can. How often are you willing to do this? Let's say you are willing to be wrong 5 percent of the time.

This simulation will help you make a decision, based on the information you provided above. Feel free to experiment by changing the names and numbers in the boxes above if you want different alternatives, success rates and error rates.

But there is one more thing you need to do: make observations. When you start observing, I assume that you have no previous evidence, and so the prior probabilities are equal. As you make observations, those prior probabilities will change as you learn more.

See also a related research study. Send questions or comments to Ted Frick.