Win, Lose or Draw?

MCMC in Action

It is difficult to make predictions,
especially about the future.
-- Niels Bohr

They don't give Nobel Prizes to just anybody but they gave one to Niels Bohr. He was someone who really deserved it. And yet, as a scientist, he knew better than most that our knowledge is imperfect. Knowledge comes from observing the world around us but any finite set of observations contains only a finite amount of information, not enough to yield an answer with infinite precision. Consequently, unknowns never become perfectly known; we are always left with some uncertainty.

Professionals who are serious about such things accompany any numerical observation or prediction with a quantitative measure of its uncertainty. When done properly, this a range within which the true answer will be found with some specified probability, usually 95 percent. The technical term for such a range is credible interval.

A Simple Example

Predictions are simplest when they are limited to just two or three discrete possibilities, e.g., True/False (binomial) or Win/Lose/Draw (trinomial). Then, to make a prediction about a future outcome, you need to know the probability for each possible outcome. These, however, cannot be observed; they must be inferred. This is where uncertainty comes in. Estimating a probability depends on the amount of information in the sample of observations which, in turn, depends on the sample size—the bigger the sample, the more the information (other things being equal).

The calculator on this page will take observed frequencies as input and estimate robust credible intervals for the associated probabilities. It assumes that these probabilities are completely unknown with no prior preference for some values over others. The (Bayesian) methodology used is state-of-the-art. Not only that, but you can almost see it happening. How it works is really interesting.

Methodology

Clicking Run in the dialog box below launches a type of simulation known as Markov-Chain Monte Carlo (MCMC) in which a simulated “walker” meanders throughout the parameter space visiting different states (parameter vectors) as it travels along. Typically, there are an infinite number of states. If the model chosen for the problem is valid, then one of these states will be the true answer. For a binomial situation, the parameter space is unidimensional, containing all possible values for Prob(True). Knowing Prob(True) is sufficient since the Sum Rule of probability says that Prob(False) = 1 - Prob(True). In the trinomial case, the space is two-dimensional for the analogous reason.

What makes MCMC special is that the walker moves from one state to another at a rate that depends on the relative probability of the two states. If the proposed move is too improbable, the walker will stay put for that iteration. Eventually, the simulation reaches equilibrium with the rate of entering any state equal to the rate of leaving it. From that point on, the probability that a state wil be visited will be equal to the inherent probabiity of that state and the record of the walker's subsequent itinerary (trace) will contain all that is knowable about the unknown(s), given the model and the data. A plot of that portion of the trace corresponding to any particular unknown (here, a probability) is called the marginal for that unknown from which a credible interval can be easily extracted.

There are, of course, some technical details, for instance, how a proposal for the next state is chosen. This calculator utilizes a strategy known as Metropolis sampling. There are other details as well all of which have been optimized here for these two types of problems.

Bear in mind that MCMC is a stochastic process and will not give exactly the same answer every time or be as precise as a closed-form, analytical solution would be. One example: If you observe 3 successes out of 10 tries, the 95-percent credible interval for Prob(success) will be (0.093, 0.588), on average (cf., the binomial calculator here).

Calculator

The MCMC process is carried out starting with the dialog box below.

The walker might take a few seconds to complete its journey, especially since only every tenth iteration is saved in the sample (to avoid autocorrelation). There is a status indicator to show progress.

Instructions:

Enter either binomial or trinomial frequencies (integers) in the fields provided, leaving the alternate set empty.
Choose the sample size.
Click Run.
When the run finishes, click Results.
For another run, click Reset.

Observed Frequencies
True	False

Win	Lose	Draw

Sample size:
50,000 100,000 200,000
Progress:

Results

A plot of the relevant marginals is shown below (as raw histogram envelopes) along with selected credible intervals.

Credible Intervals

The 90-, 95- and 99-percent credible intervals, rounded to five decimal places, are listed below. They are determined by finding the shortest marginal range that contains the specified fraction of visited states.

Note that an observed frequency of zero will result in limits that include zero and/or one exactly. Precision increases with sample size, on average, but extremely large frequencies might require more precision than this simple calculator can deliver.

Unknown	lo99	lo95	lo90	hi90	hi95	hi99

Even Better

The analysis problem addressed here is trivial and does not provide even a hint of the power of MCMC. With more difficult problems, MCMC becomes the only practicable way to get the corrrect answer and associated uncertainties.

There are several good introductions to Bayesian inference and MCMC as well as software that is far more powerful than this calculator, e.g., Data, Uncertainty and Inference and MacMCMC, resp., available from the causaScientia home page—both free, of course.

Comments and suggestions