Professionals who are serious about such things accompany any numerical observation or prediction with a quantitative measure of its uncertainty. When done properly, this a range within which the true answer will be found with some specified probability, usually 95 percent. The technical term for such a range is credible interval.
The calculator on this page will take observed frequencies as input and estimate robust credible intervals for the associated probabilities. It assumes that these probabilities are completely unknown with no prior preference for some values over others. The (Bayesian) methodology used is state-of-the-art. Not only that, but you can almost see it happening. How it works is really interesting.
What makes MCMC special is that the walker moves from one state to another at a rate that depends on the relative probability of the two states. If the proposed move is too improbable, the walker will stay put for that iteration. Eventually, the simulation reaches equilibrium with the rate of entering any state equal to the rate of leaving it. From that point on, the probability that a state wil be visited will be equal to the inherent probabiity of that state and the record of the walker's subsequent itinerary (trace) will contain all that is knowable about the unknown(s), given the model and the data. A plot of that portion of the trace corresponding to any particular unknown (here, a probability) is called the marginal for that unknown from which a credible interval can be easily extracted.
There are, of course, some technical details, for instance, how a proposal for the next state is chosen. This calculator utilizes a strategy known as Metropolis sampling. There are other details as well all of which have been optimized here for these two types of problems.
Bear in mind that MCMC is a stochastic process and will not give exactly the same answer every time or be as precise as a closed-form, analytical solution would be. One example: If you observe 3 successes out of 10 tries, the 95-percent credible interval for Prob(success) will be (0.093, 0.588), on average (cf., the binomial calculator here).
The walker might take a few seconds to complete its journey, especially since only every tenth iteration is saved in the sample (to avoid autocorrelation). There is a status indicator to show progress.
Note that an observed frequency of zero will result in limits that include zero and/or one exactly. Precision increases with sample size, on average, but extremely large frequencies might require more precision than this simple calculator can deliver.
There are several good introductions to Bayesian inference and MCMC as well as software that is far more powerful than this calculator, e.g., Data, Uncertainty and Inference and MacMCMC, resp., available from the causaScientia home page—both free, of course.
Comments and suggestions