Simple binary distribution

You often go out for dinner with your friend Joe. You both like different places, so to decide where to go, you both propose a place, then you flip a coin to choose which option. Joe is very superstitious, so you always use the same coin, which he got from his grandfather. Your method is that Joe flips, you choose. The problem with this is that you remember his grandfather once talking about a shady coin that comes out tails 60% of the time. Should you prefer heads or tails?

Which is better?

Just going with it is fine if you don’t care. Calculating the percentage of heads is a very nice in that it gives you a precise probability of getting a heads on a given toss. But it can take a while to stabilise (like 1000 tosses). The Bayesian approach allows you to be quite confident with a lot fewer tries - when starting from a prior of $50\%$ (i.e. you’re totally unsure which coin it is), you can get to over $90\%$ probability that the coin is fair or not with 100 tries. This is a lot faster than the frequentist approach. It’s also more flexible, in that it takes into account how much you initially trusted Joe. The problem with the Bayesian approach used here, is that it’s too simplistic. It only takes two possibilities into account - either the coin is fair, or it has a $60\%$ bias towards tails. What if the bias was $70\%$? The current approach doesn’t check for that, so it won’t tell you anything about it.

The frequentist approach tries to tell you something about the coin. It says something like “if you keep tossing the coin, $n\%$ will come up heads”. This is useful information about the world. On the other hand, the Bayesian approach used here is trying to help you work out “do I believe this coin is fair?”, and each subsequent toss should change your mind slightly.

One fun thing about the Bayesian approach, is that if you’re absolutely sure that either the coin is fair, or that it’s unfair (i.e. your prior that the coin is fair is $1$ or $0$), there is no way for you to change your mind. Let’s demonstrate this with the “I’m sure Joe is using a fair coin” version and getting a heads:

• $P(fair)$ - The probability the coin is fair - out prior here is $1$.
• $P(heads | fair)$ - The probability of getting a heads, if the coin is fair. This is $0.5$, sort of by definition.
• $P(heads | unfair)$ - The probability of getting a heads with an unfair coin. If we assume Joe’s grandfather was correct, this is $0.4$ as it should be tails $60\%$ of the time.
• $P(heads)$ - The overall probability of getting a heads. This is calculated as $P(heads | fair) * P(fair) + P(heads | unfair) * P(unfair)$.

So:

$$P(fair | heads) = \frac{0.5 * 1}{0.5 * 1 + 0.4 * (1 - 1)} = \frac{0.5 * 1}{0.5 * 1 + 0.4 * 0} = \frac{0.5 * 1}{0.5 * 1} = 1$$

Nothing has changed! Getting a tails will also give a posterior of 1! This means that no matter what evidence comes in, you’ll never change your mind on whether the coin is fair or not.