The teachings of Jaynes-sensei

I’ve recently been trying to get through Probability Theory: The Logic of Science, which has come highly recommended, and with which opinion I so far wholeheartedly concur. It generally explains why Bayes was right, starting with a couple of basic desiderata, i.e.:

• degrees of Plausibility are represented by real numbers
• qualitative Correspondence with common sense
• that it always reasons consistently

It turns out that these are totally sufficient to come up with a system that encompasses both probability and logic (as a binary subset of probability). From these basic assumptions, Jaynes proceeds to build up the known rules of probability, to thoroughly dis frequentialists, and to go on to show that Bayes’ rule is pretty much all you need for most scientific work, as long as you actually think through all the appropriate implications. It assumes that the reader is fluent in mathematics (i.e. basic calculus and algebra), but I personally was able to follow virtually all the proofs, so it can’t require that much.

Wonders and signs

For a while I’ve wanted to explain to people why the testimony of the apostles isn’t sufficient for me to believe them. I intuitively find that the gospels aren’t enough for be to believe them. Even if they’re better preserved than any other ancient text etc. I usually go with extraordinary claims require extraordinary evidence, which is a good heuristic and is convincing to me, but I didn’t have a good mathematical explanation of why it’s so. I could come up with various reasons of why it’s a good idea, but not a proof. It turns out that this is just yet another result of Bayes’ rule…

Telepathy

Dr Samuel Soal was a parapsychologist who did a number of ESP experiments between 1941-1943 to discover whether it was possible to read someones mind. The basic setup was an agent with a set of 5 animal cards which were shuffled before each test. The agent would pick one at random, and the subject would try to guess what card was picked. Naively, one would expect the subject to be right around once in 5 trials, i.e. p = 0.2. This is not what Soal found. A Mrs Steward, who was the test subject, managed to correctly guess the card 9410 times out of 37100 trials (I’m impressed by her patience…), which is a success rate of f = 0.2536. This is 25.8 standard deviations away from the expected success rate, which is a big deal. So we have:

• $P(H_0|X)$ - the null hypothesis that she should have random results with a success rate of 0.2
• $P(H_f|X)$ - an alternative hypothesis that she is an ESPer if the success rate is 0.2535

Plugging the values into the appropriate equations (check page 121 of the book), it turns out that the probability of getting this data with the null hypothesis is some $3.15 \times 10^{-139}$. So it appears that the only thing left is to accept that ESP is real, given the following equation:

$P(H_f|DX) = P(H_f|X) {P(D|H_fX) \over P(D|X)}$, where $P(D|X) = P(DH_0|X) + P(DH_f|X)$

Since $P(D|H_0X)$ is absurdly small, this can be approximated as $P(H_f|DX) = {P(H_f|X) P(D|H_fX) \over P(H_f|X) P(D|H_fX)} = 1$, since $P(H_0|DX) \approx 0$. Not very helpful. This pretty much says that if your prior of $P(H_f|X)$ isn’t too small, you’re going to end up with Objective Scientific Evidence that ESP is real.

Getting real

The obvious problem with the above calculations is that those aren’t the only two possible hypotheses. An alternative would be that they cheated. Or that the cards weren’t shuffled properly. Or a mistake was made while writing down the answers. Or that the agent subconsciously reacted differently to one of the cards (this one would actually be quite interesting). This changes the basic equation from

$P(H_f|DX) = P(H_f|X) {P(D|H_fX) \over P(D|X)}$, where $P(D|X) = P(DH_0|X) + P(DH_f|X)$

to

$P(H_f|DX) = P(H_f|X) {P(D|H_fX) \over P(D|X)}$,

where

$P(D|X) = P(DH_0|X) + P(DH_f|X) + \sum_{a=0}^n P(DH_a|X)$, $n$ is the number of alternatives.

This means that for $P(H_f|DX)$ to have a high value (i.e. close to 1), $P(D|X)$ has to be low, or rather to be massively dominated by $P(DH_f|X)$. Which in turn depends on your priors and the relative likelihood of all alternatives. If you assume that there are some 100 plausible alternatives which are all more likely than ESP, then they quickly eclipse the evidence for $H_f$. Even if a given alternative is shown to be invalid, it’s probability mass will simply get divided among all the alternatives, which won’t change much (unless you can exclude all the possible alternatives, of course). Striking down one alternative will just raise a different one that is more likely on your priors. Not that this is impossible - meteorites were once viewed as superstitious bunk until overwhelming evidence appeared for the contrary.

Lessons

• Soal cheated
• Read the book
• Writing stuff down is an excellent way to learn (this helped me a lot)
• Lee Strobel isn’t convincing since he has different priors
• AI risk and other outlandish claims are unbelievable because people have different priors