More from statisticla rethinking. I thought the below was an interesting section, and the last part reminded me of Paul Nurse’s Nature comment:
“Rethinking: Why statistics can’t save bad science. The vampirism example at the start of this chapter has the same logical structure as many different signal detection problems: (1) There is some binary state that is hidden from us; (2) we observe an imperfect cue of the hidden state; (3) we (should) use Bayes’ theorem to logically deduce the impact of the cue on our uncertainty.
Scientific inference is often framed in similar terms: (1) An hypothesis is either true or false; (2) we use a statistical procedure and get an imperfect cue of the hypothesis’ falsity; (3) we (should) use Bayes’ theorem to logically deduce the impact of the cue on the status of the hypothesis. It’s the third step that is hardly ever done. But let’s do it, for a toy example, so you can see how little statistical procedures — Bayesian or not — may do for us.
Suppose the probability of a positive finding, when an hypothesis is true, is Pr(sig|true) = 0.95. That’s the power of the test. Suppose that the probability of a positive finding, when an hypothesis is false, is Pr(sig|false) = 0.05. That’s the false-positive rate, like the 5% of conventional significance testing. Finally, we have to state the base rate at which hypotheses are true. Suppose for example that 1 in every 100 hypotheses turns out to be true. Then Pr(true) = 0.01. No one knows this value, but the history of science suggests it’s small. See Chapter 17 for more discussion. Now use Bayes’ to compute the posterior:
Pr(true|pos) = Pr(pos|true) Pr(true) = Pr(pos|true) Pr(true)
Pr(pos) Pr(pos|true) Pr(true) + Pr(pos|false) Pr(false)
Plug in the appropriate values, and the answer is approximately Pr(true|pos) = 0.16. So a positive finding corresponds to a 16% chance that the hypothesis is true. This is the same low base-rate phe- nomenon that applies in medical (and vampire) testing. You can shrink the false-positive rate to 1% and get this posterior probability up to 0.5, only as good as a coin flip. The most important thing to do is to improve the base rate, Pr(true), and that requires thinking, not testing”
