I aim to commit statistical sin. I’m going to accept the null hypothesis for no other reason than because I “failing to reject it”. Having tarnished my reputation with that, I’ll finish by ignoring the only data available and base everything on non-informative priors which prominent authorities assure us don’t even exist. Let the debauchery begin.
Consider a typical classroom coin flip experiment. The teacher does no physics and takes no measurements. Instead they flip a coin 100 times, observe the frequency of heads , and use it to create a 95% probability interval for the frequency in the next 10,000 coin flips. Barring some bad luck, the utility of Statistics will be confirmed for the students when they see their frequency is in the interval predicted.
After the first 100 flips we get . Using the binomial model we’d fail to reject with a p-value of .2. But as statistical harpies remind us constantly, this is not the same as accepting p=.5, and moreover, probabilities are to be equated with frequencies.
So should the observed frequency .46 be used to construct the 95% interval instead of .5?
Hell no! Remember we’ve done no physics here. So we have no idea which element of we’ll see in the next ten thousand flips. What we do know is the binomial calculation implies the following:
Since the sequence the class is about to observe is going to be in somewhere, using dramatically increases the opportunities for the demonstration to fail. Those impressionable students may conclude statistics is a waste of time after all.
Frequentists think the cold hard facts of frequencies trump any Bayesian philosophical nuances. In their mind if frequencies f do exist in a problem, then p=f and it’s all over for Bayesians but the crying. Examples like this previous post showing otherwise seem to have no effect on them.
Here we have a different example. The only data on the coin is the frequency of heads. Yet we’d be damned fools to equate it with the probability of heads and are better off using a value based on a data-free ignorance prior over . Frequentists in practice aren’t unwise enough to follow their own philosophy or texts. They’d quietly drop the data and subversively accept the null like the rest of us rapscallions.
Perhaps their use of p=.5 comes from their magical ability to intuit that each element of would come up equally often over flips. How they stumbled upon this curious “fact” remains a mystery. Maybe it was written on the back of the Ten Commandments.
Lucky for them they never try this legerdemain using . They’d fail to reject the null with a p-value of .38, but since the 95% interval constructed from p=.48 is only consistent with 2% of they’d have some explaining to do unless they got very fortunate.
If Statistics can get this knotted over the binomial distribution, then how could it ever be untangle in real applications? The mess is avoided entirely by understanding what probabilities really are. Not only are they conceptually different from frequencies, they’re usually unequal even when all we have to go on is frequencies.
If we are ignorant as to which element of will show up, then we’d better only make predictions consistent with the vast majority of . That’s all the Bayesian non-informative prior and procedure achieves. For the life of me I can’t figure out why this is so hard to understand or why so many think it’s metaphysical nonsense.
UPDATE: Just to emphasize, accepting is an extremely good idea even though it has a p-value .2. Accepting is a very bad idea even though it has a p-value .38.