Dr. Mayo responded to criticism of the Severity Principle here. The main points are (A) if SEV differs from Bayes it doesn’t mean SEV’s bad (B) you shouldn’t compare SEV and Bayes because they do different things (C) A prior can always be invented which allows Bayes to replicate Frequentists success, but that’s a meaningless attempt by Bayesians to save face, (D) it’s the philosophy of the Severity Principle which matters more than the numbers.
Taking these one at a time:
(A) The example in the original post showed is well warranted according to Error Statistics principles despite the data trivially showing the true value is standard deviations below .1. This means SEV contradicts the data. It’s irrelevant what the Bayesian answer is. The best Bayesian inference could have been “Martians smell like elderberries”, but that failure wouldn’t change the fact that SEV is wrong.
(B) Dr. Mayo claimed SEV provides a method for judging the size of the discrepancy from the null. I pointed out that Laplace achieved that same goal, in that same problem, for the same example, with the same mathematics 200 years ago. It’s disingenuous in the extreme to suggest SEV and Bayes aren’t comparable.
(C) Reminding everyone that Laplace got Mayo’s specific solution 200 years ago is not an example of Bayesians fudging priors to copy Frequentists’s successes. Laplace did not in fact have a time machine. He didn’t travel 200 years into the future to steal the mathematical stylings of one Dr. D. G. Mayo, and then return to fraudulently pass them off as Bayesian.
(D) Let’s examine the philosophy of the Severity Principle a little. It’s clear that some predictive success’s are more cogent then others. A theory that derives a known answer isn’t as impressive as one that correctly predicts an unknown result. In turn, predicting an unsurprising result is less impressive than predicting something previously thought wrong. Everyone from plumbers to Quantum Field Theorists use such judgments all the time.
But that’s talking about “experimental tests” not “data analysis”. Despite loaded terms like “hypothesis testing”, data analysis isn’t “testing” in the way “the germ theory of disease was verified in the laboratory” is.
Different test statistics applied to the same data aren’t different “tests” of the hypothesis. Rather they’re drawing conclusions from different sub-portions of the information inherent in the data. Since different sub-portions can point in different directions, it’s important to use all the information in the data. So while the goal of experiments is to subject theories to trial by fire, the goal of data analysis is to extract the most truth possible out of the data.
Error Statisticians may want to deny that characterization, but there’s one problem. As hinted at in another post, this misunderstanding allows us to create examples where two perfectly legitimate SEV analyses using different test statistics, but relying on the same data, assumptions, and even the same “informational content” in some sense, find strong support for two contradictory hypotheses.
Furthermore, such examples can be made symmetric in every respect, so which test statistics are chosen (and thus which conclusions are supported) is the merest whim. This surely makes Error Statistics the most radically subjective theory of inference ever put forth.
Experiments test theories; data analysis extracts information. They’re related but not the same.
I don’t want to dissuade anyone from reading Mayo. If a Bayesian of my temperament can get inspiration and insight from her work, then anyone can. Human nature is like that sometimes. Many of the greatest advances in the mathematical sciences were the result of Euler and the Bernoulli’s repairing problems in Newton’s Principia. But in the end, there has to be a cold hard accounting of what Newton got right and what he didn’t. What applies to Newton surely applies to rest of us slobs. So when that accounting is done for Error Statistics, “SEV” goes in the “wrong” column.