The Amelioration of Uncertainty

## Integrity and the Jaynesian/Frequentist divide.

The heart of the Jaynesian/Frequentist divide can be found in the post “The Definition of a Frequentist”. A Frequentist views weather prediction as one of getting the frequency of weather events right. A Jaynesian’s goal is to pin down the one-off true weather sequence as tightly as possible. I illustrated this difference in a way so clear it’s easy to see which is better. I got this example from Jaynes:

THE WEATHERMAN’S JOB
In a certain city, the joint frequencies of the actual weather and the weatherman’s predictions are given by:

Rain | predict Rain = 25%
Shine | predict Rain = 50%
Rain | predict Shine = 0%
Shine | predict Shine = 25%

An enterprising fellow trained in orthodox statistics (but not in meteorology) notices that, while the weatherman is right only 50% of the time, a prediction of ‘shine’ everyday would be right 75% of the time, and applies for the weatherman’s job. Should he get it? Which would you rather have in your city?

The weatherman is delivering useful information at a rate I=(entropy of distribution of predictions)+(entropy of actual weather distribution)-(entropy of joint distribution) = (0.562+0.562-1.040)/ln2 = 0.123 bits/day. As explained previously (Jaynes, 1968) this means that in the course of a year the weatherman’s information has reduced the number of reasonably probable weather sequences by a factor of W=exp(0.123x365xln2)=2.92×10^{13}. With the weatherman on the job, you will never be caught out in an unpredicted rain; with the orthodox statistician this would happen to you one day out of four.

As this example once more forces one to recognize, the value of an inference lies in its usefulness in the individual case, and not in its long-run frequency of success; they are not necessarily even positively correlated. The question of how often a given situation would arise is utterly irrelevant to the question of we should reason when it does arise. I don’t know how many times this simple fact will have to pointed out before statisticians of ‘frequentist’ persuasions will take not if it; but I think it important that we keep trying.

The key is to forget about frequencies and imagine the distribution as a way to specify the true weather in that space of possible sequences. Conceptually, it’s no different than the way most Bayesians think of a prior probability for a fixed parameter. From the post:

is preferred by Frequentists because = “frequency of rain”, while has no frequency interpretation but merely locates the true weather sequence better. If we predict rain whenever the odds favor it however, is wrong 5/10 days while always gets it right.

So do Frequentists really believe is superior to or that we should forget meteorology and just always predict sunshine?

I’m afraid so. None other than the high priestess of Frequentist Statistics herself, Dr. Mayo, has repeatedly commented on this passage in print and her blog. See here, here, here, here, here, here:

…and no matter how intimidating the rhetoric of prominent Bayesians is (e.g., E. T. Jaynes, an objective Bayesian):

The question of how often a given situation would arise is utterly irrelevant to the question of how we should reason when it does arise. I don’t know how many times this simple fact will have to be pointed out before statisticians of “Frequentist” persuasions will take note of it. (Jaynes 1976, 247)

What we error statisticians must rightly wonder is how many times we will have to point out that to us, reasoning from the result that did arise is crucially dependent upon how often it would arise.

Mayo finds Jaynes’s point so absurd it doesn’t need analysis, only a gleeful counter slogan. Some less charitable souls have suggested that Mayo didn’t understand his mathematics, but Mayo insists she does. Moreover, Dr. Mayo takes interpreting the works of others very seriously:

…there’s a difference between someone who is found fallible and someone who shows a serious and persistent lack of integrity when it comes to treating and interpreting the work of others.

Someone of Mayo’s integrity wouldn’t criticize a passage she didn’t understand, and then repeat her error constantly. We must assume Mayo did understand Jaynes, she just believes Frequentist statistics implies it’s better to be half-wrong than all-right and meteorology is a waste of time.

Sometimes I think my Marines were right about philosophy: it’s all Error and no Statistics.

September 23, 2013
• September 23, 2013Rasmus Bååth

Bayesianism is often charaterized a little bit as a religous movement (http://normaldeviate.wordpress.com/2013/09/01/is-bayesian-inference-a-religion/) but sometimes I feel that frequentism is more of the religion.

* They (blindly) follow teachings in an old book (http://en.wikipedia.org/wiki/Statistical_Methods_for_Research_Workers).
* Tests are carried out like sacred rituals and p-values are magical.
* Their teaching is not coherent, as you have pointed out.
* It doesn’t help if one shows that the frequentist teaching is incoherent. Their prior is too strong…

• September 23, 2013Joseph

Rasmus,

I disagree. I don’t think Frequentism is anything like a religion. Your first two bullet points aren’t true. The later two are true, but not because Frequentists believe things on faith, it’s because it’s difficult for anyone accustomed to think of probabilities as frequencies to think of them in any other way.

That failing isn’t unique to Frequentists; most Bayesians are the same way in my experience and most don’t understand what Jaynes was talking about in that passage either.

• September 23, 2013Jake

Somehow that Goofus-and-Gallant weather prediction example gets a little less compelling if you flip “Rain” and “Sun”.

• September 23, 2013Joseph

Jake,

Abstractly stated, the goal of assigning a (non-frequency) probability distribution over is to define a region which describes the location of .

The distribution is then used to calculate which conclusions are true for almost every element of . Since our best guess would be that those conclusions hold true for as well. This guess will tend to be highly robust for obvious reasons.

In this case the goal of weather modeling is to get a which includes the actual weather sequence in its high probability manifold and to make the size as small as possible.

That was the point of Jaynes’s entropy calculation . He was showing that the weatherman’s predictions were locating the true weather in a space whose size over a year was reduced by a factor of because of the meteorologist’s modeling, which is why their forecasts were so useful despite being seemingly “worse” by Frequentist standards.

It’s up to you to use whatever information you have to ensure . If you really don’t know much about then choose a so spread out that that nearly equals the entire space and you’re off and running.

But if you screw this up and fail to make , then your “probability” calculations showing which conclusions are true for almost every element of aren’t of much use.

So yes, failing to get things right () likely makes them wrong.

• September 24, 2013Joseph

Also, Jake, that example was deliberately simple. There are two questions here: (a) what’s our goal?, and (b) how do we reach it?

There is very serious confusion about the answer to (a). Go back and look at that Wasserman quote. Even a highly respected mathematical statistician like him is so unshakably sure the goal is frequentist calibration that he believes if Bayes Theorem doesn’t deliver that calibration Bayesian’s will drop Bayes instantly.

But frequentist calibration is never the goal. It’s occasionally a byproduct of the real goal, and it’s sometimes (but no where near as much as most people think) part of one valid strategy for achieving , but it’s never, ever, the goal.

P.S. I understood your comment, but it sounded at first a bit like “Sure Mr. Newton, your law of gravitation can be used to predict the motion of the planets better than epicycles, but what if you interchanged the sun and the moon? What happens to your fancy predictions then?”

• September 24, 2013Daniel Lakeland

Ultimately the point is that the goodness of a prediction is relative to the consequences for being right vs wrong, not relative to the frequency of being right vs wrong. Most people dislike being caught unprepared in the rain for long periods of time, so in this case, the errors in the weatherman’s predictions are of the right type. If you switch the labels, everything is symmetric except the consequences, instead of being rarely caught unprepared in the rain, you’re rarely caught unprepared in the sun.

In that sense, Jake’s comment is important since it points out that what we really care about is decision making/consequences not frequency calibration.

• September 24, 2013Joseph

Daniel,

Of course you’re right that in real life there is a big decision theoretic component to weather predictions. The loss function isn’t symmetric.

But even before that, as Jaynes showed explicitly, if is a sequence of weather outcomes, then the weather bureaus’ physics models are reducing the number of probable sequences (the size ) by a factor of .

So when you go to predict any quantity of interest , which may be a loss function but could be anything, you’re effectively only considering a fraction of the ‘s and thus can get less uncertainty in your estimates for .

In other words, accuracy in a Frequentist calibration sense is irrelevant. The only kind of accuracy you need is to pin down in the space of all weather sequences with as little uncertainty as possible (i.e. using a non-frequency which has as small an entropy as possible, but in which is still in the high probability manifold).

It’s surprising to anyone who’s drunk the Frequentist cool-aid just how different these notions of accuracy are in practice, but they really are completely different. That was the point Jaynes was lamenting frequentists weren’t understanding in his last paragraph.

• September 24, 2013Anon

“The weatherman is delivering useful information at a rate I=(entropy of distribution of predictions)+(entropy of actual weather distribution)-(entropy of joint distribution) = (0.562+0.562-1.040)/ln2 = 0.123 bits/day.”

Alright, I have never thought of it like this. Liked it.

• September 25, 2013Jake

Daniel got my point. When you write ” With the weatherman on the job, you will never be caught out in an unpredicted rain; with the orthodox statistician this would happen to you one day out of four.” you’re making use of things external to your point. It’s “Goofus the Frequentist is technically right more often, but Gallant the Bayesian never gets you wet.” If what you really cared about was concentrating your probability mass around the actual realized weather sequence, you shouldn’t load your argument up with things like this.

That’s why switching rain and shine here makes a big difference, now it’s Goofus who never gets you wet, but it’s still Gallant who is reducing your sequences.

Also you can’t say both “I don’t think Frequentism is anything like a religion” and “the Frequentist cool-aid” without coming across like Humpty-Dumpty.

• September 25, 2013Joseph

Jake,

It’s NOT true that the “Frequenitst is technically right more often”! The Frequentist gets frequencies right, but the goal wasn’t to predict frequencies. It was to predict the weather. It’s the Bayesian who gets the weather right more often.

Jaynes’s example includes two different issues, which I wish he hadn’t done because people are going to confuse them. Probably the best thing is to forget the decision theory aspect of this. It’s valid and interesting, but it just confuses things.

Also, I didn’t load anything. I made the example so simple no one could fail to see the difference between predicting frequencies and predicting the weather; and everyone could understand exactly which one the weatherman needs. And I did this specifically to clarify what the goal in creating probability distributions actually is.

If the weather is R,R,R,R,R,S,S,S,S,S then using the above is fine. If the weather in the 10 days after the first prediction is S,S,S,S,S,R,R,R,R,R then use . was good for what it needed to predict and is good for what it needed to predict. They’re both better than because they both describe the location of their respective true weather sequences better. I don’t believe you actually read my responses and understood them.

Frequentists can be wrong, without being a secular religion. I had thought this was an unremarkable observation, but maybe you have some deep insight indicating Frequentists are either completely right or a modern replacement for Christianity.