I’m working on a post defining probabilities. That’s the crux of the matter, so it worth laying out forthrightly. The unifying question probabilities aim to answer is: “When is A relevant for B?” It’s fair to surmise I’m the only living person who believes that, so as preparation for that future post, I brought in Jaynes for moral support.
The following is from his paper “Where do we Stand on Maximum Entropy“. Although this paper is only remembered for Jaynes’s dice example, it’s the most important philosophy of science paper of the second half the 20th century. It’s not just for the philsophical though; a Ph.D. in Statistics could fill a long and illustrious career by exploiting seams opened therein.
From bottom of page 16:
From Boltzmann’s reasoning, then, we get a very unexpected and nontrivial dynamical prediction by an analysis that, seemingly, ignores the dynamics altogether! This is only the first of many such examples where it appears that we are “getting something for nothing,” the answer coming too easily to believe. Poincare, in his essays on “Science and Method” felt this paradox very keenly, and wondered how by exploiting our ignorance we can make correct predictions in a few lines of calculation, that would be quite impossible to obtain if we attempted a detailed calculation of the individual trajectories.
It requires very deep thought to understand why we are not, in this argument and others to come, getting something for nothing. In fact, Boltzmann’s argument does take the dynamics into account, but in a very efficient manner. Information about the dynamics entered his equations at two places: (1) the conservation of total energy; and (2) the fact that he defined his cells in terms of phase volume, which is conserved in the dynamical motion (Liouville’s theorem). The fact that this was enough to predict the correct spatial and velocity distribution of the molecules shows that the millions of intricate dynamical details that were not taken into account, were actually irrelevant to the predictions, and would have cancelled out anyway if he had taken the trouble to calculate them.
Boltzmann’s reasoning was super-efficient; far more so than he ever realized. Whether by luck or inspiration, he put into his equations only the dynamical information that happened to be relevant to the questions he was asking. Obviously, it would be of some importance to discover the secret of how this came about, and to understand it so well that we can exploit it in other problems.
Exploit it in other problems indeed! The future truly belongs to Bayesians.