It’s believed the crises in science will abate if we only educate everyone on the correct interpretation of p-values and confidence intervals. I explained before in this long post why this isn’t true. Below is a summary.
Two technical points help explain the issue. First,
Data will always appear to be an IID draw from some distribution
If I secretly constructed such a for the data and handed it to you, then it will appear to be a good model for by whatever Frequentist tests you care to dream up.
Which brings me to the second point.
Future data rarely satisfies the same constraints as the test data.
Suppose the model created from had been created using the constraint
Then, by the Entropy Concentration Theorem , the condition that the next data appear to be an IID draw from is (basically) that it satisfies (1) with the same G(x) and approximately the same g.
There are cases where you can expect such a thing. But in general those ‘s are partial snapshots of our Universe at different points in time, and when you put them together they don’t consistently satisfy any teleological constraint like (1). Most physical, biological, economic or psychological systems evolve in their own complicated way without regard to the overarching scheme in (1) and only accidentally satisfied it the first time.
So we can always fool ourselves into thinking we’ve modeled some “data generation mechanism” and we’re usually wrong.
The original sin of Frequentism was using a special type of problem as the template for all of statistics. The tragedy is they picked a special case that’s barely useful and easily deceives.
So you can fiddle-fart around with the nuances of p-values and null hypothesis all you want. Statistically ‘sound’, but non-replicable, results will remain the norm.