The AI Journal

The conventional practice of null-hypothesis significance testing at the 0.05 threshold is now widely acknowledged among statisticians to be a hybrid of two incompatible frameworks. Ronald Fisher offered a convenient conventional line for judging whether a deviation should be regarded as significant, preferring the five-per-cent point as a practical standard while insisting that an isolated significant result was never enough to establish a natural phenomenon. Jerzy Neyman and Egon Pearson developed a decision-theoretic approach that required an explicit alternative hypothesis, controlled long-run error rates, and treated the outcome as a guide to behaviour rather than a measure of evidential strength. Practitioners fused elements of both into a single procedure that neither originator endorsed: set up a null, compute a p-value, reject at 0.05 if the number is small enough, and treat the result as positive evidence for the research hypothesis. Gerd Gigerenzer later labelled the resulting sequence the null ritual. The diagnosis itself is no longer controversial among those who follow the literature.

What remains contestable is why the ritual has proved so resistant to reform. The American Statistical Association issued a formal statement in 2016 clarifying that p-values do not measure the probability that a hypothesis is true, that scientific conclusions should not rest solely on whether a threshold is crossed, and that a p-value does not measure effect size or importance. The statement was widely viewed and cited and generated extensive commentary. Two years later a large group of methodologists proposed moving the default threshold for claims of new discoveries from 0.05 to 0.005, arguing that the lower bar would reduce the rate of false positives even in the absence of other biases. Both interventions were framed as corrections to a methodological error. Both have left everyday practice largely intact.

The persistence is not best explained by ignorance. Textbooks and software packages still teach the ritual, yet the critical literature is readily available and has been for decades. Nor is it explained by simple inertia. Fields that have adopted registered reports, mandatory data sharing, or explicit effect-size reporting have shown that coordinated change is possible when the rules of publication themselves are altered. The more plausible account is that the ritual solves a coordination problem that the critical literature rarely addresses. In a high-volume system in which authors, reviewers and editors cannot afford to invest large amounts of time in every manuscript, a binary threshold supplies a shared, low-cost signal. A result that clears 0.05 can be treated as publishable without further negotiation; a result that fails can be treated as inconclusive without extended argument about the precise strength of the evidence. The threshold therefore functions less as an epistemic claim and more as a coordination device that keeps the machinery of peer review moving under conditions of limited attention and mutual distrust.

This institutional reading helps explain the limited effect of purely methodological interventions. The 2016 ASA statement correctly identified misuses and clarified principles. It did not, however, alter the incentives that make a bright-line rule useful to the parties who must decide which papers to accept, reject or revise under time pressure. The proposal to adopt 0.005 likewise treated the problem as one of calibration. Lowering the threshold changes the operating point of the same decision rule; it does not remove the need for a decision rule that is cheap to apply and easy to defend in correspondence with authors and reviewers. Journals that merely encourage authors to report confidence intervals or effect sizes while continuing to treat statistical significance as the practical gatekeeper leave the coordination function of the threshold undisturbed.

Empirical patterns are consistent with this account. Assessments conducted in the years after the ASA statement found that the language of statistical significance remained dominant in many fields and that high citation counts for the statement itself did not translate into corresponding changes in the modal published paper. Editorial policies that discouraged the phrase “statistically significant” produced local changes in wording without eliminating the underlying reliance on p-value thresholds for decisions about acceptance. The fields in which more substantial shifts have occurred are typically those in which the publication rules themselves were rewritten—through registered reports that accept papers on the basis of design rather than result, or through requirements that force the reporting of all pre-specified outcomes regardless of significance. These are institutional reforms, not educational ones.

The strongest objection to the coordination thesis is that it understates the residual epistemic value of the ritual and overstates the difficulty of change. Critics of the institutional view can point to the gradual spread of estimation-focused reporting, the growing use of Bayesian methods in some subfields, and the fact that a non-trivial minority of journals have altered their guidelines. They can also argue that the ritual, however imperfect, still supplies a rough filter against pure noise and that abandoning any threshold would open the door to even more opportunistic interpretation of data. On this reading the slow pace of reform is evidence of prudent caution rather than of entrenched coordination failure, and further education plus incremental journal policy will eventually suffice.

The objection has force. The ritual is not wholly arbitrary; a conventional filter against results that are highly compatible with pure chance has some protective value, and wholesale abandonment of thresholds would create its own problems of selective reporting and unbounded researcher degrees of freedom. Yet the same evidence that is offered in support of gradual progress also shows how limited that progress remains. High citation counts for the ASA statement and the 0.005 proposal have not produced corresponding shifts in the modal paper across the disciplines that rely most heavily on the ritual. Where change has occurred it has typically required altering the rules of the game—what counts as a complete submission, what is decided before results are known—rather than improving the statistical literacy of participants who continue to face the same career and editorial incentives. The coordination account does not deny that education and better software matter; it claims that they are insufficient while the threshold continues to serve as the cheapest common signal available to time-constrained gatekeepers.

If the diagnosis is correct, the practical implication is narrow. Methodological consensus alone will not displace the null ritual. Reforms that leave the incentive structure of publication unchanged will continue to produce statements, special issues and lowered thresholds while the everyday decision rule remains in place. Effective change requires altering the coordination device itself—by making acceptance depend on design rather than outcome, by requiring the reporting of all pre-specified analyses, or by replacing binary thresholds with graded measures of evidence that are themselves cheap to evaluate and defend. Until those institutional conditions are met, the ritual will persist for the same reason it arose: it is useful to the people who must keep the system running.