A team of researchers has uncovered a significant flaw in how large language models handle statistical reasoning: they systematically violate the law of total probability, a cornerstone principle of basic mathematics that even simple calculators respect.

The discovery, detailed in a new paper from a collaborative research group including scientists from leading institutions, exposes a gap between how language models are theoretically supposed to work and how they actually behave. When these systems are prompted to make predictions about subgroups within a larger population, their aggregated answers frequently diverge from direct estimates about the population as a whole. This inconsistency suggests the models are not performing genuine probabilistic inference, despite widespread assumptions that they do.

How Researchers Tested the Problem

According to arXiv, the team devised an elegant experimental framework to measure this inconsistency. They used binary tree structures to recursively divide populations into increasingly specific subgroups. For instance, a population might be split by demographic factors, then each resulting group split further by additional characteristics. The researchers then prompted leading-edge language models with descriptions of these subpopulations and asked them to generate estimates. When they mathematically combined these fine-grained estimates back together, they frequently did not match what the models produced when asked about the broader population directly.

The researchers tested this approach across multiple problem domains and state-of-the-art frontier models, consistently finding violations of basic consistency properties.

The Macro Fallacy

Perhaps most intriguingly, the research identified a phenomenon the team calls the macro fallacy. Estimates reconstructed from detailed subpopulation responses were often more accurate compared to human reference data than the models' direct population-level predictions. This pattern held across different tree structures and varying estimation tasks.

This finding contradicts the intuition that less information leads to less accurate answers. Instead, it suggests models possess relevant knowledge about subgroups but fail to reliably integrate that knowledge into aggregate conclusions. The effect could be partially reduced through implicit prompting strategies, though the underlying inconsistency persists.

Implications for AI Evaluation

The research establishes statistical self-consistency as a reference-free evaluation criterion for language models. Unlike benchmarks that require human-labeled correct answers, this approach identifies logical inconsistencies within the model's own outputs. No external ground truth is necessary.

The findings raise questions about the reliability of language models for tasks requiring probabilistic reasoning, particularly in high-stakes domains like healthcare, finance, and policy analysis where contradictory estimates could have serious consequences. They also suggest that improvements to in-context learning and probabilistic reasoning represent a significant frontier for model development.

  • Tests revealed models violate the law of total probability
  • Fine-grained estimates often exceed direct estimates in accuracy
  • Effect persists across models and problem domains
  • Partial recovery possible through implicit prompting

The work underscores that despite recent advances in large language model capabilities, fundamental gaps remain in how these systems handle mathematical and probabilistic reasoning.