Same Data, Different Answers: The Hidden Sources of Irreproducibility in AI Training

In most mature scientific disciplines, the ability to reproduce an experimental result is treated as a foundational requirement—not a bonus feature. A chemist who cannot reproduce a synthesis under equivalent conditions does not publish the synthesis as fact. A clinical trial that yields inconsistent outcomes across sites triggers formal review. Yet in artificial intelligence research, a quieter and more technically complex version of this problem persists largely unaddressed, embedded in the infrastructure of the field itself.

The scenario is more common than most published literature would suggest: two teams, working from the same publicly released dataset and the same model architecture, train their systems and compare results. The numbers do not match. Sometimes the discrepancy is minor. Sometimes it is large enough to call the original paper's conclusions into serious doubt. The culprit is rarely misconduct or carelessness. It is, instead, a constellation of variables that most reporting standards do not require researchers to document—and that many researchers do not fully understand themselves.

Floating-Point Arithmetic and the Illusion of Determinism

At the most fundamental level, the problem begins with how modern processors handle numerical computation. Floating-point arithmetic, the mathematical backbone of virtually all neural network training, is not associative in the way that standard algebra is. The order in which floating-point operations are executed can produce subtly different results, even when the underlying values are nominally identical.

This matters enormously in distributed training environments, where large models are trained across multiple GPUs or across multiple nodes in a computing cluster. When gradient computations are aggregated from parallel processes, the order of summation can vary depending on which processes complete first—a function of hardware timing, network latency, and load balancing that is, in practice, non-deterministic. The cumulative effect of thousands of these micro-level variations, compounded across millions of training steps, can steer a model toward meaningfully different regions of the loss landscape.

Researchers who set explicit random seeds in their code often assume they have controlled for this. In many cases, they have not. Seed-based reproducibility addresses randomness in weight initialization and data shuffling, but it does not govern the order of floating-point operations in parallelized computation. A model trained with the same seed on different hardware configurations—or even on the same hardware at different times, depending on background processes—may produce different weights at convergence.

Hardware Variance as an Unacknowledged Variable

The hardware dimension of this problem receives remarkably little attention in published work. The dominant assumption in AI reporting is that computational resources are fungible: an A100 GPU is an A100 GPU, and a result obtained on one should transfer to another. In practice, this assumption is fragile.

Different generations of the same GPU model can implement low-level numerical operations with slight architectural differences. Memory bandwidth, thermal throttling under sustained load, and the specific version of CUDA libraries installed on a given system all introduce variance that is essentially invisible at the level of a methods section. Cloud computing environments, which now host a substantial portion of academic AI research in the United States, add another layer of opacity: the exact hardware configuration of a virtual machine instance is typically not disclosed to the user in sufficient detail to be reproducible.

The implications for peer review are significant. When a reviewer evaluates a submitted paper, they are generally assessing a result that was produced on a specific hardware configuration that is not fully described—and that they have no practical means of replicating exactly. The review process, in this sense, is evaluating a claim that is structurally difficult to verify.

Randomization Practices and the Mythology of the Fixed Seed

Beyond hardware, the internal randomization procedures of training pipelines introduce additional sources of divergence. Data augmentation routines, dropout layers, batch sampling, and stochastic gradient descent itself all involve random processes. Controlling these with a fixed seed in a single-process, single-GPU environment is straightforward. Controlling them in the multi-process, multi-GPU environments that are now standard for large-scale training is considerably more complex, and the documentation required to reproduce these controls precisely is rarely provided.

More subtly, many researchers make informal adjustments to training procedures mid-run—adjusting learning rate schedules, restarting training from checkpoints, or switching between mixed-precision and full-precision computation to manage memory constraints. These decisions are often made in response to observations about training dynamics and are rarely documented at the level of detail required for reproduction. They constitute a form of procedural knowledge that shapes outcomes but leaves no trace in the published record.

Raising the Bar for Experimental Reporting

The research community has not been entirely passive on these issues. Initiatives such as the Papers With Code reproducibility checklist and the NeurIPS reproducibility program have pushed in the right direction, encouraging authors to release code, specify hardware configurations, and report variance across multiple runs. These are meaningful steps. They are not sufficient.

A more rigorous standard would require, at minimum, the following: explicit documentation of the CUDA and cuDNN versions used during training; specification of whether deterministic algorithms were enforced and at what computational cost; reporting of results across multiple independent training runs with different random seeds; and disclosure of any mid-training interventions or hyperparameter adjustments that occurred outside the formally described procedure.

Some of these requirements impose real costs on researchers, particularly those at institutions with limited computational budgets. Running five independent training trials instead of one can multiply GPU-hours by a factor of five. This is a genuine constraint, and any realistic reform proposal must account for it. One practical approach is to establish tiered reporting standards: lighter requirements for preliminary or workshop submissions, with more stringent documentation expected for archival publication.

What the Field Stands to Lose

The stakes here extend well beyond academic tidiness. AI research increasingly informs high-consequence decisions in medicine, law, national security, and public infrastructure. When the results underpinning those applications cannot be reliably reproduced, the downstream risks are not merely theoretical. A model that performs well in one laboratory environment and inconsistently in another is not a validated tool—it is a hypothesis that has not been adequately tested.

The field's credibility with funding agencies, policymakers, and the public depends in part on its ability to demonstrate that its findings are durable. A discipline that produces results contingent on undisclosed hardware configurations and uncontrolled numerical variance is not, in any rigorous sense, producing science. It is producing artifacts.

Addressing this will require cultural change as much as technical change. Journals and conference program committees will need to enforce more demanding standards. Funding agencies, including the National Science Foundation and the Department of Energy, which support substantial portions of US academic AI research, could condition grants on adherence to reproducibility protocols. And individual researchers will need to internalize the discipline of treating experimental documentation not as an afterthought, but as a core component of the scientific contribution itself.

The computational power available to AI researchers today is extraordinary. The rigor with which that power is applied to the problem of producing verifiable, reproducible science has not kept pace. Closing that gap is among the most important methodological challenges the field currently faces.