Saving a random seed is useful, but it is not a complete reproducibility record. A result also depends on data, code, software, hardware, configuration and the operations used during execution. Two runs can share a seed while differing in other conditions that matter.
The first step is to define what is supposed to be reproduced. Is the goal to obtain the same output from one fixed configuration, recover the same metric within a stated tolerance, or confirm a broader conclusion on a new dataset? Those are different claims and need different checks.
Define the claim before collecting artifacts
“The experiment is reproducible” is too broad to evaluate without a target. Write the expected result, the permitted variation and the conditions that must remain fixed. If the result is stochastic, specify whether the comparison concerns one run, a distribution across runs or an aggregate summary.
For a document classifier, the target might be a metric calculated on a saved test set under a fixed scoring script. For a generative assistant, identical wording may not be the only relevant outcome; factual support, completeness and usable format may matter more.
Do not switch between those goals after seeing the output. A run that produces different text may still support a task-level conclusion, while a run that repeats the same text may reproduce an error. The acceptance criterion should explain which kind of agreement matters.
The seed controls only part of the process
Randomness can enter through initialization, sampling, data order or other operations. A seed is a way to control a random-number process under its defined implementation. It does not freeze every external dependency or guarantee identical behavior on a different platform.
PyTorch's reproducibility documentation explicitly warns that matching results are not guaranteed across releases, commits or platforms, and that CPU and GPU execution may differ even with identical seeds. That caveat supports a bounded claim: record the environment and relevant controls rather than treating a seed as a universal replay code.
It does not imply that reproducibility is impossible or that every run will differ. It means the scope of the claim must match the conditions actually checked.
Save the data identity, not just a filename
A file named test.csv can be replaced without changing its name. Record a content hash or another reliable version identifier alongside the dataset's origin, license where relevant and preprocessing history.
Keep the split assignment as well. If a new random split is generated during every run, the evaluation is not using the same cases even if the source dataset is unchanged. A saved row identifier or split manifest makes that boundary visible.
Document exclusions and missing-value treatment. A result cannot be reconstructed fairly if the new analyst must guess which records were removed. Preserve the raw input separately from derived data so the transformation can be inspected.
Record the code and dependency versions
Save the exact code revision used for the run. A general repository link is not enough if the code changes later. Include the scoring script, preprocessing steps and configuration, not only the model-loading file.
Dependencies can affect execution and outputs. Record package versions and the relevant runtime information. If an environment file is supplied, check that it corresponds to the run rather than being generated after unrelated upgrades.
Do not claim that a dependency list guarantees compatibility on every machine. It is a record that makes recreation more feasible and helps explain differences when they occur. Hardware and system-level components may still matter.
Separate training, inference and evaluation settings
A training configuration can include initialization, optimizer settings, data ordering and stopping rules. An inference configuration can include sampling controls, input formatting and output limits. Evaluation adds labels, metric definitions and aggregation rules.
Each layer can change the reported result. Saving the model weights without the evaluation script does not establish how the headline score was calculated. Saving a prompt without the model version does not establish which system produced the answer.
WorldLensia's temperature-setting guide explains why reduced randomness is not the same as increased factuality. A reproducibility record should keep those controls explicit without treating them as a quality guarantee.
A fictional run manifest
An original manifest template might contain: run ID, timestamp, code revision, dataset hash, split identifier, model identifier, package versions, device, seed, input settings, scoring version and output-artifact location. Add the reason for any deviation from the planned configuration.
| Manifest field | What it identifies | What it does not prove |
|---|---|---|
| Dataset hash | The saved input bytes | Correct labels or representative coverage |
| Code revision | The selected implementation | Absence of bugs |
| Model identifier | The intended model artifact | Performance on an untested task |
| Seed | A random-process setting | Identical execution everywhere |
| Metric script | The calculation procedure | Whether the metric fits the decision |
This template is not a record of an experiment performed for this article. No training or inference run is being reported. The table explains what a useful record would contain and the boundary of each field.
Repeated runs can reveal variation
If the research question depends on stability, one favorable run may be insufficient. Define a repeated-run procedure before inspecting outcomes. Keep the selected seeds or repetition rule and report the planned set, not just the best result.
For a fictional illustration, suppose five recorded scores were 0.78, 0.80, 0.79, 0.81 and 0.77. Their mean is 0.79 and their range is 0.77–0.81. Those calculations describe only the invented values. They are not measured performance of a model or a confidence interval for future deployment.
The point is to show how a summary should preserve variation. A mean without the run count or spread can conceal whether the result was stable. A range without the procedure can conceal whether unfavorable runs were omitted.
Do not turn a repeated-run range into every kind of uncertainty
Variation across selected random seeds is not the same as uncertainty from sampling a population of documents. It is also not a measure of future data drift or labeling ambiguity. Different sources of uncertainty require different designs.
Name what changed between repetitions. If only the seed changes, describe that. If data splits also change, the experiment has another source of variation. If hardware changes, it may become an environment comparison rather than a simple repeated-run check.
A useful report can include several analyses, but it should not collapse them into one unlabeled error bar. The reader needs to know what the interval or spread actually represents.
Keep deterministic controls and tradeoffs visible
Some software offers controls intended to reduce nondeterminism or choose deterministic operations where available. Read the version-specific documentation before using them. Record the controls actually applied and any operations that remain outside them.
Do not promise identical results merely because a flag is enabled. The documented guarantee may be narrower than the whole application, and some operations or environments may have restrictions. A setting is an input to verification, not a substitute for observing the result.
Performance can also be part of the decision. If a reproducibility-oriented configuration changes runtime, measure and report that under the same workload rather than hiding it. A tradeoff can be acceptable when it is explicit.
Distinguish a rerun from an independent check
Re-running the original code on the original data can establish whether the provided artifacts produce the reported result under the tested conditions. An independent implementation asks another question about the method. A new dataset asks a question about generalization.
These checks can reinforce one another, but their names should remain separate. Do not claim independent replication when the exercise simply reran a supplied notebook. Do not claim deployment validation because an original benchmark was recreated successfully.
The same discipline applies to published claims. “The script reproduced the reported metric in this environment” is precise. “The model works reliably everywhere” is not supported by that one exercise.
Check labels and scoring separately
A perfectly repeatable metric can be calculated from faulty labels. A scoring script can consistently implement the wrong denominator. Reproducibility preserves a procedure; it does not automatically establish that the procedure answers the intended question.
Audit a meaningful sample of labels and inspect the metric definition. Check how invalid outputs, missing records and exclusions are counted. Confirm that the number of scored cases matches the intended evaluation set.
The AI citation-audit ledger offers a related way to connect claims to evidence. A reproducibility record can link the reported score to its exact artifacts and verification notes rather than relying on a summary sentence.
Preserve failures and deviations
An experiment can fail to run because of a missing dependency, inaccessible data or a configuration mismatch. Record the failure and what was attempted. Do not replace it with a different configuration and then describe the result as an exact reproduction.
If a workaround is justified, label it as a modified run. State which component changed and why the comparison remains useful or limited. The difference may be minor, but readers should not have to discover it by inspecting files.
Keep failed outputs when they are relevant to the conclusion. Removing them without a stated exclusion rule can turn an ordinary troubleshooting process into selective reporting.
Include a short README identifying the claim, required environment, input sources, execution steps and expected outputs. Link each reported table or figure to the artifact that produced it. State which steps were actually executed and which remain instructions for another environment.
Use stable names and a manifest rather than a folder full of unexplained final_v2 files. A reader should be able to identify the version supporting the published result without guessing which file is newest.
Preserve source dates and licenses where relevant. Public availability can change, and a future reader may need an authorized archive or a documented alternative. Do not imply that a data link guarantees permanent access.
Monitor changes after the first successful run
A new model revision, dependency update or dataset can require targeted rechecking. Retain the earlier baseline and identify which conditions changed. That makes it possible to distinguish an expected update from an unexplained regression.
Do not rerun every unrelated test automatically when a narrow change has no bearing on it. Focus verification on the claim and affected components, while preserving the evidence needed to justify that focus.
For a deployed application, reproducibility is one part of the evidence record. User outcomes, current data coverage and failure monitoring remain separate responsibilities. A reproducible historical experiment is valuable without being a permanent approval for future conditions.
A practical acceptance note
At the end of a review, state the exact claim checked, the environment used, the artifacts retained and the observed agreement or difference. Identify any unexecuted steps or inaccessible inputs. Keep the conclusion conditional on those boundaries.
The seed belongs in that note, but so do the data, code, configuration and scoring method. A useful reproducibility record makes the whole chain visible. It helps another person repeat the work, diagnose differences and understand what the repeated result actually establishes.
Output artifacts need identities too
Save the predictions or generated outputs that underlie the score, subject to the data's permitted handling. A metric alone can be difficult to audit because it hides which cases changed. Use stable case identifiers so a reviewer can connect an output to its input and label.
Keep any post-processing explicit. If invalid answers are repaired before scoring, the result describes the repaired workflow rather than the raw model alone. If a human edits an answer, distinguish the original output from the edited artifact and record the intervention rule.
For figures, retain the numerical table and plotting configuration. A screenshot of a chart is not enough to reconstruct the calculation. The bundle should connect the published visual to the underlying values and the transformation that produced them.
Compare differences at the case level
Two reruns can have the same overall metric while disagreeing on individual cases. That may matter if the affected cases have different consequences or belong to important input groups. Aggregate agreement does not establish identical behavior.
Create a case-level comparison where appropriate: same outcome, changed outcome, missing output or invalid output. Inspect material changes with the original labels and evidence. Do not assume that every difference is an error, and do not dismiss it because the rounded headline score stayed constant.
Conversely, a different aggregate score can result from a changed case count rather than changed predictions. Reconcile the evaluation population before interpreting the performance movement. The denominator belongs in the reproduction record.
A tolerance must have a reason
If exact numerical identity is not the target, define a tolerance that fits the claim. It might concern a metric difference, a numerical output or a task-level acceptance rule. The justification should come from the decision and measurement method, not from whichever tolerance makes the rerun pass.
Report the observed difference as well as the pass/fail result. A reader can then assess whether the chosen boundary is meaningful. Where the evidence is insufficient to set a defensible tolerance, state that limitation rather than inventing certainty.
This is especially important when comparing environments. A small numerical difference can be immaterial for one use and consequential for another. The same decimal threshold should not be treated as a universal definition of reproducibility.
Preserve the distinction between a plan and execution
A repository can contain a complete-looking evaluation script that has never been run in the stated environment. A README can describe expected outputs that were not observed. Mark instructions, proposed tests and executed results separately.
When sharing a bundle, include an execution log with the command or procedure, date, environment and completion status. If a step failed, retain the error category without exposing secrets or unrelated private data. If a step was skipped, say why.
A reviewer should never have to infer execution from the presence of a file. That simple status discipline prevents a preparation package from being mistaken for completed experimental evidence.
Sources and methodology
Reproducibility limitations were checked against PyTorch's version 2.14 reproducibility documentation, which displays creation and update dates of May 14, 2026, on October 19, 2026. The manifest, score sequence, worksheet and review procedure are original explanatory material. No model training, benchmark execution or independent replication was performed for this article.
To preserve the evaluation split and model documentation with the run, see Data Leakage: Why a Clean Test Split Matters More Than a Strong Score and How to Read a Model Card Without Treating It as Certification.