EnumaElish123 / ReproLLM

Make LLM experiments reproducible. 
 Quick start ·
 Documentation ·
 Examples ·
 CI ·
 Contribute 
A reproducibility linter, experiment recorder, lockfile system, and drift detector for LLM research. It records the LLM-specific state that other tools ignore — model revision, tokenizer and chat-template hashes, prompt hashes, generation parameters, LLM-as-a-Judge configuration, and the pinnability of closed-source API models — and tells you why two runs differ.
**Check what is missing. Capture what actually ran. Share the evidence.**
ReproLLM works alongside your evaluation scripts and experiment tracker, with
reviewable YAML, JSON and Markdown artifacts in your own repository.
## See what needs attention
An audit connects a finding to its source and a concrete next step:
 [code] 
This is an actual excerpt from `reprollm --no-color audit .` on the
quick-start fixture,
before initialization; other findings are omitted. No model runs during audit.
After you record two runs, a semantic diff identifies the fields that changed.
The executable tour captures a small Python
probe twice, changing only the declared and observed token limit:
 [code] 
Excerpt from the generated diff; its other fields are omitted. This is a
standard-library capture exercise, not an inference result. A `HIGH` change
means the runs need review before comparison; it does not measure model quality.
## Why use ReproLLM?
- **Before sharing an experiment:** find missing revisions, dependency pins,
 prompts and reproducibility settings while you can still recover them.
- **When results change:** compare recorded state across runs and see whether
 the model, prompt, sampling settings, judge or environment drifted.
- **When preparing a paper artifact:** export the available evidence and its
 limitations into a document reviewers can inspect.
- **When your method has custom parameters:** declare project rules and bind
 them to real CLI flags, config keys or environment variables.
### Capture the state behind your result
| LLM-specific state | What ReproLLM records or checks |
|---|---|
| **Model identity** | Model and tokenizer revisions, chat-template hashes, dtype, quantization and adapters |
| **Prompt identity** | Content hashes, so an edit remains visible even when the filename stays the same |
| **Generation and backend** | Sampling settings, inference configuration and installed backend versions |
| **LLM-as-a-Judge** | Judge model pinnability, prompt hashes, sampling parameters and repetitions |
| **API model pinnability** | Provider `snapshot_alias` labels versus `unpinnable` mutable aliases; no invented exact revisions |
These sit alongside Git state, dependency declarations, dataset configuration,
training settings and privacy assumptions. Browse the rule catalog
for individual checks and their fix hints.
**Fits your existing stack.** Keep MLflow, W&B or TensorBoard for metrics and
visualization, and your evaluation framework for executing tasks. ReproLLM adds
reproducibility