Prometheus
Nobody has ever released a full training script for a frontier model. Labs publish weights. This is the training code, against an open spec, written mostly by an agent.
- Stars
- 37
- Commits
- 396
- Production code
- 38k lines Sep 25, 2026
- Verified on an H200
- Nothing yet
Approach
JAX for the model, Rust for the systems, a modified SGLang for rollouts and serving. The model thinks in latent space instead of writing out a long chain of thought: each thought is the whole hidden state fed back in, not one token sampled from it. Reverie shows it at small scale. It has never been done at frontier scale.
There is no cluster. An agent swarm writes most of the code, and anything that doesn’t need 220k GPUs is tested on 32 shared H200s first.
Code against the spec
Production lines per module at commit 4892c3f (bright) against the spec’s estimate (the band). Five of twenty rows; the twenty summed to 35,466 lines against 320k to 640k for the first stage alone.
Tests and verification
| check | result on 48c279c |
|---|---|
uv run pytest | 2,286 passed, 30 skipped (GPU only), 0 failed |
cargo test --workspace | 2,554 passed, 0 failed, 367 test binaries |
cargo clippy -- -D warnings | clean |
Green tests mean the code matches its oracle. They don’t mean it matches the spec: the audit found real bugs in the model, the training step and the RL loss that the tests don’t catch, because the reference oracles copy production’s design instead of deriving from the spec or the papers.
| H200 stage | state |
|---|---|
| V0 NCCL, kvm, NVMe | 1 and 2 GPU jobs passed; the 2×4 job was cancelled while pending. Not verified. |
| V1 JAX vs torch parity | Not run. |
| V3 FP8 vs BF16 | Runner is a 128-dim toy, below spec scale. |
| V4 fault | A stand-in: a NumPy re-implementation. |
| V5 rung 0 | Passes by construction: it fits the law it generates losses from. |
From the repository’s PROGRESS.md, as of September 25, 2026.
Repository
model/The FP32 JAX reference model.data/extractHTML, PDF and LaTeX text extraction, in Rust.data/loaderShuffling, packing and a resumable loader, in Rust.evals/The eval harness and decontamination.contracts/Versioned contracts shared by Python and Rust.harness/slurm,harness/ledgerThe Slurm backend and experiment ledger.
