Teddy Tennant

Prometheus

Nobody has ever released a full training script for a frontier model. Labs publish weights. This is the training code, against an open spec, written mostly by an agent.

Stars
37
Commits
396
Production code
38k lines Sep 25, 2026
Verified on an H200
Nothing yet

Approach

JAX for the model, Rust for the systems, a modified SGLang for rollouts and serving. The model thinks in latent space instead of writing out a long chain of thought: each thought is the whole hidden state fed back in, not one token sampled from it. Reverie shows it at small scale. It has never been done at frontier scale.

There is no cluster. An agent swarm writes most of the code, and anything that doesn’t need 220k GPUs is tested on 32 shared H200s first.

Code against the spec

harness8,710 of 68k to 131k
data2,209 of 55k to 110k
control1,148 of 40k to 80k
kernels1,320 of 25k to 50k
parallel198 of 10k to 20k

Production lines per module at commit 4892c3f (bright) against the spec’s estimate (the band). Five of twenty rows; the twenty summed to 35,466 lines against 320k to 640k for the first stage alone.

Tests and verification

checkresult on 48c279c
uv run pytest2,286 passed, 30 skipped (GPU only), 0 failed
cargo test --workspace2,554 passed, 0 failed, 367 test binaries
cargo clippy -- -D warningsclean

Green tests mean the code matches its oracle. They don’t mean it matches the spec: the audit found real bugs in the model, the training step and the RL loss that the tests don’t catch, because the reference oracles copy production’s design instead of deriving from the spec or the papers.

H200 stagestate
V0 NCCL, kvm, NVMe1 and 2 GPU jobs passed; the 2×4 job was cancelled while pending. Not verified.
V1 JAX vs torch parityNot run.
V3 FP8 vs BF16Runner is a 128-dim toy, below spec scale.
V4 faultA stand-in: a NumPy re-implementation.
V5 rung 0Passes by construction: it fits the law it generates losses from.

From the repository’s PROGRESS.md, as of September 25, 2026.

Repository

  • model/The FP32 JAX reference model.
  • data/extractHTML, PDF and LaTeX text extraction, in Rust.
  • data/loaderShuffling, packing and a resumable loader, in Rust.
  • evals/The eval harness and decontamination.
  • contracts/Versioned contracts shared by Python and Rust.
  • harness/slurm, harness/ledgerThe Slurm backend and experiment ledger.