Reverie
Coconut-style latent thoughts, plus a PonderNet halt aimed at the teacher’s hop count. Every thought is trained to decode to its gold step. One loss, one stage, no RL.
- Parameters
- 0.43M
- Seeds per cell
- ~600
- Written in
- JAX + Rust
- Task
- Graph reachability
Result
On BFS-verified graph reachability, one layer at width 192: Reverie 0.864 against 0.761 with no chain of thought, a gap of 44 sigma. At two layers Reverie loses at every width, by 23 to 32 sigma. A two-layer transformer resolves these graphs in one pass. Shrink it to one layer and the same extra thoughts start paying, and the edge grows with width.
Accuracy; chance is 0.5. From the project README.
Halting
The halt pins to the depth of the problem: a rank correlation of +1.00 on 606 seeds. Drop the term that asks for it and the model collapses to zero steps, which helps two-layer accuracy by 8 sigma. Calibration is real. It costs.
Loss
L = Σₙ pₙ · CE(answer, W yₙ) # PonderNet answer
+ α · Σᵢ CE(path[i], W yᵢ) # decode each thought
+ γ · (−log p_m) # halt at teacher depth
+ β · KL(p ‖ Geometric(λ_prior)) # don't collapse
Limits
Toy graphs, binary labels, chance at 0.5. Not language, not scale. Prometheus is the attempt at scale.
