Teddy Tennant

Reverie

Coconut-style latent thoughts, plus a PonderNet halt aimed at the teacher’s hop count. Every thought is trained to decode to its gold step. One loss, one stage, no RL.

Parameters
0.43M
Seeds per cell
~600
Written in
JAX + Rust
Task
Graph reachability

Result

Reverie, 1 layer, d=1920.864
No chain of thought, 1 layer0.761
Chance0.5

On BFS-verified graph reachability, one layer at width 192: Reverie 0.864 against 0.761 with no chain of thought, a gap of 44 sigma. At two layers Reverie loses at every width, by 23 to 32 sigma. A two-layer transformer resolves these graphs in one pass. Shrink it to one layer and the same extra thoughts start paying, and the edge grows with width.

Accuracy; chance is 0.5. From the project README.

Halting

The halt pins to the depth of the problem: a rank correlation of +1.00 on 606 seeds. Drop the term that asks for it and the model collapses to zero steps, which helps two-layer accuracy by 8 sigma. Calibration is real. It costs.

Loss

L =  Σₙ pₙ · CE(answer, W yₙ)          # PonderNet answer
  +  α · Σᵢ CE(path[i], W yᵢ)          # decode each thought
  +  γ · (−log p_m)                    # halt at teacher depth
  +  β · KL(p ‖ Geometric(λ_prior))    # don't collapse

Limits

Toy graphs, binary labels, chance at 0.5. Not language, not scale. Prometheus is the attempt at scale.