The Lab · 0 of 6 toward the Notch paper
Teaching a world model
what to remember.
Toward one question: what should a robot remember? And what should decide what goes into a small memory, without giving up reliability or speed?
One question
Isolate a single mechanism and ask when it appears or breaks.
A scaling ladder
Three or four points along one axis: does the effect hold, grow, or vanish?
Honest limits
Every write-up ends with what would still need testing at scale.
Runs in your browser
Trained in Python, exported, and run live on this page. No server, no waiting.
The Notch paper
- 01FoundationsIn progress
The Baseline
Reproduce LeWorldModel, a small JEPA world model that plans from pixels, on one of its tasks. Measure success rate and planning time per decision on my own hardware, and check whether the planner can show the plan it held before and after each new observation.
- 02EnvironmentUp next
Tasks That Need Memory
Hidden-state tasks, where the outcome of an action depends on something no longer visible: a controlled diagnostic task and a standard benchmark task, the shell game. Does the memoryless planner fail and an oracle given the hidden state succeed, as they should?
- 03SignalsPlanned
The Signal Audit
Before building any memory: log two signals, the world model's surprise and whether each observation changed the plan, and compare them with labelled moments that matter, including under distractors, recolouring and repeated frames. Which signal marks what is worth remembering?
- 04MemoryPlanned
Remembering Everything
The expensive answers: stack recent frames, give the predictor a recurrent state, add always-on learned slots, or keep every past state and attend over all of it, plus an oracle handed the hidden state. These set the ceiling and the price.
- 05SelectionPlanned
The Write Rule
One model, one memory budget, five rules for what gets written: time, chance, surprise, plan change, and surprise weighted by plan change. Which fills a small memory best, and does what it keeps really guide the plan?
- 06SpeedPlanned
The Price of Memory
Success rate against planning time per decision as episodes get longer, for every memory variant, over several seeds. Where does each kind of memory sit on the trade-off between reliability and speed?
- →CapstoneResearch
Notch
What should a robot remember? Comparing what should decide what a world-model planner writes into a small, fixed memory, measured on both success and speed.
After the paper
- On a Real ArmThe same planner and memory on a low-cost arm with a camera
- Another BodyThe same memory, unchanged, on a simulated drone
- Lessons Across EpisodesDistilled experience in a compact graph, recalled only when needed
- Memory in the WorldCan an agent learn to leave itself traces in the scene?
- Many BodiesA leader steering followers through a learned world model