The 535B-A23B hero run advanced from 1.82T to 3.23T tokens this week, but the path was eventful. The original run hero-12d8b6f0-dee637 reached roughly 2.68T tokens and Paloma macro loss 2.291 (0.732 BPB) before being paused at step 58014 on Sep 3 to free eleven racks for debugging the ragged all-to-all hang #8870. A forced temporary checkpoint captured the state at the cost of about one step. Matt Wittmann then relaunched from that checkpoint as hero-wd-gate-router-p02-step58k #8890, now with gate and router weight decay enabled. That run is currently running on 704 GB200 GPUs at 21.7% MFU, with train loss 1.228 at 3.23T tokens.
The gate/router weight decay was introduced by Larry Dial in #8833, motivated by analysis of attention-gate norm growth at hero scale #8818. At step 42k the per-head attention-gate L2 norms reached 8–10 in early layers, effectively gating attention off in layer 0 and driving a divergence from the scaling ladder trajectory. The decay peaks at 0.02 and anneals linearly to zero over the full run. Scaling-ladder ablations at d1024 showed slightly lower Paloma loss with decay on both gate and router weights.
On Sep 5 the production hero stalled for fifteen minutes and died of an NVLink fabric fault on domain dh1-r137, one of three racks the earlier telemetry window had flagged. Matt Wittmann filed #8934: two NVL72 domains on cw-us-east-08a show fabric-level faults, with one domain accumulating 500+ NVLink recovery events. This is now the leading explanation for the ragged-expert-parallelism (EP) hangs and the strongest evidence that the root cause is hardware, not software. The hero restarted from its hourly checkpoint, losing 125 steps and 20 minutes.
The ragged all-to-all investigation #8870 consumed a substantial debugging effort. Matt Wittmann ran multiple eleven-rack reproduction windows, eliminated data dependence, residency starvation, watch-step aliasing, and checkpoint collectives as causes, and discovered two independent bugs along the way. First, a step-counter sharding bug caused JAX to compile two train-step executables, each registering its own NCCL symmetric-memory window over the same arena — the root cause of the capacity-limited eval crash in #8861, fixed in #8911. Second, the deployed QuACK 0.6.1 had a GEMM scheduler defect that left output regions unwritten under contention — reproduced on four GB200s and fixed upstream in QuACK 0.6.4. Despite these finds, the hero was rolled back to the pooled-wave transport #8884, which costs 0.16 MFU (22.71% vs 22.87%) but drops 2.67% of routing assignments versus 0.018% on ragged.
Will Held closed the open-weight perplexity comparison #8824 after evaluating fifteen base models. The hero's final Paloma forecast is 0.7406 BPB, putting it just above DeepSeek V4 Flash (0.7351) and well above Nemotron 3 Ultra (0.7567), but below Kimi K2 Base (0.6754) and DeepSeek V4 Pro Base (0.6759). The forecast is based on fits of BPB against dropless Paloma macro loss across 77 ladder evaluations.
Preregistration: Ricardo Dominguez preregistered Snowball post-distillation SWE-bench pass@1 at 33% #8933. The prediction uses a multiplicative scaling law relating pre-training steps and SFT budget to BPB on teacher trajectories, then mapping BPB to pass@1 via the Grug fine-tune series.
Most of the other W&B runs this week were short diagnostic and debugging runs. The ra2a-* runs (all crashed) were one-rack reproductions for the ragged-EP investigation. The b8753 and cor-j* runs (all finished) were correctness and comparison tests at one-rack scale. The mx09012328d1narrowfix run (crashed) was a narrow-fix diagnostic.
Marin: Week of August 31st summary
Larry Dial
Percy Liang
CMU · NeuLab
Common Crawl Foundation
Princeton · Dao Lab
3 comments on 3 threads