Marin: Week of September 21st summary

The status thread on #8506 was quiet from September 21 to 23. The run left last week at step 131,046 as hero-main-step121638, and it posted no stops before it reached step 146,340 on September 23, at a median of 24.1% MFU. On September 22 Matt Wittmann fixed main, which could no longer build the hero's shape after #9117, with #9346. That fix cleared the way for the next handoff.

#9378 merged at 20:51Z on September 23, and at 21:39Z mcwitt-agent started the cutover. It forced a checkpoint at step 146,139 and launched hero-fa4sm100-nomask-step146k from main ad754c6d67. The new build adds native SM100 FlashAttention 4 (FA4) kernels #9332, removes the ragged MLP tail mask #9333, and runs coordinated garbage collection every 100 steps #9266. The old run kept training to step 146,340 as the paired control. The 200-step trial passed at 22:41Z, with the new build ahead on every speed measure:

• median MFU 26.75% against 24.10%
• step time 14.67 s against 16.29 s
• throughput 3.14M tokens/s against 2.83M
• peak memory 103.6 GiB against 120.9 GiB

Train loss was higher by 3.6e-4 on average, the same small drift seen at the step-121,638 cutover. On Discord Matt Wittmann put the gain at about 10% throughput.

About an hour later the run hit a storage limit. At 23:44Z the US-EAST-08A CoreWeave account reached 104.56 TiB against its 100 TiB quota, mostly from the cutover's forced checkpoints and handoff copies, and CoreWeave suspended writes. The save at step 146,585 hung in an upload that has no timeout. At about 00:00Z on September 24 the step watchdog restarted the gang from step 146,355. Russell Power cleared space, and writes resumed at 00:39Z. The next save, at 01:01Z, then stalled: 256 of 704 processes stopped writing without an error. The job was preempted at 01:40Z and again resumed from step 146,355. The cause of the second stall is not confirmed; the main suspect is a storage-side stall left over from the suspension. Saves at 02:43Z and 03:44Z completed normally. The first evaluation, at step 147,000, gave a dropless loss of 1.9086, against 1.9098 at step 144,000. The incident cost about two hours and forced 377 steps to be retrained. Cleanup removed about 49 TB of stale checkpoints. The durable copies from steps 108,778, 121,638 and 146,139 were kept. The step-81,716 handoff copy was deleted during cleanup, and no durable copy of step 58,014 was found. Russell Power's #9395 removed the rule that blocked deletes in the hero checkpoint bucket, which had also blocked cleanup. #9402 now projects the finish date from the most recent 1,000 steps, so outages no longer skew it. At about 14.9 s per step it gives a finish date of about November 5.

The run then trained without a posted stop until September 27. At 08:20:35Z the gang stopped advancing. The 900-second watchdog ended the ranks at 08:36Z and Iris retried the gang. loom-oa-dev reported that the cause is unknown: CUDA peer-memory errors first appear during the watchdog teardown, so they do not show a GPU fault. After more setup retries the run resumed and reached step 166,006 at 14:19Z. That is about 7.66T tokens, estimated from last week's rate of 46.2M tokens per step; the thread gives no token count and no end-of-week loss. The step-150,000 save, the first permanent save of the new run, should have landed during the week, but the thread does not confirm it. Two monitoring and scheduling changes are still open. #9415 would feed the hero's W&B evaluation history to the eval-regression alert, which has never received input. #9478 and #9479 propose letting Iris jobs require a specific NVLink rack.

The hero switched to new code on September 23. #9378 handed the live run off at step 146,139 to hero-fa4sm100-nomask-step146k. The run's W&B history forks from the old run at step 146,138. Both code states restored the live step-144,000 checkpoint on one GB200 rack and trained the same 20 batches. The new code took 14.12 s per step against 15.79 s, and MFU rose from 24.88% to 27.81%, an 11.8% gain. Losses matched to within 0.00015 at every step. Three changes account for most of the gain. #9332 runs upstream FlashAttention-4 (FA4)'s native SM100 forward and backward kernels in place of Marin's port. Each direction was worth about 4% in paired 64-GPU studies (+4.03% backward, +4.02% forward). #9333 drops three tail masks from the ragged expert MLP, each a full pass over the receive buffer, for +1.76% (27.69% to 28.18% MFU over three pairs). #9266 turns on last week's coordinated garbage collection every 100 steps. It is projected to cut elapsed time 3–4%, though that has not been measured at 704 GPUs, and MFU does not show it. #9366 had to land first. A uv link-mode change had stripped executable bits from wheel binaries, which left pods in one gang with mismatched CUDA 12 and CUDA 13 NCCL libraries, so NCCL bootstrap failed. #9346 fixed an attention-output sharding error that the context-parallel work had brought to main on the hero's mesh. Matt Wittmann announced the relaunch at roughly +10% throughput. Over steps 146,139–150,632 the 11-rack run's median step time was 14.71 s at 26.68% MFU. Larry Dial said the gains give him more confidence to try bolder architecture changes in later runs.

Communication overlap was the next target. A re-measurement on #8317 found that collectives now overlap with compute for 60.5% of their time, up from 16.5% on August 17. Exposed collective time fell from 3.22 to 1.51 s per step, and compute fell by 1.74 s. Most of the remaining exposed time came from a barrier between the ragged backend's two expert chunks. Draft #9481 pipelines those chunks, keeps small collectives out of the transport windows, and adds a tool that builds a profile-guided latency estimation (PGLE) profile from one trace. With its profile, overlap reached 81.0% and the step dropped from 13.94 to 13.72 s (1.6%), with loss within 3e-4. Without the profile, the same code slows the step by 1.3%, so it cannot go to the hero until the production recipe uses PGLE. A later push toward 90% was stopped. Ranks in one rack differ by 4–6% in kernel speed, so faster ranks wait inside collectives. Removing one sync point only moved the wait to the next. One PGLE schedule also let loss drift up by about 1e-3, which suggests a race. Registering NCCL symmetric buffers for the all-gathers and reduce-scatters in #9374 saved 0.24% on typical steps but showed no significant change in mean throughput. #9319 and #9318 split two fixes out of the FA4 work: stale NaNs in masked shared-memory tiles of the old segmented backward kernel, and a data-loader thread that crashed JAX shutdown between benchmark arms. #9393 closes the review follow-ups on #9332.

Romain Yon tested whether the router should compute its scores with FP32 accumulation. Against an FP64 reference, the change cut router-score RMS error from 0.0075 to 0.000028. On a restored step-126,000 batch it reordered the top-8 experts for 9.0% of tokens and changed the chosen expert set for 2.6%. It cost 1.5–2% per step. Over a 20-step paired continuation, training loss was the same and Paloma macro loss was 0.0007 higher. His advice was to keep the current router arithmetic for the rest of the run and to test the change on a fresh run instead. The work also found that checkpoint-only evaluation skips the checkpoint's pending query-bias update, so it can score a different routing state than the next training step would use. #9352 tracks the fix.

Work toward the long-context phases continued. #9116, #9117, #9118 and #9119 merged on September 22. They add a context axis to the Grug mesh, context-sharded queries in FA4, and a context-parallel short convolution. An earlier revision trained 262K-token sequences on 64 GB200s with expert parallelism (EP) 16 and context parallelism (CP) 4 at 10.03% median MFU, with 1.42% of expert assignments dropped. #9344 lists performance and precision follow-ups. #9278 lets Muon keep expert sharding when the expert stack needs no padding. Draft #9460 saves and resumes pipeline-parallel checkpoints. In tests at reduced scale, resumed losses matched uninterrupted controls to within 5e-7 and 3e-6, but the full 48-layer, 65K-token shape has not been tested. On why the run will train longer at 65K before going to 262K, Larry Dial said the MFU cost is small. Throughput in tokens per second drops 30%, which adds about 4% to total run time if applied to the last 10% of training. He added that the team has no clean long-context scaling laws that could favor 0.5T over 2T tokens. For checkpoints, #9306 and #9307 find the hero's permanent checkpoints by following its W&B run history, which found 22 checkpoints. The FA4 backend rename had changed every sample ID and orphaned 8 finished sample sets. #9387 fixed that and moved completion sampling to a full 64-GPU rack, which halves job time to about 2 hours. #8534 was closed on September 25. It tracked the untimed post-restore barrier that once held 704 GPUs idle for 4 h 34 m. No run was preregistered this week.

Will Held published the September 21 Datakit SFT checkpoint. It adds explicit Terminus training and supports /think and /nothink. A √32 learning rate on the seven special-token rows fixed the think-marker stutter seen in the September 16 cut: Harshit Varma measured doubled starts at 0.1–0.7%. In #8089 Held reported that the mix now has data for every cold-start skill except model identity. That covers 1.37B tokens of reasoning-off and 1.19B of reasoning-on chat, plus 95.6M tokens of GLM-5.3 compaction summaries for #8095 and #8091. David Hall asked whether to declare the recipe done; Benjamin Feuer wants the identity data in first. The need was clear on the public endpoint, where Kaiyue Wen found the model claims Google made it when asked in English and Qwen when asked in Chinese. The mode matters for agent evals. With thinking off, Luke Lee scored the cut at 21 of 100 on SWE-bench Verified random-100 and 6 of 85 on Terminal-Bench 2 (TB2). Harshit Varma confirmed that /nothink beats /think under Terminus-2: 8.7% vs 6.5% on TB2 and 21.4% vs 14.8% on TB-lite. Held had predicted this because the Terminus training data routed to no-think. Held also opened #9434, which keeps teacher reasoning in Terminus chats and raises usable AgentTrove rows from 13 to 95 of 100 sampled. In #9441 he graded 112 trace sources on a five-part rubric: 14 A, 94 B, 4 C. On GPUs, Held's #9316 trains Snowball through Levanter at 32K context on 32 H100s. Russell Power tied curriculum-generated GLM-5.3 data to that path in #9331 and #9430, after an earlier native Grug run went NaN at step 2 and scored 0 of 500 on MATH500 against the base's 306.

RL results came from three directions. Luke Lee showed SFT and RL stack on SWE-bench. SFT on OpenThoughts-Agent traces (minus the IssueTasks slice, which contains rewritten Verified issues), successful Recursive-Task-Synthesis rollouts and instruction-following data lifted the Stage-3 base from 0.145 to 0.241. Thirty Group Relative Policy Optimization (GRPO) steps on a 939-task R2E-Gym band screened from that checkpoint reached 0.307 [0.26, 0.36], each step outside the previous interval. Nothing in the thread beat the base's 0.094 on TB2. Benjamin Feuer closed the TailSFT coverage-gate proposal #8876, citing marginal benefit in early runs. He also sealed his mixed-domain RL with verifiable rewards (RLVR) campaign #9359. The first stage, RLVR1, learned clearly from the September 11 SFT under both sync and async schedules. The second stage, RLVR2, was flat to regressing, and checkpoint pruning deleted the true pass@16 peaks, so the public release uses the best retained steps (116 sync, 146 async). For TB2, Harshit Varma found that 88.5% of Recursive-Task-Synthesis tasks are never solved in 8 tries. Endless Terminals climbs 13 points on its own tasks but moves TB2 only from 6.7% to 7.9%, within noise. His SFT on the September 21 cut cut Terminus parse failures from 67% to 25% of turns, but the model then declares done early and never recovers from errors. Only rejection-sampling fine-tuning on its own verified traces held up: TB-lite 20.0% vs 12.1% (+7.9, CI [+1.4, +14.8]). He also found a Levanter bug that zeroed router biases at SFT init. Marianna Nezhurina began a TaskTrove sweep that starts from 77 tasks with headroom on the easy curriculum set. For PivotRL #9176, Feuer named two NVIDIA pivot datasets, 50,308 SWE rows and 31,111 terminal rows, as first targets.

Feuer replaced the September 16 policy #9193 with Eval Policy 2026-09-24 #9409 and posted the first full panel in #9412: 11 models on 26 benchmarks. Among five Snowball-lineage checkpoints, the RLVR sync step-92 model won 68.8% of paired comparisons and became the release pick, though it was chosen on this same panel. It scores 83.8 on MATH500, 43.9 on AIME24, 43.1 on GPQA Diamond, 43.7 on MMLU-Pro and 73.8 on FinanceBench, but only 3.0 on SWE-bench and 1.2 on TB2. Qwen3.6-35B-A3B scores 91.4, 90.3, 81.3, 78.3 and 72.2 on MATH500, AIME24, GPQA Diamond, MMLU-Pro and SWE-bench. Two ablations fed the policy. The context study #9410 found SWE-bench does best at 32K context with 8K output per turn and Terminal-Bench at 65K with 16K, so the Terminal-Bench setting will move to 65K/16K. The harness study #9411 found strong models care little about harness, so agent evals will split between Terminus-2 and Pi. Will Held asked that every eval set thinking mode explicitly, and his open #9461 checks runs against a chosen policy before launch. Terminal-Bench 2.1 replaces 2.0 in the next revision.

On serving for RL, Russell Power merged #9289, which retrains Snowball's EAGLE-3 draft inside the RL loop every four policy steps. In a seven-step smoke test, holdout agreement rose from 14.37% to 23.37%, and each accepted update took 31.92 seconds including the hot reload into all eight vLLM engines. #9312 lists the launch and data-integrity bugs found along the way. #9313 now builds each SkyRL launch from one config document, and a Snowball smoke run under it finished four steps on five H100x8 nodes and exported the 134 GB policy. Ahmad Qamar merged the async RL launcher #9256 and its dashboard #9191. #9431 registers the September 21 checkpoint for serving with the new Grug reasoning parser. Romain Yon's open #9400 passes all 64 parity prompts on TPU v6e-8 with vLLM. For task data, David Hall and Russell Power are still debating how TaskCompendium tasks map to harnesses in #9214, and Mark Muchane shared a first GLM-5.3 synthetic RL task for feedback before scaling up.

On timing, Benjamin Feuer moved the Snowball release and the expert-model delivery back one week, so the first official post-train now lands about a week after the October 6 date set last week. He reminded contributors that the cold-start checkpoints are not tuned for TB2 or SWE-bench and need further post-training to become experts. His current guidance: mixed-domain RLVR is more stable than narrow expert RL, RL stacked on RL shows forgetting, and per-turn output limits bind more often than total context.

Iris task environments. Russell Power moved installs to clone link mode so uv cache clean could no longer break running tasks #9340 and made a failed uv sync retry once against a task-local cache #9322, #9330. On CoreWeave, clone mode left wheel binaries without execute bits: ptxas looked missing, the CUDA toolchain step exited early, and pods in one gang loaded different NCCL builds and hung in their first collective #9373. Matt Wittmann switched to copy mode #9366, which Romain Yon verified on a four-node, 32-H100 gang; Will Held's narrower fix #9365 closed as superseded. Copy mode then filled the 215 GB disks on GCP TPU v4 hosts at about 6.5 GB per task until the controller removed the workers, as Rohith Kuditipudi reported #9487, so TPU hosts return to symlinks while Kubernetes keeps copy mode #9489. David Hall grouped JAX GPU startup under iris.jax and added a compile-key exchange that fails fast when ranks lower different programs #9364, rather than waiting out XLA's 24-hour coordination timeout #9042. His #9368 makes ranks agree on fused cross-entropy block sizes, which ends the compile deadlock Benjamin Feuer reported #9038.

Iris scheduling and monitoring. CPU jobs can now pass --reserve H100 and queue on an H100 peer without taking a GPU #9354, #9356. After five such coordinators all landed on cw-rno2a #9396, Rafal Wojdyla made placement prefer the peer with more free H100s #9399, and fast-track --submit uses the reservation #9398. Romain Yon proposed letting Kubernetes jobs name one NVLink rack #9478, #9479. Russell Power made endpoint search live and grouped by user and job #9386, sent Evalchemy to in-cluster vLLM addresses instead of the public proxy #9480, and cut a repeated Finelog log tail from 1.3 s to 65 ms #9347, #9310. Matt Wittmann's Grafana panel now projects the hero's finish from the last 1,000 steps #9402; after a rollback, the whole-life step rate had pushed the estimate a month late. His open #9415 reads hero eval history from W&B, since the eval-regression alert had never received any input.

Checkpoints and Hugging Face interop. Russell Power made Levanter load safetensors shards through up to 16 concurrent workers under a 16 GiB staging budget #9464 and export them through concurrent writers #9471. The first export version let process 0 enter a multi-host collective from a worker thread, which #9473 fixed. @mridulagarwal03 split repo@revision tokenizer refs so pinned revisions stop failing Hub validation #9450. David Hall's #8978, open since early September, exposes the June Snowball model as pipeline stages that each load only their own tensors. Zephyr's batch writer no longer hangs when its thread dies #9376, and a verified copy between GCS and R2 #8900 guards against loading a half-copied checkpoint.

Eval harness. @ayushnangia's #9419 stops a scratch retry from merging into the canonical record, where a 2/30 run had shown as 10% over 60 items, and his open #9445 fails lm-eval runs whose shot count differs from the request, as HumanEval's does at ten shots. Karan Sagar counted aggregate-only AIME24 and OlympiadBench tasks as scored #9156, added pipeline-parallel (PP) vLLM serving for eval models #9200, and kept external judge credentials apart from the candidate's #9197. Benjamin Feuer bounded archive-export memory after a 2.84 GB MMLU-Pro sample file exhausted a 16 GiB orchestrator #9130, co-hosted verifier judge models #9128, and applied Harbor output budgets to Terminus requests #9327. Will Held forwarded /tokenize through the serving proxy for Harbor's context counts #9482 and restored Evaldash after a new tokenizer-revision field #9314 broke older records #9355. Feuer also proposed judging eval-policy conformance from trusted evidence rather than a launcher string #9458. David Hall asked whether Evalchemy's math graders and TaskTrove's should merge; Russell Power said his change only keeps an LLM judge out of his self-SFT loop.

Serving, Chat and sandboxes. Romain Yon measured vLLM's --enforce-eager at 44.0 ms per token against 6.52 ms compiled on Grug-67B #9339, and now requires an explicit flag to use it #9345. Benjamin Feuer promoted CUDA 13.2 vLLM wheels #9314. Harbor catch-ups #8816, #8826, #9362 and TPU vLLM refreshes #9301, #9397 remain in draft or review, and a lineage check blocks the GPU fork's refresh #9300. Russell Power's marin-shellbox #9433 gives Harbor trials one machine API over QEMU, ShellSim, gVisor, Daytona and Iris. ShellSim became a marin-core dependency #9326, #9477 after its absence broke marin-serve #9323. Chat gained Git-aware workspaces #9263 and per-request vLLM stats #9342.

Datakit. Rafal Wojdyla's decontamination work from mid-August merged: restartable Bloom-index, drop-set and mark stages #8327, and pinned outputs for all 292 hero sources #8328. Two fuzzy-dedup stages merged, cluster solving #8723 and text materialization #8724, while verification #8725 and the parent draft #8405 stay open. Mark Muchane's #8303 would register the hero quality scores as Datakit steps. Will Held published a blog post on the pipeline behind the pretraining and SFT mixes, and David Hall offered to build a metadata-only hero mix so outside groups can skip the storage steps. Smaller fixes retry Hub and S3 read errors #9447, #9435. A change redacting remote environment values #9446 was reverted the same minute it merged #9459; Russell Power had questioned hiding the whole environment.

Loom automation and access. GitHub agent workflows now launch as Loom sessions #9449, gated on the event author's live write permission #9457, #9465. Load exposed limits within a day: prose cleanup filled all eight shared slots in half an hour and moved to its own profile #9456, idle one-off sessions now archive after a minute #9466, and PR review and agentic lint run once per PR instead of on every push #9463, #9468. Infrastructure as code (IaC) changes added a Pulumi operator role #9329, #9336, a contractor account for Iris submissions #9388, and IAP grants #9338, #9311; org admins can now bypass main's rulesets only through a PR #9453. David Hall rewrote onboarding around an agent skill #9304. The rest were Evalchemy and Harbor pin bumps #9394, #9404, #9405, #9413, #9429, #9440, #9444, #9470 and dependency security updates #9262, #9455.

Fast-track ladder. Larry Dial's fast_track #9287 merged: a dense-versus-MoE ladder of four rungs, d512 to d1280, each on one 8×H100 node. It uses a 16k-vocabulary tokenizer, so its losses don't compare with the 128k-vocabulary baselines. Doubling the residual stream was about FLOP-neutral but won 1.13–1.28× in wall-clock time from higher MFU #9317; omni-neurons, per-expert latents, an earlier router fp32 upcast and a decaying scratchpad channel closed as no-ops or not worth their cost #9321, #9320, #9357, #9370. Tuned block attention residuals at 12 layers reached 2.915 Paloma macro loss against 2.963 #9392. Larry Dial and Kaiyue Wen weighed QK-norm under multi-head latent attention (MLA), which Kimi passed on for its extra KV cache. The pieces combined into "KMA": Kimi Delta Attention (KDA), MLA with an Inkling relative-position bias, and attention residuals #9438, which beats the baseline's compute efficiency by 1.14–1.43× across the rungs #9451; FP8 was shelved after a best speedup of 6%. On September 25 Larry Dial opened Marin Ladder Climb #9451, where an agent hill-climbs d512 loss within eight minutes of 8×H100 time and anyone can steer it by commenting; it reached batch 77 by September 27. Rafal Wojdyla is wiring Datakit into these runs #9371.

Other experiments. David Hall reran the boundary operator from arXiv 2609.19107 #9280, #9281 after finding his earlier runs used a 5% minimum learning rate the baselines did not. With matched recipes the effective speedup grew with scale, 0.909 at d512, 1.026 at d768 and 1.046 at d1024, though a three-point fit still projects the variant falling slightly behind past 1e20 FLOPs; d1280 is running. His dense replication #9292 traced 92% of the paper recipe's early penalty to its WTE/UIS init pair. Kaiyue Wen found the RMNP optimizer 0.032 C4 BPB behind tuned Muon at 130M, a gap a 32× learning-rate sweep did not close #9299. Michael Siu reported that a student model's Rank-Surprisal Ratio reproduces published teacher rankings on agentic traces #9348, though selecting data by it gave an inconclusive gain. A new issue plans the first Snowball training on AMD GPUs #9462.

The outside PRs the page already covers came from Ayush Nangia, who opened six small fixes across Levanter, Zephyr, Grug and the eval harness and asked in #code-review for reviewers, and from Mridul Agarwal, Michael Siu, Kaiyue Wen and Luke Lee. Not covered: Calvin Xu closed nine sub-issues from the Olmix data-mixing swarm, among them #6602, #6603 and #6610, and Pranshu Chaturvedi closed the chunked-routing Agent MoE experiments #5851 and #6509.

When cole posted DeepSeek's DSec sandbox paper, Russell Power said Iris has had gVisor support since #7339, so Marin could soon drop Daytona. Marianna Nezhurina said gVisor halved her RL speed and she uses Apptainer on HPC nodes; Power then shared a proposal to wrap the Docker interface and let each task declare what it needs. After reading the DataKit blog post, Ahmed Ahmed asked what the ">1,000 proxy runs" count adds to the 872 in the mixture regression, which underlies the claim that the optimized mix beat a proportional one on larger models, and whether any run separates the gain from topic and quality bucketing from the gain from the new source pool; no reply appears in the thread. The Alt Man asked why the hero data is not public. Will Held pointed to the source-pool doc and said Marin avoids rehosting others' datasets, and David Hall offered to build a metadata-only copy of the hero mix.

Six people introduced themselves: an incoming Stanford MS student who spent a year as a quant researcher, a member of the PyTorch team, a senior research scientist building open foundation models at ifm.ai, an ML-for-biology researcher working on DNA language models, an engineer on robotics foundation models, and a Snowflake Kubernetes engineer who contributes to vllm-metal. The ifm.ai scientist, AldoPareja, added to a thread on private pretraining data that ifm.ai trains only on public data, half of it synthesized by open models. TheWizard, the robotics engineer, brings context on multimodal models; told that multimodality is off the roadmap, he plans a GitHub issue for an audio-and-image model.

The week's reading ran to RL environments, from Xiaomi's released MiMo-V2.6 RL environments to the OpenRSI index, which Kaiyue Wen noted builds research-agent tasks on Marin's pretraining ladder.