Marin: Week of September 14th summary

The run entered the week as hero-ragged_a2a-nccl2307-ep-step81k at step 98,362 and 4.54T tokens, and passed step 100,000 at 22:17Z on September 13. A 27-minute interruption at step 103,004 on September 14 has no entry in the status thread. On September 15 at 13:00Z the run hung at step 107,501 in the #8870 pattern and this time the capture caught the missing peer; the watchdog restart cost 21 minutes. Decoding that GPU dump, mcwitt-agent traced the stall to a self-deadlocked thread block inside QuACK 0.6.4's grouped GEMM: only the load and scheduler warps wait on the preceding kernel under programmatic dependent launch (PDL), so the MMA and epilogue warps can read stale expert-group boundaries, decide there is no work, and exit while the load warp waits on them forever. The data-mixture swap, planned for Monday, waited for the permanent checkpoint at step 108,000 to finish writing at 15:54Z. Will Held cancelled the old run at step 108,257 and at 17:15Z launched hero-mix-996f4891-step108k from main with the phase-2 weights from #9162: more command-line agent transcripts, law and world news; less general software, finance, history and math. On Discord Will Held pointed to the mixture log and the ladder comparison #9126, whose d1536 run finished the same day at a 1.20× compute-equivalent macro speedup over the old ladder. At step 108,004 the new run showed loss 1.2393 at 17.33 s per step and 22.64% MFU; since the sampled data changed, train loss is no longer comparable across the swap. The mixture run itself hung at step 108,711 at 20:57Z, 711 steps after its restore, and lost an hour; its dump held seven stuck clusters and confirmed the same warp split.

Matt Wittmann merged #9183, which launches the three QuACK grouped GEMMs without PDL at no measurable cost, and #9179, which pins a PJRT wheel with a faster ragged all-to-all kernel, and at 23:20Z on September 15 deployed both as hero-nopdl-step108k under the deploy-hero-change playbook: handoff at hourly checkpoint step-108778, with the mixture run trained through 108,981 as the paired control. On Discord Matt Wittmann called it a 3% throughput gain and a possible fix for the hangs. The trial passed at 00:33Z: mean loss delta −3.8e-5 nats over 194 paired steps, MFU 23.62% against 23.03%, drop fraction and router metrics unchanged, no hang. By 09:34Z on September 16 the run had gone 2,052 steps without a hang, where the PDL build had hung every one to two thousand steps, and it has not hung since. A comment from x-itm on #8870 cautions that the captures also implicate QuACK's cluster launch control scheduler, so the PDL-off run tests one mechanism and not the other. On September 17 David Hall posted a summary of every change made to the run so far, and Larry Dial added that ragged expert parallelism (EP) cut token dropping from 3% to about zero, which improved train loss but took the model some time to adapt in its final output norm.

The next three interruptions were storage and fabric, not GPUs. At 02:09Z on September 17 the run died at step 114,198: #8974, merged September 15, had changed checkpoint metadata publication to a copy-and-delete rename, and the bucket policy from August 22 that forbids deleting hero checkpoints denied the delete, so the permanent save at 114,000 completed but the error surfaced at the next save and ended the attempt. The retry resumed 70 minutes later and #9215 by loom-oa-dev now writes metadata straight to its final object. At 01:53Z on September 18 an InfiniBand port failure stalled step 118,837; the retry restored 118,722 and resumed 55 minutes later. At 09:28Z the same day checkpoint 120,000's pending delete error stopped the run after step 120,206, and the retry rolled back 206 steps and resumed 70 minutes later. Separately, #9205 traced the recurring 1.5-second gaps between steps to staggered Python garbage collection; the coordinated-GC fix #9224 merged but is disabled on the hero pending #9266.

To bring the metadata fix onto the live run, Wittmann cut the hero over to main on September 18. A pre-deploy d768 test on main showed router z-loss excursions he flagged on Discord; Larry Dial noted the z-loss is logged but not applied, so it did not by itself mean anything was wrong, and the deploy went ahead under the 200-step protocol. #9242 configured the launcher and, using the W&B run forking that David Hall added in #9231, hero-main-step121638 forked the parent's history at step 121,637 and restored a retained copy of checkpoint step-121638, 15,394 objects and 4.29 TB verified, while the old run trained on to 121,979 as the control. The overlap was accepted: mean child-minus-parent loss +0.0003, maximum 0.0008, time per update within 0.1% once the child's hourly checkpoint was excluded, peak memory 115.5 against 103.6 GiB. At 10:32Z on September 19 task 14's GPU 3 raised CUDA_ERROR_CONTAINED after step 124,715 and the node picked up GPU-reset and NVLink-mask conditions; Iris retried from 124,642 and was back in 40 minutes. Permanent checkpoint 126,000 then completed cleanly, the first to verify the metadata fix. The run crossed 6T tokens at step 130,000 at 13:16Z on September 20 and ended the week at step 131,046 at 18:15Z: 6.05T tokens, train loss 1.216 as a six-hour mean, 24.0% MFU and 2.83M tokens/s at about 4,880 steps a day, roughly 32,700 steps and 1.5T tokens for the week.

The data-mixture swap planned last week went ahead. The d1536 rung of Will Held's H100 ladder in #9126 finished all 90,767 updates on September 15 at 0.91844 Paloma macro bits per byte against the original mixture's 0.92507, 0.72% lower, with 15 of 16 subsets improved and Wikipedia the one regression at 0.50%. Read through the original ladder's scaling fit, that is a 1.20× compute-equivalent speedup: the old mixture would need 20% more compute to match. The gain grows with width, from 0.41% at d768 and 0.70% at d1024, and code, which regressed at the two smaller rungs, improved 0.46% at d1536. A forecast frozen from the 75–95% window before the rung finished predicted 2.54351 macro loss; the measured final was 2.54317. The d512 mixture-search swarm that chose the weights gives 1.17× by the same method and 1.43% lower mean Uncheatable BPB, though that swarm helped select the mixture and switches at 10% rather than 25%, so it is not independent confirmation. #9162 merged September 14 and switches the hero at step 108,000, about 27.7% of training, to a mixture with more command-line agent transcripts (1.3% to 6.0%), law (1.1% to 3.6%) and world news, and less general web code, finance, history and math; a cooldown phase at step 312,192 raises natural-science research to 11.3% and the top quality bucket to 28%. Will Held announced the swap on Discord with a pointer to the new docs/reports/hero-mixture-log.md, and Larry Dial observed the experts rebalanced within ten steps and that the new data is slightly harder to predict. Will Held expects one more adjustment before the final cooldown.

The gradient-norm question from last week closed. Frozen-gradient probes filed by mcwitt-agent on #9149 and #9148 restored checkpoint 101,316 on one rack, took no updates, and computed the gradient on the hero's next six batches under the production QuACK 0.6.4 kernels, the XLA ragged_dot fallback and the pooled-wave transport with 3.4% drops: the lm_head gradient came out 0.5735, 0.5736 and 0.5665, so neither the kernel nor dropping moves it at fixed parameters. Its cosine with the weights is −5e-5, meaning the growth steepens the loss along directions that change the model rather than pulling on logit temperature, and turning the z-loss off changes it by 0.28%. The slope per thousand steps was already rising before the transport switch (0.0040, 0.0062, then 0.0133, then 0.0085), and d768 rungs trained dropless from scratch show no such effect, so the reading is a trajectory effect of a model that had adapted to noisy experts and then got complete ones. Larry Dial closed #9148 on September 16 with the norm settled at a new level; on Discord he noted that ragged expert parallelism (EP) took token dropping from 3% to about 0, which improved train loss but took the final output norm time to adapt. The residual hang under audit in #8870 now has a named mechanism. Matt Wittmann decoded the GPU dumps from the stalls at steps 107,501 and 108,711: in every stuck thread block of QuACK 0.6.4's SM100 grouped GEMM, the MMA and epilogue warps had retired on a tile the load and scheduler warps still held as valid, and one cluster retired on its first tile. Only the load and scheduler warps execute griddepcontrol.wait before reading the expert group boundaries from cu_seqlens, so under programmatic dependent launch (PDL) the other warps can read that buffer before the previous kernel has written it. #9183 launches the three grouped GEMMs without PDL, at a cost inside 0.3% on the six-GEMM expert MLP (87.6 versus 87.7 ms per iteration); it is a mitigation, since neither probe reproduced the hang in 100,000 launches and the hero's rate is about one per million GPU-steps, so the recurrence interval is the test. #9179 pinned a rebuilt PJRT wheel at XLA 708c3a4 with 32 blocks of 512 threads, sender-relative peer ordering and eight-load copy batches, measured at 2.766% higher throughput over three matched EP64 pairs. Both went out together on September 15 as hero-nopdl-step108k from checkpoint 108,778, with the old run continuing 200 steps as the paired control; Matt Wittmann announced the relaunch. Jeff Hammerbacher asked whether to file upstream, and #9201 now tracks removing the fork pin once openxla/xla#47928 and the other fork changes land in a release. x-itm cautioned that a clean PDL-off run does not by itself clear the CLC scheduler and proposed a third arm with CLC disabled. Through September 19 the run's three interruptions were an InfiniBand port failure, a checkpoint-metadata deletion, and a CUDA_ERROR_CONTAINED fault; none carried the GPU-hang signature.

Several smaller throughput and correctness fixes landed or queued behind the run. #9205 traced recurring gaps outside the step timer to staggered Python garbage collections, 1.4 to 1.5 seconds each on a rank scanning 4.67 million long-lived objects, recurring every 260 or so updates and costing 3.83% of elapsed time; #9224 collects on every rank at shared step boundaries instead, 2.46% faster in a 64-GPU trial, and #9266 would turn it on for the hero; Matt Wittmann put the expected gain near 3%. A jemalloc dirty-page-decay change for the checkpoint-save stalls in #9180 was closed in #9182 after cutting purge calls 99.7% without a latency benefit. #8741 moved FlashAttention 4 to beta28, with identical losses over 100 hero updates and 15.90 versus 16.03 s per step, and #9228 would use the native SM100 attention backward, 4.03% faster over three matched pairs with planned device memory down from 115.04 to 102.59 GiB, pending a rerun after the rebase. #8974 let Fully Sharded Data Parallel (FSDP) and pipeline training resume each other's MoE checkpoints, but its sibling-rename metadata commit introduced a delete under the permanent checkpoint prefix that CoreWeave denies, which ended a training attempt; #9215 commits the metadata directly. #9159 excludes attention-mask padding from MoE dispatch, which matters for SFT packing rather than pretraining, and along the way fixed a capacity rounding that could exceed the buffer by one row and a per-receiver clipping loop that cost 8% more kernel launches. When Matt Wittmann saw router-metric excursions while validating main on d768, he found the same excursions in August ladder runs, confined to one layer; Larry Dial agreed to deploy with the caveat that router logits moving after 110,000 stable steps would mean a bug, and the main cutover ran a 200-step overlap at checkpoint 121,638 on September 18.

Toward the long-context phases, #9277 reports the full 535B, 48-layer model at 65,536-token sequences on 192 H100s: pipeline parallelism (PP) 24 by EP 8 with batch 384 reached 15.523% median MFU at 174 s per step, a context-parallelism (CP) variant at PP24/EP4/CP2 at batch 96 reached 9.877%, and DualPipeV stopped at 4.917% after needing extra recomputation and offload. These are ten synthetic updates from fresh initialization, and the recipe still depends on scratch JAXPP overlays; #9279, #9283, #9282 and #9278 carry the pipeline adapter, CP support, the SM90 attention backward and a Muon expert-sharding fix as drafts. Checkpoint evaluation became a pipeline: Rafal Wojdyla's #9147 samples every retained permanent checkpoint hourly and publishes a report, #9181 and #9199 score reference continuations with per-token probabilities and the top five alternatives, #9189 draws three seeded samples per prompt on 32 GPUs, and Tim O'Donnell's #9178 added five in-corpus/held-out biocorpus pairs, each verified present or absent by accession and full sequence, to separate reproducing a document from knowing its form. Larry Dial posted step-102,000 completions on #8827 and noted the step-6,000 prompts now give little signal. #9188 trained 16k to 128k BPE tokenizers on the step-108,000 mixture; going from 16k to 128k saves only 19.5% of tokens. The attention-gate issue #8818 and the MuonH expert-norm question #8621 saw no new findings.

The cold-start SFT line produced a second checkpoint and a first full eval of the first. Will Held posted evals for the September 11 Datakit SFT at step 159,000 in #8954: 38.67% on AIME24 at a 32K budget against 37.00% at 8K for the Nemotron-Terminal SFT (the prior best), but 27.67% at the same 8K budget, 69.40 vs 78.20 on MATH500, and 74.89 vs 73.50 on MMLU; its agentic scores on the DCAgent dev set were 5.54% with OpenCode and 4.33% with Terminus 2. On September 16 Will Held published a new cut that behaves more reasonably across the board; router and bias freezing goes into the next cut because token routing still drifts under SFT. The data behind it changed in eight merged PRs: GLM-5.3 compaction and WildChat format-following sets #9161, 75 structured chat partitions from 21 Nemotron post-training v3 datasets #9237 (Competitive-Programming and SWE-v3 stay out), the full Terminus prompt and JSON command protocol preserved and six deprecated AgentTrove lineages dropped #9239, 13 Open-SWE-Traces sources #9245, a 3,450-row misconceptions set #9271, and three normalization rules: assistant turns must end in an answer or tool call #9243, a third identical tool call after two identical replies drops the record #9259 (101 of 210,031 sampled conversations across 246 sources), and every conversation carries an explicit thinking mode #9261. Held's dedup and decontamination report #9212 covered 159 sources and 113,347,086 conversations: 48,265 verified duplicates (0.04%) and 8,421,460 flagged for eval overlap (7.43%), though Will Held found the detector flags every KernelGym rollout because a shared import torch paragraph matches three Terminal-Bench tasks, so the overlap rule needs tuning. #9150 scores all 113M conversations under the base model (0.108–1.355 nats per token across 21 datasets) to compare against loss after 270 SFT updates and propose per-source weights. Held closed #7697 with the GLM-5.2 code responses on Hugging Face. In the SFT channel Will Held asked how to initialize chat-template special tokens; Larry Dial noted the embedding RMSNorm shrinks the effective learning rate of anything initialized at the scale of trained embeddings, and Benjamin Feuer argued for burning in fresh embeddings with a first stage of low-loss web chat data.

The R2E-Gym RL line got its first read under the new eval policy. Benjamin Feuer posted #9193, Eval Policy v0.1, which fixes the in-distribution and held-out benchmarks, 32,768 input and 8,192 output tokens, and Terminus-2 with summarization on; Luke Lee asked whether both settings were meant, since his runs train at 64K without summarization. His evals in #9225 answered part of that: 24 Group Relative Policy Optimization (GRPO) updates on R2E-Gym from the Nemotron-Terminal SFT score 23 of 100 on SWE-bench Verified random-100 against 12 for the base, solving 15 tasks the base cannot and losing 4, while Terminal-Bench 2 stays flat (4 of 87 vs 6 of 84). The base overflows its context on 94–97 of 100 tasks without summarization; the RL checkpoint's lead holds at +11 with summarization on even though training never used it, so the policy setting stands. Update 48 and a refreshed 1,247-task pool did not move SWE-bench. On the training side Luke Lee showed the repaired stack beats the old one on 265 held-out tasks, pass@1 50.1% to 59.8%. A follow-up put SFT on Kimi K2 SWE-smith traces before the same RL: standard SFT fell to 7 and RL brought it back to 20, TailSFT reached 13, neither above 23, and a weight audit found the two SFT checkpoints near-identical, so the 13 vs 20 gap is run-to-run spread; 39% of the Kimi traces are failed trials that still declare completion. Luke Lee also found that distilling a prompt built from thousands of R2E-Gym rollouts into weights scored below the base. Marianna Nezhurina reported that a heavily SFT'd Qwen3-8B sits flat under binary rewards but learns under pass-rate rewards on tasks with headroom. For Terminal-Bench, @hrsht.v and Lee want an SFT stage to lift the base above its 5 of 87 before RL and are generating Terminus-2 traces on the 37K-task Recursive-Task-Synthesis set; Mark Muchane ran them through GLM-5.3 at eight trials per task and published the rollouts, and Benjamin Feuer pointed at the existing GLM-5.2 Terminus-2 traces. Feuer's roadmap puts the first official post-train of Snowball around October 6, and his notes for expert contributors ask for experts within 10–12 days, from the September 11 SFT checkpoint, on public data with no closed-model traces. Speculative decoding for rollouts moved from experiment to plumbing: Luke Lee saw no draft drift over 60 RL steps, Romain Yon pointed to the merged EAGLE-3 support in the vLLM fork and MarinSkyRL, and Russell Power's open #9289 adds typed artifacts that cache target rollouts and distill a reusable draft.

Ahmad Qamar's async RL program #8955 reached its final tradeoff statement. Across three matched seeds of 100 updates on 40 H100s, the age-cap-four configuration (copy weights every update, deep generation pool, abort in-flight requests, parser-only stopping, mismatch masking) scored 46.74% completed-correct on the 256-question development set against 49.35% for the synchronous control, a −2.60 pp difference with population interval [−9.77, +3.91], which meets the 15-point non-inferiority margin. Age cap one was escalated to eleven matched pairs under the preregistered spread threshold and finished at −7.64 pp, missing the 15-point margin by 0.56 and clearing only the 20-point outer bound; its running mean flipped sign twice on the way, and the control's own eleven-seed range is 22.66 points. Against raw main at seed 17, the branch is 22–36% faster at every staleness allowance and 65.5% faster end to end (8,524 s to 2,942 s), mostly because main spends about 2,100 s of every run in weight copies against about 500 s here; making that measurement required fixing a served-model name mismatch, adding generation markers to Snowball's template so any token is trainable, and documenting that main trains on re-tokenised text and writes eval dumps to pod disk. Two caveats stand: the answer-stopping criterion failed, with 94% of control responses length-stopped by update 100, and every async arm discards 21–23% of its groups as stale while the control discards none. Qamar's draft #9256 turns the loop into a launcher with one-line overrides; its default preset ran 100 updates with a 67 s median update on 40 GPUs against 540 s for the synchronous Fully Sharded Data Parallel (FSDP2) curriculum launcher on 64. Romain Yon's draft #9291 adds a Megatron async launcher and a Qwen score-centering study: three seed pairs favored centering by 36.5, 43.5, and 9 completed answers of 756, but duplicate evaluations at the same weights differ by up to 30, so centering stays off by default. Yon's #9144 makes Snowball training deterministic: checkpointing each scan layer cuts the 67B backward estimate from 292.59 to 59.49 GiB per H100, and replacing the atomic scatter-add MoE combine with an ordered gather made repeated scoring bitwise exact, so a two-update 67B GRPO run on 32 learner H100s passed exact zero-drift and unit-ratio gates. His shadow test in #9127 reversed last week's proxy result on real Grug weights: sparse publication took 127 s against 19.5 s dense because change detection and index building ran on the host, though both paths reproduced the target byte-for-byte on all eight ranks. #9143 merged the separated Snowball checkpoint oracle, closing #8828.

Feuer closed the non-agentic math RL campaign #7786 with a corrected checkpoint collection, all ten frozen selections verified against their router-bias tensors, and published a reproduction package for the RLVR1 sync and async arms from the September 11 SFT #9253; both used TP1/PP2/CP2/EP8, the same tensor, pipeline (PP), context (CP) and expert (EP) parallelism layout, so the 2,807 s vs 3,366 s matched-step gap is not geometry. Those 64-GPU runs hit checkpoint stalls: 875–939 GB Megatron saves sat in S3 multipart uploads with live workers #9209, and a SlowDown error on two ranks stranded the rest in a barrier until the NCCL watchdog fired #9213; MarinSkyRL#665's direct S3 writer then saved full Snowball checkpoints in 161.66 s and 153.54 s. Feuer's #9250 reports on-policy distillation (OPD) and multi-teacher OPD (MOPD) in MarinSkyRL, with two reproductions: one OPD step from a Qwen3.5-9B SFT scored 26 of 30 on AIME24 against the Tinker cookbook's 76.7% target, and a SmolLM3-3B student at step 32 landed within 0.47 pp of Open-MOPD on AIME24, AIME25, and IFEval, though Benjamin Feuer noted the stack runs two to three times slower than the SIA-Lab implementation; #9176 proposes PivotRL step weights next. Power published the 45-subject cross-domain curriculum as a typed artifact #9232 (2,405 nodes, 1,999 assignable capabilities) and opened a learning-progression pass #9294 that produced 69 prerequisite edges across two pilot subjects. David Hall's TaskCompendium draft #9187 ran a one-step Snowball smoke on workplace and chat tasks; none of the 64 workplace attempts recorded a tool call because Snowball emits its own <|tool_call|> markers while the run used a Hermes parser. Gonzalo Benegas opened biology eval tracking #9258 with a GLM-5.3 baseline of 101 of 187 graded on BixBench CLI; no Snowball biology run has completed. Work continues on the next SFT cut with router freezing, a Terminal-Bench SFT stage, and the expert runs that feed the October launch.

Grug training and kernels. Several Levanter changes that the hero run depends on, the checkpoint-format fix #8974, its metadata follow-up #9215, padding-free MoE dispatch #9159 and the FlashAttention 4 beta28 bump #8741, are covered in the hero-run engineering section above. Beyond those, David Hall folded common QB routing into one QBRoutedMoE #9238, deleted the unused FA4 THD backend #9241, and capped W&B artifact uploads at 20 MB after a run uploaded its .venv #9153. Yizheng Jiao's #8932 clamps untuned fused cross-entropy block sizes to Hopper's shared-memory budget, ending the 25-second autotune sweep on every call that Neha Hulkund reported #8853. Hall's #9282, still open, lets the native SM90 attention backward take context-parallel query slices.

MoE experiments. David Hall flagged arXiv 2609.19107 for its take on depth scaling and boundary operators, and had two variants open within a day: moe_boundary #9281 adds the paper's parameter-free prelude/core/coda operator to the grug MoE template, tracked in #9280, and paper_rep #9293 replicates its base-size dense experiment. The replication #9292 reproduced the transfer probe but not the central claim: under its own recipe Operator-1 was 0.208 worse than vanilla, not 0.022 better. Larry Dial's fast_track #9287 adds a self-contained 16k-vocab dense-vs-MoE ladder for eight H100s with an in-region data pipeline. He trained 16k to 128k BPE tokenizers on the step-108k hero mix #9188, and Larry Dial reported that vocab beyond 16k buys "shockingly small" compression on the pretraining distribution, a 12.5% gap on held-out documents. Ablation issues followed: hold-one-feature-out #9288, standard feature additions #9290, and a residual-width scan with latent-decoupled attention #9295, where doubling the residual stream at d2048 costs 18% more FLOPs. Stale Agent MoE issues such as #3993 and #4012 closed, along with the 20%-MFU tracker #6693; Hall kept FP8 expert GMM #6699 open: "still think we should do this".

Eval policy and Evaldash. Benjamin Feuer published the Marin Eval Policy of 2026-09-16 #9193, which fixes the canonical benchmark set, the Harbor pin, and which benchmarks may be hill-climbed and which are held out. Luke Lee asked whether the 32,768-token input limit and Terminus-2's default summarization were intended, since his RL runs use 64k context; Held asked whether TB2 overlaps the held-out TB4. Karan Sagar's #9145 has harnesses emit a versioned benchmark protocol with canonical metric names and item counts, so a one-sample capped GSM8K run can no longer stand in for a complete result #8148 and DROP reports F1 rather than exact match #8144. He also consumed Harbor's error taxonomy, recorded by commit SHA #9109; derived agent context limits from the served model #9105; grouped benchmark variants under one family column #9112, #9235; and found that AIME24's accuracy_std_err spelling had left every model tied with a [0, 1] interval, so the column sort did nothing #9196, #9198. Still open: pipeline-parallel vLLM serving for GLM-5.2 at a 1M-token context #9200, #8215, and mirroring weights in a CPU job after a 474 GB download timed out the GPU job #9233.

Eval launcher and FineStore. Feuer's #9251 repaired the launcher for chat-served and reasoning models: it no longer loads the checkpoint tokenizer on chat routes, which had rejected Gemma 4 and Kimi-Linear #9227, and it drops per-config max_tokens caps after 17,925 of 17,944 TriviaQA requests came back truncated and Qwen3.6 scored 0.023 on AIME24. #9254 lets Harbor load hf:// datasets itself, ending a preflight regression #9229; #9113 checks in publication and recovery policies; #9221 adds a --seed override. Co-hosted judge models #9128 and isolated judge credentials #9197 remain open against #9070. loom-oa-dev published the FineStore evaluation archive contract #9158, then had Evalchemy and Harbor write to FineStore directly instead of converting afterwards #9177, and normalized both harnesses' conversations into a shared rollouts_v1 table #9236. Held raised direct inference timeouts from 35 minutes to 12 hours after 64K-token AIME responses outran the deadline #9226. Gonzalo Benegas opened a biology evals tracker with GLM-5.3 baselines #9258.

Finelog and Grafana. Regional forwarding on cw-us-east-08a had stopped on September 11: R2 selected catalog generation 221521 while Finelog's in-memory selection stayed at 221520, so every publication presented a stale token. #9135 adopts a newer same-writer HEAD as the next compare-and-swap base, and Russell Power's #9139 adds a relay heartbeat outside the row path so Grafana pages when forwarding stalls. #9152 keeps a hub restart from firing every alert. The rest of the week went to query cost. Power's #9286 moves repeated panel SQL behind shared bridge endpoints: one Training run's 16 queries and 16.88 seconds became three source queries and 3.21 seconds. Romain Yon's #8930 took the inference overview from 84 scans, with all ten panels timing out, to two scans and 1.19 seconds cold; #9274 puts run triage above the charts, and #9269, #9260, #9272 index session and training-run discovery. Ahmad Qamar fixed the RL view #9123, #9186, drafted an async RL dashboard #9191, and bundled vLLM histograms into one event, cutting rows from 6,208 to 608 #9276. Wittmann's #9268 stops fork efficiency crediting parent tokens against the child's elapsed time, and his #9202 lets FineStore read archives whose format marker a 30-day lifecycle rule expired, which had crashed the GPU canary #9194; #9206 notes shards and schemas expire the same way. Power's #9297 keeps Prometheus collector failures out of the workload's own error.

TaskTrove and TaskCompendium. Hall posted a design and spike for TaskCompendium #9187, a representation that separates a task's semantics and private verifier from its submission format and harness, so agentic and non-agentic data share one schema; the published spike holds 63 specifications and 164 Harbor lowerings. Power's review asked where specs come from and how validation runs; one answer was that Harbor cannot accept a task without a test.sh. Review moved to smaller PRs, all open: the direct-chat lowering #9214, a verifier registry #9218, a cleaned MCQA importer #9216, and NeMo final-action tasks #9220. Power's #9264 routed TaskTrove's 611,698 MCQA tasks with GLM-5.3: 23,860 to RL, 458,562 to SFT, 120,868 as garbage, released as 2026.09.18.3 without changing the default mixture. Hall's #9163 stops an unterminated \boxed{ earning credit for an earlier answer. Marianna Nezhurina asked for documentation of where TaskTrove subsets come from; Feuer said none exists.

vLLM serving. Yon's #9136 makes marin-community/vllm/main the one maintained vLLM source for GPU and TPU, pairing it with tpu-inference v0.29 and retiring the separate TPU release drafts #8430, #8431, #8951; #9270 then aligned the TPU pin with MarinSkyRL's. #9143 splits the Snowball checkpoint oracle into a bit-exact golden separate from the tolerance-based Levanter and vLLM controls, closing #8828, which had blocked fork refreshes since August. The CUDA 13.2 alignment #9102 was closed and deferred; MarinSkyRL moved first, as Romain Yon noted, from PyTorch 2.11 to 2.13 and CUDA 12.9 to 13.2. #9166 adds the vLLM release qualification to the nightly matrix. loom-oa-dev's #9204 gives the Iris Chat UI typed Python tools and simulated agent workspaces exposed as a bash tool, with Git-aware workspaces in flight #9263. Feuer's open #9273 fixes quick-serve cache hits dropping the pinned revision, which made Kimi-Linear and Trinity-Mini fail on every launch after the first.

Iris and Marina. Power's #8914 splits controller request handling into per-resource modules, shrinking ControllerServiceImpl from 3,634 to 319 lines with the wire API unchanged, and #9234 lets iris-batch Pods preempt CoreWeave verification Pods through the scheduler, removing Iris's own eviction loop. Sagar served the Zephyr applet at zephyr.marina.oa.dev #9203; his Zephyr stage views on the Iris job page #9088 stay open. Yunpeng Xia's #8767 keeps Fray from turning application errors into ActorUnavailableError. Feuer's #9190 registers the CLI owner alias in the interactive budget tier, since local submissions had been silently demoted to batch once spend crossed the 10,688 default. Rafal Wojdyla removed an access grant #9168, and #9252 links the dashboard heading home.

Zephyr and Datakit. ravwojdyla-agent's read-only coordinator dashboard #7911 merged after Sagar asked whether Zephyr views should outlive the process; Wojdyla prefers fewer long-lived services and left post-mortems to Sagar's Marina execution history #9173, still open, whose smoke run found one reducer taking 31 times the median input. Sagar's #9087 reports reducer input sizes to Finelog, and his draft #9222 splits skewed reduce targets: a 90%-skewed local run fell from 63.4 to 57.1 seconds while worker CPU time more than doubled. @ModeEric's #9140 lets shutdown() cancel a blocked execute(), after drivers on August 22 held threads for the full 12-hour worker wait. marin-ops-agent's #9170 deletes external-sort spill files one at a time so one failed delete no longer strands the rest, and removes fifteen markdownify options that were never applied. Tinuade Adeleke restructured the Common Crawl WARC layer #7917 and DOCX pipeline #7919 as StepSpecs at Wojdyla's request, with extractor ablations on a 150M model. Mark Muchane's Code Alchemy registration #8687 counts 916.5B tokens across five subsets; @jeremybarreto710's first contribution adds PleIAs/SEC filings #9275. Sagar found interrupted model mirrors re-download finished shards #9230.

Storage, dependencies and CI. marin-ops-agent's #9146 fixed a GCP token-expiry bug that read a naive UTC datetime in local time: a Pacific host cached a one-hour token for 7.9 hours and then got 401s until the skew elapsed. The same PR cut the Zephyr coordinator's worker-liveness probe from every 0.5 seconds, 7,200 controller RPCs an hour per pool, to every five. Power's #9285 gives Iris jobs launched from GitHub Actions server-side deadlines, six hours for canaries and 30 minutes for the CoreWeave integration parent, after jobs kept running once their runners vanished. Hall's #8900, open, adds fsutil verified-copy for publishing the roughly 125 GiB 67B exports from GCS to R2 with per-object hashes and a completion manifest; #9262 updates vulnerable locked dependencies. Yon's #9154 explains beside the Harbor and MarinSkyRL pins why they track branches.

Agents, docs and housekeeping. Power's #9138 stops agent workflows creating design, plan and logbook files in the repository, routing research output to issues, PRs and reports instead, and #9137 trims AGENTS.md to repository-wide policy; #9255 and #9265 follow. marin-ops-agent's nightshift runs found that Gemma's FLOP count passed glu=False for a gated MLP, so reported MFU for Gemma runs had been low since the FLOP calculations landed #9195; #9223 removes a second packing implementation and stops Wikipedia cleanup parsing each article three times; #9132 and #9296 fix documentation drift; #9249 is open. Hall closed the June slop audit #6424 and its sub-issues #6427, #6428, #6429 as stale, and Isaac Hodes closed the July milestone placeholders #6701, #6702, #6715 and opened the week's standup #9171. @dkajtoch's request for the full Marin 32B training trajectory as Hugging Face revisions #8949 closed. @yun1104 submitted a single-3090 speedrun #9097; Hall replied that speedrun is paused and pointed them at the Agent MoE gates.

The outside PRs the page already covers came from Tim O'Donnell, Yizheng Jiao, Tinuade Adeleke, Yunpeng Xia, @ModeEric and first-time contributor @jeremybarreto710, whose #9275 adds 7.2B words of SEC 10-K filings. Not covered: Ethan Liu's #8883 registers OT-TBLite under its benchmark name, closing #8150, and @dkajtoch clarified in #8949 that the stage-end revisions Will Held pointed to are not enough; they want Marin 32B checkpoints about every 100B tokens, as the 8B has.

Russell Power gave Tuesday's community-meeting talk on the infrastructure behind Marin. After David Hall flagged the depth-scaling paper, Ahmed Ahmed asked whether MoE and looping could be composed: pick 2K experts, run K, then loop the token through the next K. Kaiyue Wen thought looping fits a data-limited regime more than a compute-bound run. @LChoshen asked whether any university serves open-model inference to its community; Colin Raffel said Vector runs vLLM endpoints, @ayushnangia pointed to Compute Canada's, and David Hall said Stanford's AI Playground offers chat but no bulk inference. @adi_baradwaj asked whether individuals can contribute their own agent traces; Will Held said traces from open models uploaded to Hugging Face in messages format work as is, but not traces from models whose terms restrict distillation.

Nine people introduced themselves: a Waterloo undergraduate interested in evals and RL environments, a Cambridge PhD student on AI for mathematical discovery, a Purdue undergraduate on post-training for engineering tasks, a São Paulo economics undergraduate building a route-optimization app, a CMU undergraduate who has evaluated Terminal-Bench Science, and a Yale applied-math PhD student on long-trajectory RL. @CodePothunter, Xueyang Wu, works on allocating a fixed training budget across domains by marginal value, the question behind the mixture swap above. @htlou, Hantao Lou, a Berkeley visitor in formal verification and mechanistic interpretability, asked for an interp channel; Mark Muchane said his pre-Marin work was interp, David Hall opened one, and Jeff Hammerbacher seeded it with three papers. @austinvhuang, Austin Huang, who released gemma.cpp and gpu.cpp, wants to try ideas on inference. Accounts that joined without posting include Yoav Artzi, the Cornell NLP researcher; fedebotu, the handle of RL4CO maintainer Federico Berto; and dbeaglehole, the handle of Daniel Beaglehole, whose recursive-feature-machine work on feature learning bears on the new interp channel.

The week's reading ran to RL at scale: Xiaomi's live-streamed MiMo RL run, a training-inference-mismatch paper that Romain Yon will try for staleness budget, the looped-transformer scaling paper, and @kyletokens's post on how research non-profits buy compute.