Marin: Week of June 15th summary

Bring-up of the 67B-A2B hero run continued through the week without a launch; it was originally slated to kick off around June 19 as an ambitious target. After last week closed with David Hall working the production d=2560 shape past step-0 XLA collective-startup failures in #6367, this week traced the startup hangs to a memory problem and surfaced a deeper, structural one underneath. The d2560 MFU thread ground through some forty diagnostic runs (MAY-020 through MAY-060) localizing the pre-step OOM: the full-depth d2560/26-layer shape at batch 256, seq 4096 reliably requested ~74 GiB before the first step and could not be rescued by fp32-vs-bf16 master params, Pallas-vs-XLA cross-entropy, or even recompute_all remat (which only trimmed the request to 72 GiB). Batch 128 fit, one-layer batch 256 fit, and batch 256 at seq 2048 fit, pinning the blocker to full-depth activation/remat liveness that scales with sequence length rather than to static parameter or optimizer state. The one genuinely usable artifact was MAY-042, a batch-128 seq-4096 profile on 16 nodes: roughly 65.7% compute, 33.7% communication, with FA4/CuTe segmented-attention backward as the top compute op and NCCL all-gather as the dominant collective. No run cleared the issue's ≥20% MFU bar.

The week's central finding came from a new sibling experiment, #6493, which asked why the single-node Grug MoE step was so slow and isolated the answer cleanly: MuonH ran at 1.49 s/step (~22K tok/s, ~4.75% MFU) against an SGD A/B at 0.44 s/step (~75K tok/s, ~16.2% MFU) on the identical shape, attributing roughly 1.05 s/step of pure overhead to the Muon update. The investigation pushed the optimizer hard — grouped and stacked Newton-Schulz layouts, a bounded cap that found cap-256 fastest while a flat cap-512 batch was ~2x slower, and finally a 4D [group, expert, m, n] layout that sidesteps the flat-batch cliff — eventually getting the full-production MuonH update at the real 26-layer shape to about 50% of nominal 8xH100 bf16 peak with zero hidden collectives, roughly 1.55x faster than the prior H5 path. But that was the update kernel in isolation.

Cross-posting from #6493 into the gpu channel, David Hall laid out the structural problem: Muon is not friendly to Fully Sharded Data Parallel (FSDP)-style sharding at this cluster size. The Newton-Schulz orthogonalization wants whole 2D matrices local, so the natural “grouped Muon” layout stacks expert matrices and shards over the expert axis — but after computing updates, redistributing them back into the FSDP weight layout is an expert-weight-sized payload (~121.9 GiB bf16 globally) that does not shrink as nodes are added and, unlike ordinary gradient reduce-scatter, naturally sits after the backward/optimizer compute where it is hard to overlap. In effect Muon roughly doubles the per-step all-gather, in a way that does not hide. The candidate remedies are minibatched grouped Muon (in progress), cross-node expert parallelism, and pipeline parallelism — the last of which the codebase is not built around and which David Hall had been deliberately avoiding. After Russell Power noted Muon's favorable staleness properties under pipeline parallelism, Hall concluded “well then i guess we know where we're heading. i really didn't want to do this.”

Around the throughput fight, the run's supporting prep moved but stayed pre-launch. John Larry Dial opened #6449, a June prep-run plan on the new data mix: five cells at a uniform ~1M-token batch — d512/768/1024/1280 in the 5000-tokens-per-param overtraining regime plus a d2560 reference column at 10T tokens — and in the moe channel noted that swapping shared+routed for alternating dense/MoE blocks held loss flat with ~20% better tokens/s on TPU, flagging it as a possible June option while choosing to delay it for run-to-run stability. The loss-prediction registration sub-issue #6046 drew a contributor asking where to follow that work but still has no defined path. Infrastructure around the canaries took repeated hits: Russell Power root-caused and fixed a three-day-running MoE-canary eval crash where the grouped-matmul Pallas kernel ran the eval in f32 and overflowed TPU VMEM, shrinking the canary to hidden_dim 768 in #6378 to close #6363; the Grug multislice smoke was killed when its second v6e-8 slice never booted under us-east5-b preemptible-capacity starvation #6420; the TPU canary sat PENDING ~3.5 hours through a full v5p-8/v4-8 capacity stockout before timing out #6523; and Power filed #6530 after finding the region-replicated nemotron_mix tokenized cache in eu-west4 diverged from the us-east5 reference — proofpile ~90% incomplete — crashing a real europe-west4 training run on cache load. The CoreWeave canary-scaling enablement for d2560 GPU probes is staged in draft as #6386.

The mixture epic moved from open questions to a finished deliverable this week. Will Held reported the production model swarm complete on #5365 (with results browsable at oa.williamheld.com), closing both that launch issue and its spec #5364. He then opened and closed #6391 to hand the resulting mix to John Larry Dial for forecasting, landing it via #6440 (“New Data Mix in Grug”, +443/-3, merged). That clears the epic's must-have for June: four of its five sub-issues are now closed, leaving only metric selection #5362 open.

In parallel, Rafal Wojdyla opened #6521 to evaluate MATES / Group-MATES data selection as a possible replacement for RegMix-style offline mixture sweeps — online, model-aware influence selection that would avoid brittle quality/domain bucketing. Two agent-posted result dumps moved it quickly: an ICL-as-influence proxy study on the adamh-scaling-ladder-nemotron-optimal-1e+23 checkpoints found influence rankings are checkpoint-dependent (real extracted web text dominates GPQA influence at every step; synthetic buckets rank last), and a staged online closed loop on a 50M model showed that, done with a warm-started model and consume-once selection, influence selection beats a random control reproducibly — with Group-MATES's diversity-aware variant edging out plain top-k. Earlier static-mixture and from-scratch attempts that tied or lost to random were diagnosed as methodological artifacts (collapsing diversity; scoring a near-random model). The work is explicitly directional, not a production-scale proof.

Discord activity centered on the corpus feeding these swarms. In tokenizer, Pranshu Chaturvedi described training MoE ladders on a 100B-token subset of Rafal Wojdyla's normalized and deduped 1.1T corpus (Common Pile, FinePDFs, Nemotron, StarCoder2, plus many individual sources), and asked for held-out sets given likely C4 / Paloma leakage into train; Rafal Wojdyla replied that a deduplicated, decontaminated sample could serve as held-out and flagged that he should measure how much Paloma leaks through decontam. Will Held also posted a “Data Mixing Rorschach test” in data-curation.

The CoreWeave US-EAST-02A H100 cluster this epic is about is live and its canaries are green, and work continued toward starting the June run on it. The production d=2560, 26-layer, 256-expert top-4 "May Recipe" MoE was still being worked past its first training step all week, tracked in #6367. David Hall ground through dozens of MAY-0xx diagnostic runs on the new pool, and the picture that emerged is a memory wall rather than a flag: full-depth seq-4096 runs fail with deterministic ~63–74 GiB first-step allocations that XLA remat cannot shrink (even recompute_all only got 91.96→90.93 GiB before OOMing on a 27 GiB request), while a seq-2048 variant of the identical shape fits and a one-layer seq-4096 variant fits — localizing the blocker to full-depth activation/remat liveness. A layout probe traced the worst offender to an involuntary SPMD full-rematerialization of the [1,4096,2560] input-embedding activation; replicating the input embedding made that warning disappear but did not unblock the run, and the 128- and 256-device launches separately hit Initialize clique JAX-coordination rendezvous failures before any train step. As the epic closed, #6367 still had no profile-bearing run clearing the ≥20% MFU bar it was opened to hit.

With the at-scale shape blocked, the throughput work pivoted to single-node SGD profiling on #4312, where the clean reference (FA4/CuTe attention, XLA cross-entropy, ring MoE, batch 8, one H100 node) sits at only ~16 MFU. David Hall added jax.named_scope regions around the router and MoE-backend internals to make the trace navigable, and the resulting forward/backward attribution split out as MoE 43%, XLA cross-entropy 17%, FA4 flash attention 10%, and collectives 5%. The deepest thread of the week was an exhaustive attempt to replace ring expert-parallelism with DeepEP cross-node transport: an assignment-native DeepEP path with a CUDA gradient-fold-back FFI recovered a chunk of the gap on EP16 (~19.7 MFU vs ~16), but a long campaign proved that while upstream DeepEP itself runs cross-node on CoreWeave — once pointed at the right NVSHMEM bootstrap interface (enp157s0np0) and HCA prefix (ibp) — Marin's one-JAX-process-per-8-GPU-node model is fundamentally incompatible with DeepEP normal-mode's one-process-per-GPU buffer ownership. The fix was to launch DeepEP under a supervised process-per-GPU shape (2 Iris tasks of H100x8, each spawning 8 single-GPU workers), under which internode dispatch and combine smokes finally passed, and to build out the full internode FFI plus autodiff plumbing (dispatch/combine custom VJPs through cached internode dispatch). Backend bake-offs over the week kept finding ring the fastest measured EP16/N2 path, with ragged, padded, and assigned-token all-to-all variants losing to it. The profiling tooling that made this legible was itself filed as follow-up work: #6468 to recover named scopes and operand shapes in GPU traces, and #6470 to report the FFI all-gather-under-explicit-mesh footgun upstream (filed as jax-ml/jax#38551).

The week's most consequential conclusion was strategic rather than a number. Profiling Muon on the single-node shape #6493 led David Hall to conclude in the gpu channel that Muon is structurally unfriendly to Fully Sharded Data Parallel (FSDP)-style sharding: Newton-Schulz orthogonalization wants each weight matrix held locally, so the optimizer-side compute and especially the comms are large relative to the forward/backward pass and, unlike the NS compute, the comms do not scale away with node count and are hard to hide. After an overnight minibatched-grouped-Muon attempt failed to fix it, the read was that becoming a Muon shop will require embracing pipeline parallelism (PP) ("well then i guess we know where we're heading. i really didn't want to do this"). Russell Power had independently been looking at PP and noted Muon's favorable staleness properties under it; the delayed-gradient PP spike with an O(W) Muon weight-prediction corrector is now tracked in #6431 and #6423. On the cluster-operations side, an Iris controller OOM on 6/16 (triggered by a researcher launching far too many eval jobs at once) left jobs unstoppable until Russell Power restarted the controller and committed to capping submissions; a preemption-marked-as-failure regression and a duplicate-worker-IP race #6445 were also fixed in the same window.

Several smaller fixes landed cleanly. Russell Power's #6378 closed the three-day TPU canary failure tracked in #6363 — the MoE grouped-matmul Pallas kernel overran its 16M VMEM scratchpad by exactly 452K at first eval because the eval loss function runs the gmm in float32 while the train step casts to bf16, doubling the per-expert weight window — by shrinking the canary to a smaller-but-representative hidden_dim 768 shape that leaves comfortable VMEM margin. #6444 replaced behaviorless Levanter entrypoint smoke tests with assertions on stable outputs across train_lm, viz_lm, export_to_hf, and vLLM inference, and #6397 nested the manual cw-scale MoE run outputs under a dedicated experiments/grug-moe-cw/ prefix so they stop cluttering the storage root. The d2560 profile harness, FA4/CuTe metadata improvements, and the DeepEP-remat-policy plumbing the throughput campaign depends on are staged in the still-draft #6395, #6396, and #6463. Two older derisking tasks were retired as the cluster came online: the tiny grug/moe-on-GB10 smoke #3456 and the first-few-steps x00B-MoE-on-TPU-and-GPU check #3098 were both closed.

No GitHub PRs or issue comments landed on this epic this week, so the action was all on Discord, and it was substantive. Will Held posted the long-context token distribution for a new data pool, and it is far richer than the nemotron_mix that motivated last week's "8k pretraining is a no-op" finding #6277: 20.5% of tokens sit in documents longer than 32k and 11.8% past 256k, versus the ~8.4% of nemotron tokens beyond 4k. On the back of that, John Larry Dial sketched a concrete extension ladder to run at small model scales while the compute is available: 4k→4k, 4k→32k, 8k→32k, and 8k→64k, each with an LR decaying to a 0.05 peak. His current plan leaves the final 32/64k→262k stretch as something to iterate on the 67B/A2B model directly after its 10T-token pretraining, reasoning that the last stage typically fits in ≤150B tokens (about 1.5% of the run budget), so the team can try 2-3 options and ship the suite downstream with a recommended best.

In parallel, Pranshu Chaturvedi is training d512 and d768 MoEs from the Grug MoE ladder on a 50B-token subset, tokenized at both 8k and 32k seqlen and run with and without the numeric-specific optimizations, which feeds directly into the seqlen question this epic is chasing. Bilibird flagged GLM 5.2's use of an IndexShared mechanism in each 4-layer block for agentic long-context training as a relevant external data point. On the tracked work, the four context-extension experiments #6170, #6171, #6194 remain closed with their stability conclusions intact, while the long_context_32k sensitivity sweep #6232 and the 8k-pretraining experiment #6277 are still open and unchanged since last week.

A quiet week here, with one new issue and no shipped leaderboard or UX work. Benjamin Feuer opened #6503, an "Automated eval system" spec: for a given checkpoint, run Marin's full eval list (or a subset), publish to the eval leaderboard from a single-source-of-truth database, support both Marin's own models and others served over the OpenAI-HTTP inference contract from #4827, and define an agent-skill path for adding new evals so the eval set stays visible on the leaderboard. The definition of done is end-to-end with zero manual steps — a new-checkpoint trigger fires the configured eval subset, results land in the DB, and they appear on the leaderboard tracked by #6043 — leaving the trigger definition and TPU-versus-GPU execution (#6041, #6042) as open decisions. The issue drew no comments this week.

On Discord, the only eval-ownership signal was Benjamin Feuer noting that, until Marin onboards an eval DRI, he would continue standing up evals himself rather than relying on the OT-Agent crew. No leaderboard or eval-dashboard UX surfaced directly otherwise.

The GPU side of GrugMoE vLLM inference stayed quiet. #6042 still carries no sub-issues and no assigned pull requests, and drew no comments this week. The one item that newly touches it is #6503, Romain Yon's spec for an automated eval system — new-checkpoint trigger to configured eval subset over the OpenAI-HTTP contract to results in the DB to leaderboard #6043 — but that issue explicitly leaves the TPU-vs-GPU decision open and names #6042 only as one of the two downstream inference paths, not as work landing against this epic's definition of done.

Activity elsewhere is easy to mistake for this epic but lands beside it. All of the week's GPU-flavored Grug work — DeepEP remat #6396 and #6463, the d2560 H100 profile stack #6395, FA4 sliding-window attention #6377, and the grouped-Muon GPU experiments #6492 and #6493 — is MoE training on H100, not inference, and all of the week's vLLM plumbing (the fork-refresh skill #6467, the pin refreshes #6453 and #6288, and the audit issues #6480 and #6481) is TPU-side. In Discord, the gpu channel was entirely Muon-on–Fully Sharded Data Parallel (FSDP) training discussion, and nothing across the server touched GPU-specific GrugMoE serving. No GrugMoE checkpoint loading, correctness testing, or GPU performance work moved this epic forward.

The dependency wiring that sat in draft last week landed in main. Romain Yon merged #6453, refreshing Marin's TPU-vLLM fork stack to the vllm-project/tpu-inference v0.22.1 release pair and switching marin-core[vllm] off the opaque vllm-tpu==0.19.0 PyPI package onto source-pinned Marin forks. The merge aligned the CPU, TPU, and vLLM stacks on a shared set of pins (jax==0.10.0, libtpu==0.0.40, torch==2.11.0), so the tpu and vllm extras resolve together again and evals, brokered vLLM workers, and RL rollout workers can request both; TPU vLLM workers now pick up VLLM_TARGET_DEVICE=tpu through the run-environment helper. The blocker that had held this back, #6451 (the marin-community/vllm publish push failing for lack of workflow OAuth scope) was closed once the token was re-scoped and the clean refresh branches published, and the older draft #6288 — the centerpiece of last week's writeup — was closed as superseded. Romain Yon announced the landing in infra: "I just updated the versions of vllm, tpu-inference, jax, jaxlib, libtpu, torch and torchvision in main. If anything related breaks, please LMK."

Two follow-ons cleaned up the edges of that landing. Russell Power merged #6485, bumping the marin-docs CI uv pin to 0.11.7: the old 0.7.20 pin could not parse the [tool.uv.extra-build-variables] field #6453 added to build vLLM for TPU without CUDA, so it silently re-resolved and built vLLM from git without VLLM_TARGET_DEVICE, tripping a CUDA_HOME is not set assert on any docs-touching PR. Romain Yon then merged #6484, moving Marin's Python floor to 3.12 while preserving the TPU/vLLM source pins from #6453 and regenerating resolver state, with TPU-vLLM lock and dry-run sync checks across the tpu, vllm, and combined extras as validation.

The refresh also produced a repeatable process and a backlog of constraints to retire. Romain Yon merged #6467, a repo skill capturing the deterministic base-selection, carry/drop/fix overlay-replay, SHA-pin wiring, and smoke-validation steps of the scheduled fork refresh, citing #6453 as the example run. He filed #6480 to audit the Marin-owned pins that still make refreshes harder — chiefly Levanter's transformers>=4.57.1,<5.0 cap, which forces a Gemma4 registration guard in the tpu-inference overlay — and #6481 to remove the now-historical marin-core[vllm]-vs-marin-core[cpu] conflict entry that the aligned pins made obsolete. The open #6483 pushes further, locking a Transformers 5 TPU-vLLM target (transformers==5.12.1, huggingface-hub==1.20.1) and dropping the Gemma4 guard, while #6376 tracks a brokered Harbor hello-world smoke to exercise the served path end to end. As the dependency wiring authors are careful to note, none of this yet proves full runtime correctness for Evalchemy, Harbor, or lm-eval on a GrugMoE checkpoint, so the epic's definition of done remains ahead of the plumbing.

This week's local-metric climb was almost entirely #6491, a tightly-logged hill-climb run by Rafal Wojdyla's agent against a new LLM document-intruder coherence metric for datakit domain clustering. The metric itself landed in #6476: each trial shows a panel five documents (four from one bucket, one intruder from a different bucket on the same side) and asks it to name the odd one out, with detection above the 1/5 chance floor measuring within-bucket coherence under an anytime-valid Robbins confidence sequence. The locked goal was a bucketing at least 20% more coherent than the v0 Luxical K=40 baseline at no more than 40 buckets. The investigation systematically ruled out the obvious levers one at a time on matched same-sample comparisons — stronger embedders (Qwen3-Embedding-0.6B/4B), whitening, nuisance-PC removal, and plain LLM-assigned domains each asymptoted in the single digits — before a stacked config (40 LLM-domains plus core-tightening that keeps each domain's top-60% by centroid cosine and ejects the rest to a catch-all "misc" bucket) cleared the bar at +22% on a high-power frontier panel including David Hall's opus judge.

The agent did not declare victory. It flagged that roughly 40% of documents land in the misc bucket, so most of the lift is core-tightening that exploits the metric's uniform per-bucket sampling rather than genuine separability; the clean all-docs-in-real-domains gain is only about +8%. A frontier ceiling probe then pinned the theoretical maximum genuine improvement at K=40 to roughly +20.5%, achievable only if 40 topic domains were as mutually distinct as different writing scripts — which real topic clustering cannot reach — while also exposing the self-hosted ≤32B judges as near-useless rulers (about 50% irreducible error versus 18% on the frontier panel) and prompting a re-locked panel of gemini-3.5-flash, gpt-5-mini, and qwen3.7-max after dropping qwen-plus. A closing hard-negative variant of the metric, designed to resist tightening by drawing the intruder from the in-group's nearest neighbour, surfaced a real but small boundary-sharpness gain from clustering stronger embeddings (about +5–6%, with single-sample reads swinging from -3% to +13%), below the +10% bar at the available statistical power. The honest net: the metric work succeeded, but the achievable genuine coherence gain at K=40 is modest. Off the issue, the dedup/decontam side surfaced only in the tokenizer channel, where Rafal Wojdyla clarified that the 1.1T-token sampling corpus is normalized but not deduplicated, and that he should check how much Paloma leaks through decontam before picking heldout sets for the MoE ladder.

The pretraining-data work this week shifted from registering sources to making the datakit reference pipeline actually run on the CoreWeave fleet. #6504 (draft, ravwojdyla-agent) adds a CW_SCALE preset (--scale cw): the full K=5000 DAG over every source, but with per-step fan-out capped to the fixed 36-node cluster's ~4,863 allocatable cores and ~3,960 pod ceiling, which the GCP-tuned DEFAULT_SCALE oversubscribes. It also bumps the per-worker resources that the larger CoreWeave shards overflow — embed workers to 128g (finepdfs-eng and stackv2_code OOM the 64g default on oversized single records), dedup workers to 64g RAM / 128g disk (the 800-way vs. 4096-way shards are ~5x larger, so a hot LSH bucket reaches ~70 GiB), and the institutional_books transform to 32g — while excluding the resource and max_workers fields from hash_attrs so a CoreWeave run still shares cached step outputs with a GCP run. Supporting infra also landed: #6405 (merged, ravwojdyla-agent) gives the temp-bucket scheme an R2 (s3://) backend with object-lifecycle TTL rules and points the daily CoreWeave canary at a 7-day TTL bucket so its outputs are swept automatically. On the source-registration side, Tai Vu posted a batch of pretraining and midtraining data PRs for review — Nemotron Specialized v1.2 #6255, Nemotron Code v3 #6257, AutoMathText-V2 #6370, PlanetMath #6372, and more — all still open.

In the tokenizer channel the corpus itself came into focus. Pranshu Chaturvedi described sampling 20B and 50B token sets from a 1.1T-token corpus that Rafal Wojdyla had normalized, spanning Common Pile, FinePDFs, Nemotron, and StarCoder2 plus a long tail of individual sources (hplt_v3, institutional_books, coderforge, numinamath-1.5, swe-rebench-openhands, synthetic-1, and others). Rafal Wojdyla clarified that the 1T sample is drawn from normalized — not deduplicated — data, though sampling removes much of the duplication in practice. When Pranshu Chaturvedi worried that C4 and Paloma had leaked into the training set for his MoE ladder, Rafal Wojdyla said he would check how much Paloma survives decontamination and floated drawing held-out evaluation sets from a deduplicated, decontaminated slice. The epic's own definition of done — tokenized data staged on CoreWeave with confirmed regions and a blessed R2 store — saw no direct comment activity; the work this week was the pipeline tuning and corpus characterization that precede that staging.

The scaling-recipe epic ran hot as an agent-MoE optimizer/architecture sweep this week, but it remains a research line rather than a locked recipe — and notably, no preregistration was filed; the #5359 pre-reg the epic's definition of done defers to is still outstanding. None of this week's many one-change-at-a-time completions set a new per-budget effective-speedup frontier, so the standing frontiers (e.g. 4.20x at d1280) held. The single most promising result was Kaiyue Wen's MuonH magnitude-direction decoupling #6388 — factorizing each weight into a fixed-norm direction (updated by Muon) plus learnable per-row/per-column Adam gains. It beat MuonH at both scales (d512 3.5438→3.5329, d768 3.2330→3.2307) and passed gate-1 at d512 at 1.034x effective speedup, with the clean rule that the gain Adam LR should equal the other Adam-group LR; the apparent throughput "sink" turned out to be a checkpointing artifact, and lm_head gains proved opposite-signed by scale (a wash). Adjacent norm work also landed: Kaiyue Wen's parameter-free RMSNorm-before-swish #6519 made the SwiGLU gate scale-invariant for free — loss-neutral to <1e-3 at ~0.5% throughput cost — and Larry Dial's embed/norm ablations #6442 showed that either the pre-embed RMSNorm or the GatedNorm alone helps (B and D at ~1.04–1.06x) while dropping both (C) is clearly worse (~0.79–0.81x), so running both norms is mild overkill.

A clean, recipe-relevant precision result came out of Larry Dial's bf16 master-weight ablations #6486. The bf16-weight runs trailed the fp32 baseline at the final step despite tracking it mid-run, and Kaiyue Wen isolated the cause to last-mile cooldown numerics rather than an intrinsically worse optimum: the May Recipe decays LR linearly to zero, so the final tiny updates get rounded away in bf16. Ending the LR at 5e-4 instead of 0 closed the gap entirely (the bf16−fp32 delta flipped from +0.0075 to −0.0043, within noise) and the gap did not grow at d768 — so bf16 weights are equivalent to fp32 as long as the cooldown LR stays reasonably large. The catch, per an optimizers-channel exchange on switching Adam→AdamH mid-run, is that on TPU the MFU win from bf16 is essentially nonexistent — leaving the change more useful for memory than throughput. A companion bf16-Newton-Schulz experiment #6505 targets the NS body specifically, where the optimizer FLOP share grows from ~2% at d512 to ~19% at the d2560 production shape.

The week's sharpest optimizer finding came from the delayed-gradient / pipeline-parallelism (PP) investigation #6431 (harness in #6423), where Russell Power studied whether async PP's stale gradients are recoverable with O(weights) extra state. Using a software delay-injection testbed (a depth-τ gradient FIFO inside the optimizer state, τ=0 bit-identical) on a d512 grug MoE, the headline is that Muon is 2.6–5.3x more delay-robust than AdamH, and a cheap forward-side weight-prediction corrector closes ~45–48% of the staleness gap — an effect that is Muon-specific and reproduces tightly across seeds. The work then moved out to a standalone rjpower/marin-experiments repo, so the marin tree is no longer being modified for it. The motivation traces to Russell Power's observation that Muon's conditioning might reduce how much a layer-N update perturbs layer N+1, raising the question of how far PP could be stretched. Separately, the single-node GPU Muon speedup line #6493 by David Hall confirmed Muon costs roughly an extra 1.05s/step versus SGD on the H100 shape (1.49 vs 0.44s/step) but found grouped/stacked Newton-Schulz prototypes uniformly negative so far — unbounded grouping OOMs during autotune and bounded variants don't improve throughput — pointing the next effort at production optimizer sharding boundaries rather than more full-train profiling; the closing prompt is Will Held's #6527 to evaluate Prime Intellect's Fully Sharded Data Parallel (FSDP)-friendly Muon implementation.

Two recipe-level decisions surfaced in Discord. Larry Dial reported that swapping the shared+routed layout for alternating full-dense/full-MoE blocks holds total and active params constant, matches loss at small scale, and buys ~20% more tokens/sec on TPU — and asked David Hall whether the gain carries to the d2560 26-layer shape on GPU, while flagging he'd likely defer including it to keep the hero run stable run-to-run. And the GLM-5.2 release prompted Larry Dial's read that the two biggest things the leading open model has over the current Marin recipe are Multi-head Latent Attention (with index-share across groups of 4 layers) and a multi-token-prediction head — directly seeding the new MLA exploration tracked in #6494 and the simplified-attention MLA experiment #6522, which pairs MLA against a GQA-4:1 sibling to attribute any gate-1 outcome correctly. Meanwhile the Newton-Schulz coefficient question #6404 — raised when Ahmed Ahmed asked where Marin's "quintic" coefficients came from and pushed for the Polar Express variant — was largely settled by Kaiyue Wen noting that in his prior attempts the choice lands in the noise. The actual d2560 production shape stays on its own track #6367, where David Hall spent the week chasing a first-step memory blocker on H100s (now narrowed to the loss/output/vocab SPMD path rather than AdamH hyperball materialization, after the cleanup PR #6385 was closed as no help), and the 32B-A4B TPU recipe search #4012 remains a planned E64/K2-vs-E128/K4 isoFLOP trial rather than a settled shape.

The epic's own sub-issues were quiet — the optimal-teacher distillation pilot #3956 and the SWE-ZERO 140B scale-up #4719 drew no new comments, with no PRs or comments landing directly on #3192 this week — but the synthetic- and post-training-data work the epic tracks moved hard on the Delphi line. Ahmed Ahmed's Delphi RL scaling-laws study #6279 went from setup to a finished SFT readout: Benjamin Feuer closed out the full main SFT downstream-eval grid (54/54 cells — all 27 midtrained checkpoints across scale 3e18→1e22 and mixes p33m67/p50m50/p67m33, each SFT'd from a math-strong magpie and a math-weak wc386k cold start). MATH-500 rose monotonically with FLOPs from ~2 to ~45 best-cell, the midtraining-mix ranking held stable at p33m67 ≥ p50m50 ≥ p67m33, the cold-start effect was large at small scale and converged at the frontier, and GSM8K/AIME24 capability emerged at 1e22. An iso-FLOP, same-harness comparison clamped to Delphi's 4k boxed-grader budget put Delphi-1e22 ahead of every Llama-3.x / Gemma-2 instruct peer up through 17x its training FLOPs, and a base-vs-midtrained pass@k sweep produced the clean negative that SFT on pre-midtraining base checkpoints does essentially nothing for math. All raw artifacts were consolidated to a published evals dataset; Rohith Kuditipudi noted MATH pass@5 for the 1e22 model is predictable from ≤3e20-FLOP points. Benjamin Feuer walked the whole pipeline finding through in midtraining.

On the data-generation side, the agentic-trace index #6191 grew substantially: Benjamin Feuer refreshed it on 6/18 to 71 HuggingFace datasets totaling 323,474 rows of teacher-model trajectories from the OT-Agent datagen pipeline, expanding the original MiniMax-M2.7 @131k cohort (45 datasets, 209K rows) and adding a new Qwen3.5-122B-FP8 @32k cohort (26 datasets). Benjamin Feuer also filed three role-definition epics that reorganize this work explicitly — a Post-training Data manager #6496 owning dataset synthesis, rollout/sample inspection, ETL, and eval-set decontamination; an RL experiment lifecycle manager #6495; and an RL Stack Infra designer/maintainer #6498 — sharpening the boundary between the data/experiment-ops backbone and accountability for the hero runs. On the framework side, Tai Vu merged a k1 KL-loss mode for the RLOO objective #5943, adding the signed sampled-KL estimator alongside the existing k2/k3 modes.

Several long-running synthetic-data experiments were also closed out. The synthetic-data-efficiency study #3905 closed with a positive result: mixing web data with synthetic rephrases reaches ~1.48x data efficiency at 32 rephrases per document, and stitching same-source generations into longer "megadocuments" (or stretching documents with inserted rationales) pushes that to ~1.80x — with the gap over plain rephrasing widening as more synthetic data is generated, and especially large gains on long-context loss. The baxbench post-training-dataset request #3906 and the dense-vs-MoE finetunability/steerability proposal #2953 were both closed as well. Elsewhere in Discord the topic surfaced only at the edges: in tokenizer, Pranshu Chaturvedi described training tokenizers over a 1.1T-token corpus that folds in synthetic sources (gpt-oss-rollouts, swe-zero-12m, synthetic-1, NuminaMath), and Rafal Wojdyla floated reusing Michael Ryan's synthetic data as a decontaminated heldout set for the MoE ladder.

A large repo-hygiene push ran in parallel to the epics. Russell Power consolidated the top-level scripts/ and infra/ trees into a single location, deleting dead code per the audit in #6515 #6517, pruned dead CI workflows and unified the datakit canary tiers #6374, and dropped duckdb plus other now-unused dependencies across packages now that the Iris controller queries logs through the native finelog server #6512. The nightshift agent removed completed isoflop migration scripts and stale pyrefly excludes #6516 and excised orphaned W&B-metrics dead code #6383.

On dependency and platform hygiene, Romain Yon moved the supported Python floor to 3.12 across package metadata, CI, Dockerfiles, and lockfiles #6484, closing #6233, and #6398 and #6421 took the dependabot security bumps for the uv and npm groups. Russell Power also opened a policy thread on pinning native (Rust+Python) wheels via the lockfile rather than version floors #6456, while Jeff Hammerbacher proposed grouping the project's PyPI packages under a single Marin organization #6461.

Process and ops tooling rounded out the week: a contribution-scope and AI-generated-PR policy was documented in the contributing guide #6511, a new ops-steward agent persona and a set of ops runbooks consolidated scattered operational docs into single sources #6379, #6403, and a wait_for.py helper now lets CI scripts select over CI, PR, and shell events #6513. David Hall kicked off a broad code-quality ("slop") audit of Marin, Levanter, and Haliax #6424, spinning off work to remove permanently-skipped tests #6425, rewrite implementation-pinned tests #6428, and replace sleeps in real-service tests #6429. Smaller cross-cutting fixes landed too: a BackgroundTracker race that dropped W&B config.yaml uploads #6400 and relative output paths in the ExecutorStep-to-StepSpec bridge #6409.

External GitHub activity this week clustered on cluster tooling and the optimizer line. @mcwitt built out a CoreWeave-side developer experience, opening #6472 and landing #6473 to reserve ad-hoc H100 dev pods through Iris as a GPU analog of dev_tpu, plus follow-up issues on GPU-count validation #6478 and a unified dev_box tool #6502 — directly on the path the stalled H100 hero run depends on. Calvin Xu added opt-in IAP SSH support #6407, @AjAnubolu fixed relative output paths in the ExecutorStep-to-StepSpec bridge #6409, and Will Moss continued the Zephyr-to-Polars/Arrow migration with stats reporting on stages and workers #6126. On the optimizer side, @s-h-yang revised the PRISM matrix-orthogonalization optimizer #6301 with a faster approximate polar-factor path, reviewed by Kaiyue Wen — adjacent to the week's central Newton-Schulz throughput fight.

On Discord, the most substantive cross-pollination came from the optimizer work spilling into shared reading. After the single-node MuonH profiling in #6493 concluded Muon is hostile to FSDP at scale, John Larry Dial surfaced NVIDIA's Megatron blog on scaling Muon-class optimizers as a possible reference for the pipeline-parallelism pivot the cluster work forced. In news, Will Held flagged an RL paper he found valuable, dovetailing with the Delphi RL scaling-laws grid that closed out in #6279.

Eleven people introduced themselves, skewing toward students and applied engineers: several rising high-school seniors already doing lab research (at MIT CSAIL, the UIUC SALT lab, and on SciML drug-design and MRI work), an OpenEuroLLM contributor working on scaling laws, founders building human-agent annotation and synthetic-data tooling, and an ML engineer in Berkeley taking time off to contribute to open science. Two arrivals intersect active threads directly. Abhash, on the OpenEuroLLM project with ELLIS Tübingen, brings context on the scaling-laws and alignment work the recipe sweeps lean on, while Albert, an LLM research engineer who helped open-source Avito's Avibe model family, intersects with the post-training and function-calling RL line. Two recognized researchers also joined silently, without intros: Huanzhi Mao, a lead author of the Berkeley Function Calling Leaderboard and Gorilla, whose work bears on the agentic-trace and RLOO function-calling effort in #6191 and #5943, and Enrico Shippole (conceptofmind), known for large-scale open long-context datasets and RoPE context-extension work, which maps onto the 4k→32k→64k extension ladder being scoped in epic #6047.

Shared reading this week ran toward optimizer scaling and RL, the same two fronts the hero-run debugging kept circling back to.