Marin: Week of July 20th summary

The 67.1B-total / 2.01B-active MoE on TPU v4-2048 — tracked here and logged in detail on #6044 — crossed the halfway mark of its 10.07T-token horizon this week. The live resume run went from about 3.8T tokens at the start of the week to 4.98T by Sunday evening — step 81,722, roughly 49% of the horizon — holding MFU steady at 18.6% and train cross-entropy around 1.37. Paloma macro loss fell from 2.3576 to 2.3318 across the week's four eval points (macro bits-per-byte 0.8518 to 0.8430). The stage-1 loss target for this run was preregistered before launch — 2.269 Paloma macro loss at the 8T mark, from a three-point scaling fit over d=512/768/1024 anchors — and with 3T tokens still to run and the learning rate not yet decayed, the current trajectory is consistent with it. Percy Liang summarized the state for the wider community in Discord: the 2T checkpoint is ready for people to post-train against, and this run continues to 10T.

Most of the week's engineering on the run went into keeping the slice up. Larry Dial filed #7430 after three consecutive submissions of the same launcher died with an identical signature at random few-hour marks: one non-leader task's coordination-service RPC drops with gRPC UNAVAILABLE, that task aborts, and iris then bounces the entire 256-task gang. Two defaults were compounding. XLA builds the coordination service's error-poll RPC with max_retries=0, so a single transient socket close is fatal and no JAX 0.10.1 flag changes that; and the Grug dispatcher left iris's max_task_failures at 0, so the first task failure killed the job before any retry budget could engage. Russell Power scoped an in-task supervisor process that would restart the workload in place, but after an in-person discussion that implementation was dropped entirely in favor of the much smaller fix: #7451 merged raising the failure budget to 10 for shared Grug runs and 100 for the June hero runs, letting iris's existing retry path reschedule the gang. The run still absorbed several multi-hour restart gaps — roughly a day of wall-clock in total over the seven days — but it also strung together a clean 58-hour stretch mid-week.

The 2T-token intermediate cooldown from #6811, now carrying the name Snowball, moved toward a public release. Romain Yon opened #7447 to settle a config.json naming contract: the exporter had been writing 11 fields under both their grug and Hugging Face spellings, a hedge that is unambiguous only because the exporter happens to write equal values to every pair, while the three loaders — the Marin vLLM fork on GPU, tpu-inference on TPU, and Levanter re-import — disagree on which name wins. The recommendation carried: universal transformers names for core fields, the most common public spelling per MoE field, bare names for the grug-specific extras. Larry Dial's counter-proposal to rename qk_mult to query_pre_attn_scalar was dropped once it turned out Gemma already uses that key for something else. #7458 landed the change and re-cut the step-42150 bf16 export — exactly one file differs from the old tree, all 39 weight shards byte-identical, so the logprob goldens carried over untouched — with writes now strict and reads left deliberately tolerant so existing artifacts still load. Romain Yon then filed the full release plan #7495, which puts four decisions up for sign-off: the repo id, whether to ship or strip the base model's chat template (the model has no chat training, so a chat-completions request answers and then hallucinates fake user and assistant turns, and anything calling apply_chat_template silently treats it as chat-capable), a public-upload-then-verify sequence forced by the free-org 100 GB private cap against a 125 GiB tree, and license plus a training-data paragraph covering the full 2.3T-token lineage rather than just the 211B cooldown mix. Every card command has been tested end to end on 8×H100; the plan is waiting on team review.

Two smaller items. The epic's own thread saw its first traffic in a while: a request for raw intermediate checkpoints from the 10T run at roughly 3× spacing — about 70B, 220B, 660B, and 2T tokens, no cooldown needed — to anchor post-training scaling experiments. And #7403, a large proposal to put native FP4 (NVFP4/MXFP4) pretraining under the end-of-year hero run, was closed: David Hall said he strongly wanted to stay away from NVFP4 absent a very good reason to take on the risk, and Jeff Hammerbacher confirmed it had been filed by an over-eager agent alongside the issue he actually wanted, the quantization-aware training (QAT) during SFT issue #7404.

The hero run itself is finished and the epic thread stayed quiet this week — no new comments on #6811 since the Hugging Face question was raised. What moved was the checkpoint's reach. Percy Liang told the community that the 67B-A2B trained on 2T tokens is ready for people to play with for post-training, with training continuing toward the 10T horizon under #6704. Worth restating what that checkpoint cleared: the cooldown landed at 2.2772 Paloma macro-loss from only ~2.1T pretrain tokens plus ~211B cooldown tokens, essentially matching the 2.269 stage-1 target Larry Dial preregistered for the full 10T run — a preregistered number hit at a fifth of the token budget. A small echo of this epic surfaced in code-talk, where Ryan Williams asked for eyes on #7166, which makes step-N on disk mean N completed training steps — exactly the off-by-one that produced this run's step-42149-versus-step-42150 correction.

The publish prerequisite Larry Dial raised in the epic thread — standardize the config against Hugging Face conventions — got resolved as a written decision. Romain Yon filed #7447 showing that to_hf_config wrote 11 fields under two spellings each (hidden_dim and hidden_size, num_experts and num_local_experts, and so on), a deliberate hedge because the marin vLLM fork, tpu-inference, and Levanter each prefer a different name. The argument for fixing it before publishing: the two names can disagree with no arbiter, so GPU and Levanter could silently load different models from the same file; and keys shipped to the Hub can be added later but never removed. Option A — universal transformers spellings for core fields plus the most common public spelling per MoE field, which happens to land on Qwen2-MoE's key set — loads on all three backends with zero loader changes. Larry Dial proposed renaming qk_mult to query_pre_attn_scalar to match its behavior; after discussion they kept qk_mult, since Levanter already uses it and Gemma uses the other name for something else. #7458 merged the contract: writes are now strict, reads keep their fallbacks so existing artifacts still load, and the step-42150 bf16 export was re-cut and re-pinned. Old and new trees differ in exactly one file — config.json, 1368 to 1059 bytes — with all 39 weight shards byte-identical, so the logprob goldens are untouched. The one red check was the vLLM-GPU rank sentinel, which fails the same way on the old export and which #7354 had already recommended against using as a blocking gate.

With the contract in, Romain Yon put the whole release up for review as #7495, the execution plan for #7366: what ships, the complete model card, a tested inference path, and four decisions that need a human to sign off. The repo id marin-community/snowball-67b-a2b-base is the one to argue about now, since every tested command assumes it. Whether to ship or strip the bundled chat template is a real question — the tree carries the shared Marin tokenizer's template, but this is a base model with no chat training, so /v1/chat/completions answers and then hallucinates its own user and assistant turns, and anything calling apply_chat_template will quietly treat it as chat-capable. There is no private staging step, because the free-org private cap is 100 GB against a 125 GiB tree, so the plan is upload public, verify, then announce, with the plan candid that the repo is discoverable from the moment upload starts. The license proposal is apache-2.0 on the marin-8b precedent, and the team still owes a paragraph describing the full 2.3T-token data lineage and decontamination status.

The most useful part of that plan is that the card's commands were already run in an end-user-shaped environment rather than the internal harness: one 8×H100 node with the NVIDIA driver only, no CUDA toolkit, uv as the sole Python tooling. The synced tree hashed bit-exact against SNOWBALL.export_sha256, and greedy completion worked, but two flag-free launches crashed — first FlashInfer's JIT looking for nvcc during worker startup, then its sampler warm-up doing the same. Both fixes, an explicit --attention-backend FLASH_ATTN and VLLM_USE_FLASHINFER_SAMPLER=0, are now baked into the published command, which is exactly the class of failure that would otherwise have greeted the first external user. What remains is team review of the four decisions, the upload preflight, and a pass/fail verification gate — credential-free download, digest and single-name-config check, a serve-and-reproduce of the greedy completion, and a rerun of the 64 representative vLLM parity cases — before anything is announced.

The 2T cut moved from "checkpoint exists" to "checkpoint other people can use." Percy Liang told the server the 67B-A2B trained on 2T tokens is ready for people to play around with for post-training, with pretraining continuing toward 10T under #6704. The week's centerpiece is #7505, the first canonical baseline eval sheet for the SFT'd model — grug-67B-A2B-thinking at step-630, the checkpoint the end-to-end pipeline record #7321 produced — measured beside six reference MoEs (Qwen3-30B-A3B-Thinking-2507, Qwen3.5-35B-A3B, Qwen3-Next-80B-A3B-Thinking, Moonlight-16B-A3B-Instruct, DeepSeek-V2-Lite-Chat, Ling-lite). Benjamin Feuer built the tracker around a deliberately strict rule: a task counts only when it has both a numeric results file and its matching samples file, the launcher passes the actual served context (32,768 tokens, 8,192 for Moonlight) to evalchemy rather than letting lm-eval's ~2k default silently apply, and everything else is retained under old/ for provenance rather than deleted. His read in Discord: Snowball already lands mid-pack on most evals against models with far more training compute, which he takes as a good sign for the public data mix — with LAMBADA (43.0 against 72–76 for the Qwen models) and the GPQA-Diamond / IFEval / AIME24 cluster called out as the visible gaps.

Then David Hall read the samples rather than the scores, and a sample-level error analysis of the public artifact bundle changed the interpretation of much of the table. The dominant failure was output control, not knowledge: the thinking model exposes its planning in the answer channel and loops, so 496 of 500 IFEval outputs begin with "Okay", 253 of 500 hit the 30,720-token generation cap exactly, only 9 of 500 contain an </answer> delimiter, and the grader ends up scoring the raw reasoning trace — minimum-word and minimum-sentence constraints pass almost universally while maximum-length, JSON-only and lowercase-only constraints are near zero. The same looping suppressed math and code: of MATH500's 66 empty extractions, 45 contained an earlier complete boxed answer (30 of them matching gold) before drifting into a malformed later box, so scoring the last complete box gives a demonstrated lower bound of 69.8% against the reported 63.8%; 42 of HumanEval+'s 56 failures reached the cap without ever closing a Python fence. He also found three graders that were simply broken. GPQA-Diamond builds random.Random(42) fresh inside the per-question shuffle, so all 198 gold labels land on D and the reported 45.286% is exactly the model's D-selection rate of 269/594 — and the extractor takes the last standalone A–D anywhere in the completion instead of the boxed choice, disagreeing on 78 of 521 boxed outputs. That is filed as #7527 and announced in evals; the fix lives in the evalchemy fork, not this repo, so it needs a fork patch plus an image and commit pin bump. MMLU had 546 of 14,042 rows where every option scored zero (three subjects entirely, whose "scores" are therefore answer-position frequencies), and DROP is unusable outright — its few-shot answers serialize as the literal keys number,date,spans,worker_id,hit_id, which 9,421 of 9,536 generations dutifully copy.

Feuer turned that around by the end of the week. A series of Hugging Face upload and model-ingestion bugs were quashed, GPQA was patched, and the whole Tier-2 suite was rerun against a freshly pinned checkpoint identity, marin-community/grug-67b-a2b-sft-s2-thinking-step630 at revision 6808fe5c, with both serve jobs verified to pin that revision. All five Tier-2 tasks and all three AIME policy seeds now come back paired with zero empty responses, and generations mostly terminate naturally on hard problems. The headline effect is that Snowball's chat evals now improve with access to full context, as they should: MATH500 67.8 (+4.0), HumanEval+ 70.7 (+4.8), MBPP+ 65.3 (+2.6), GPQA-Diamond 49.8 (+4.5) and IFEval 27.2 (+3.2) against the retained short-window artifacts, with AIME24 the one task that reads lower at 17.3±0.7. Feuer is careful that the rerun changed both the context protocol and the chat template, so these are observed differences rather than an isolated context-length effect. The full bundle — 287 results files and 240 samples files, about 1.4 GB — is public as marin-community/grug-67b-a2b-snowball-post-sft-evals, with the protocol, policy and harvest script in a companion gist. Hall's recommendation stands regardless: separate reasoning from user-visible output and enforce a real stop condition before these numbers drive post-training or data-mix decisions.

The other substantial result is a negative one, and a well-controlled one. #7427 reports a paired-init A/B on Qwen3-30B-A3B-Thinking where the only functional difference between arms was DenseMixer's dense-forward router gradient: same seed, same data and order, same LR, 366 steps. The agentic in-distribution eval gave 0.0959 for the sparse control against 0.0589 for DenseMixer — nominally worse, not statistically significant — and a weight-space sweep over all 18,867 tensors and 6,144 experts explained why. Top-8 routing overlap against the shared init was 0.859 for dense versus 0.860 for sparse, dense-vs-sparse output divergence was a mean Jensen-Shannon of 0.0035, and of 178 experts near-frozen under sparse SFT, DenseMixer revived exactly zero. The dense gradient did reach the router and moved it in a distinctly different, more low-rank direction, but that re-steer never propagated to different expert selections or different outputs. The recommendation to the Levanter port #7088 is to keep it flag-gated and validate it first on a regime with real routing demand, rather than pay a ~16x dense-forward cost by default; Feuer read it in midtraining as clearance to unify Marin and OpenThoughts-Next SFT on Levanter. Alongside, the agentic post-training track kept building: the tools-aware opencode chat template and its Levanter chat-format wiring landed in #7455 on top of the tool-calling passthrough #7454, with rendered token IDs matched against Axolotl's strategy on all 60 sampled rows; a step-1903 Grug OpenCode SFT was registered on the unified eval launcher in #7606 with a measured serving contract proposed in #7595 (65k context, a 7,168-token prefill budget on 8xH100, and 32 sequences per shard); and #7445 closed out the reason agentic runs on the cw-rno2a cluster silently produced zero finalized trials — that cluster was federated without the public endpoint-serving ingress its sibling got, so opencode inside a Daytona sandbox could never call back into the served model. Looking further ahead, the QAT proposal #7404 names this run as its phase-2 target, recommending quantization-aware distillation rather than plain quantization-aware training for a model that has been through RL, since plain QAT can land below naive post-training quantization; Hall noted that collaborators at Bonsai could go further, to 1-bit or ternary.

The H100 validation work returned a second data point this week. Four multi-head latent attention (MLA) variants of the same 11B-A1.5B shape — d2048, 24 layers, 64 experts top-4, seq 4096, batch 512, datakit mixture, MuonH — ran on 32×H100 at CoreWeave and wound down on July 23–24. The primary arm grug-d2048-L24-mla-500B-r4-b512-datakit completed all 238,419 steps for the full 500B tokens and landed at a macro loss of 2.2875 (0.8014 bpb), a shade under the grouped-query attention (GQA) reference run's 2.3014 (0.8059) at the identical token budget. It paid for that in throughput: ~21.2% MFU against GQA's ~23.7%, or ~19.1k tokens/s per GPU versus ~21.4k, at effectively the same counted FLOPs per token (3.662e9 vs 3.657e9). The trade on this stack is roughly 12% throughput for 0.014 of macro loss, and MLA also carries slightly more total parameters (11.05B vs 10.77B). A second arm that scaled the Q and KV low-rank projections grug-d2048-L24-mla-500B-datakit-qkvscale reached 99.7% of the budget at 2.2882 — indistinguishable from the base.

The other two arms turned the 2048 sliding window off, setting the window to the full 4096 sequence, and they are the week's most useful caution about reading MFU. They report ~25% MFU against ~21% for the windowed arms, which looks like a large throughput win, but the FLOP counter charges them 4.41e9 FLOPs per token instead of 3.66e9 and actual throughput moved the other way: ~599k tokens/s versus ~613k. The sliding window removes about 20% of counted attention work at this shape and buys back only ~2% of wall clock, so MFU is not comparable across the two arms and the windowed figure is the honest one. Both no-window arms stopped short of the budget — at 93% and 91% of steps, within a few hours of the primary run completing — so their macro losses of 2.3083 and 2.3153 (the second also with the 1.3× query-logit multiplier turned off) are not a clean read against the completed runs.

None of this has reached the epic logbook yet #6716; the results live only in W&B, and the thread's last entry is still the d512 ablation tables from July 11. The GPU-tuning sibling that produced the MLA attention-kernel work — the asymmetric 192/128 FA4 CuTe forward and the native SM90 backward — closed out on July 16 with "moving testing to B200" #6979, and attention has followed the hardware: Larry Dial reported 25% MFU at 360B-A23B on B200 with Muon, Hyperball, attention gate, XSA and gated norm all enabled, and Jeff Hammerbacher asked what happens to that number across multiple racks.

The 20% bar this commitment set held and moved up. On the leading architecture candidate — a ~355B-total / ~20B-active MuonH model at d6144, 48 layers, top-4 of 128 routed experts plus 2 shared — Larry Dial stacked three increments onto last week's 23.2% single-rack result: splitting the shared expert in two, grouping same-shape non-expert Newton-Schulz calls, and a batch bump took it to 23.7%, and then profile-guided latency estimation (PGLE) added a further ~1.1 points for ~24.8% MFU / 276.8K tokens/s on one 64×GB200 rack, reproduced across two runs #7201. PGLE is the notable one because it is config-only: profiling showed the step exposed-comm-bound on the Fully Sharded Data Parallel (FSDP) expert-weight all-gather, with the FlashAttention-4 kernel stalling ~1.3 ms per layer on inputs it already had, and XLA's latency-hiding scheduler making the wrong overlap call from heuristic collective latencies. Turning the scheduler on without PGLE gained nothing — the win is feeding it measured latencies. Dial's read in #gpu was that with Muon, Hyperball, the attention gate, Exclusive Self Attention (XSA), and gated norms all in the config, he was not sure much faster was available in bfloat16.

What the commitment is now really about is the multi-rack number, since the run in question is a ~12-rack job and the bar was measured on one. Cross-rack cost is the dominant term: the same full-feature recipe runs ~23% MFU on one rack but 18.6% MFU / 412K tokens/s at 2 racks / 128 GPUs on the 707B 4-of-256 variant, where FSDP weight-gathers and Muon stack reshards cross InfiniBand instead of NVLink, and larger batch plus finer expert chunking (chunk-4 at batch 2304) recover only part of it. Projections to 12 racks with the standard 7% weak-scaling penalty land the best candidate near ~23% MFU / ~3.09M tokens/s, or about 75 days for 20T tokens — but Dial flagged that the first cross-rack hop empirically costs closer to 19% than 7% for these recipes, so those projections should be read as optimistic. Jeff Hammerbacher asked the direct version of the question — whether there are tricks left for the multi-rack setting — and Dial's answer was honest about not knowing yet, adding that speedups likely remain in bfloat16 but that the work is nearing the point where two days of effort buys one day of training time. The precision paths that would change that arithmetic, MXFP8 and native 4-bit, are tracked under B200 training MFU & perf #6710.

Externally the commitment now reads as met at the level it was written. In the Tuesday community meeting update, Percy Liang described the team as nailing down the architecture and getting MFU to a reasonable spot on B200s, with a 360B-A22B candidate on 20T tokens over 84 days at 20 MFU in bfloat16 with MuonH and FSDP, and the next big hero run launching in the coming weeks. The epic's own thread stayed quiet all week; its status lives in the tokens-per-second tracker #7201, which is where the architecture menu and every MFU measurement now accumulate.

The shape is still open, and every live candidate now sits well above the epic's nominal ~120B-A10B framing. Selection runs through the GB200 Architecture TPS Tracker #7201, whose shortlist this week became a d6144 top-4-of-128 at ~360B total / 22.6B active (~82 days for 20T tokens), a d6144 top-4-of-256 at ~707B / 22.6B active (~99 days), and a Multi-head Latent Attention (MLA) variant with full-width 72-head attention, full causal attention on every layer, and a learned relative-position bias replacing RoPE, at ~360B / 21.3B active and ~120 days — which the tracker itself calls an optimistic floor, since the first cross-rack hop empirically costs far more than the 7% weak-scaling penalty the projection assumes. Larry Dial noted on Discord that the 707B option has entered consideration, and gave the working read on sparsity: it will land between 4-of-128 and 4-of-256, with 4-of-128 most likely. Will Held flagged why that choice reaches past throughput — a sparser model memorizes faster and tolerates epochs less, so the data mix's epoching penalty moves with the sparsity decision. At the community meeting Percy Liang gave the outside view: one candidate is 360B-A22B on 20T tokens over 84 days at 20% MFU, MuonH, Fully Sharded Data Parallel (FSDP), bfloat16 compute, with the next hero run launching in the next few weeks.

Most of the week's movement was MFU on a fixed shape. David Hall landed per-long-layer KV heads with a static FA4 shape — one homogeneous 16-KV-head stack where global layers slice the first 8 projected heads and repeat each twice, numerically equivalent to 8-head grouped-query attention (GQA) but leaving no shape-changing conditional inside the differentiated scan — worth 238.5K tok/s at 21.55% MFU on one rack, 4.7% over the two-stack flat-conditional implementation. Larry Dial then stacked three increments on the d6144 4-of-128 baseline: splitting the shared expert into two half-width experts, shape-grouping the non-expert Newton-Schulz calls, and profile-guided latency scheduling (PGLE). Together they took 23.17% to ~24.8% MFU and 276.8K tok/s, or ~75 days for 20T tokens. PGLE is the single biggest lever at +1.1 points and it is config-only: profiling showed the step exposed-comm-bound on the FSDP expert-weight all-gather, with the FA4 attention kernel stalling ~1.3 ms per layer on inputs it already had, because XLA's latency-hiding scheduler picks the wrong overlap from heuristic collective latencies — enabling that scheduler without PGLE buys nothing. Larry Dial's read at 360B-A23B was that with Muon, Hyperball, attention gate, XSA and gated norm all included, much faster may not be possible; David Hall's reply was that there is no David-LLM without dial_moe. Jeff Hammerbacher asked whether multi-rack has the same headroom, and it does not: the same tunings bought only +1.0 point at 2 racks, where cross-rack InfiniBand traffic dominates and PGLE can reschedule the weight-gather but not shrink it. Larry Dial's answer was that speedups likely remain, but the point is approaching where two days of effort buys one day of run time.

The bigger reversal was on parallelism strategy. Last week FP8 read as the leading throughput lever; Matt Wittmann's matched d6144 A/B retired that for now — same source, same config, only the MXFP8 flag changed, and MXFP8 came in at 0.749× bf16 throughput, a regression he attributes to quantization, cast and lowering overhead rather than to the tensor-core arithmetic. Native FP4 was ruled out in the same spirit: David Hall closed #7403 saying he strongly thinks we should stay away from NVFP4 without a very good reason to take on the risk, with Jeff Hammerbacher noting the issue was filed by an over-eager agent. Meanwhile expert parallelism (EP) went from worst option to best. The EP64 top-8-of-256 path opened the week at ~13% MFU with all-to-all dispatch and combine eating ~28% of GPU time, reached 17.8% on a fixed-capacity all-to-all, then 20.558% once the repeated-activation scatter was replaced by an int32 assignment scatter plus an activation gather. Matt Wittmann then flagged a fidelity problem in that whole lineage, tracked in #7279: those runs had aux-loss-free quantile balancing (QB) routing off by default, and the router collapses — 85–89% of routed assignments dropped early and oscillating 17–79% over 120 steps, invisible in the loss curve because the always-on shared expert keeps loss descending, and heavy-drop runs read higher MFU because a dropped assignment gathers a zero pad row and does no real work. With QB on and a custom scatter-add adjoint, EP64 on one rack now beats the best FSDP figure at both shapes measured: 24.594% at d6144 4-of-128, or ~23.9% adjusted to a ≤3% drop bar against FSDP's 23.1%; and 20.708% at d5120 8-of-256 while dropping only 1.44% of assignments, against 19.2%.

On the token side, the mix is converging on roughly 15T tokens after dedup from 23T pre-dedup. The Stack v3 corpus #7558 was filed and closed within two days via #7564, which also retires the Stack v2 sources it supersedes. The dedup false-positive cascade #6854 got a real fix in #7591 — word 5-grams instead of character 5-grams, plus canonical-star candidate marking to stop transitive closure from fusing unrelated documents — and is under a full 100B-token A/B over 103.7M documents that has already cut MinHash CPU by 70% and connected-components rounds from 53 to 24; the exhaustive semantic review is about a quarter through 755,281 disputed pairs, with the baseline arm running ~63.5% false positives against the treatment's ~51.7%. Decontamination picked up a global document-frequency filter for cross-source boilerplate, #7126 into #7635. What has not happened is the preregistration: #6702, which would file the predicted loss before the run starts, is still an empty issue, and the hero-run tracker #6689 still carries blanks for FLOP budget, token count, data mix and pre-registered loss. Work continues toward the Aug 1 target — an ambitious goal set when the candidate shapes were roughly a third their current size — with sparsity, hidden dim, expert count, EP-versus-FSDP, and the tokenizer, where Larry Dial asked where things stand for the next run, all still to converge before a loss can be predicted.

The budget itself did not move: #7074's own thread was quiet this week, and Benjamin Feuer's pricing stands — roughly a billion to ten billion post-training tokens, but each costing about 700× the hardware FLOPs of a pretraining token, so 0.02% of the tokens turns into ~14% of the FLOP budget. The action was on the on-policy distillation (OPD) teacher-serving RFC #7236, which sizes the OPD leg against this cost model. Jeff Hammerbacher put two papers in front of it — Lightning OPD, which Benjamin Feuer called worth a try and offered to take to the OpenThoughts group, and then AsyncOPD — and posted a research writeup on both.

The writeup's load-bearing finding is a correction to the RFC's own premise. #7236 describes MarinSkyRL's OPD as top-K sparse forward-KL at K=256, but that is the library function, not the live objective: the trainer sets rewards = -(action_log_probs - teacher_action_log_probs), a k1 estimator of reverse-KL, while the top-K forward-KL tensors are computed and never touch the gradient. That matters because AsyncOPD's headline result is precisely about KL direction — forward-KL tolerates stale rollouts, reverse-KL does not — so Marin's real objective is the staleness-vulnerable one, and decoupling teacher scoring from the optimizer step means either wiring up the already-present forward_kl_from_sparse as the live loss or adopting AsyncOPD's reverse-KL fix of recomputing the advantage under the current student. The writeup also notes that Marin's own K ablation in #6218 already puts k=32–64 near-lossless, which lines up with Russell Power's read that top-K is safe and makes the full-vocab bandwidth question largely moot — though the safe-K ablation is still the decision that gates the serving choice.

For the budget, the two papers land differently. Lightning OPD's 3.6–4.0× comes from precomputing teacher logprobs offline and giving every GPU to the student, but it requires the SFT-generating model and the teacher to be identical — which the multi-teacher path this epic prices, RL over K experts distilled into one student, structurally violates. Its win therefore applies to the single-teacher cases: strong-to-weak distillation of the smaller sizes from the flagship, and the off-policy SFT-distill fallback. AsyncOPD's 1.6–3.8× is the one that touches the flagship leg, and it also de-risks the RFC's preferred permanently hosted teacher endpoint by taking teacher tail latency off the critical path. Neither multiplier has been folded into the headline number yet. Adjacent plumbing is stirring too: Rohith Kuditipudi asked whether there is a canonical logprob-scoring codepath, noting Levanter's support lacks batching and the old save_logprobs path is stale — exactly the plumbing a teacher-logprob oracle would sit on. And the quantization-aware-training-during-SFT proposal #7404 would add a 0.3–3B-token pass on top of the post-training budget, a rounding error against it, with David Hall noting the Bonsai group could push to 1-bit or ternary.

The GB200 MFU ledger moved on two fronts at once. On the Fully Sharded Data Parallel (FSDP) line, Larry Dial stacked four increments onto the d6144 4-of-128 baseline in the architecture tracker #7201 — splitting the single shared expert into two half-width ones, shape-grouping the non-expert Newton–Schulz calls into one, raising batch 1024 → 1152, and turning on profile-guided latency estimation (PGLE) — taking one rack from 23.17% to ~24.8% MFU / 276.8K tokens/s. PGLE was the single biggest lever at +1.1 points and is config-only: profiling showed the step exposed-communication-bound on the FSDP expert-weight all-gather at ~30% of the step (bf16[64,6144,6144], ~4.8 GB per chunk), with the FlashAttention-4 kernel stalling ~1.3 ms per layer on inputs it already had. Enabling XLA's latency-hiding scheduler without PGLE gives nothing — the win is feeding it measured rather than heuristic collective latencies. Projected to twelve racks that is ~23.1% MFU and ~75 days for 20T tokens, down from ~82. Dial reported it in #gpu as 25 MFU at 360B-A23B with Muon, Hyperball, attention gate, XSA and gated norm all included; when Jeff Hammerbacher asked whether the same tricks carry to multiple racks, Dial allowed that speedups likely remain in bf16 but that “we will soon approach the point where it takes 2 days of effort to find a speedup worth 1 day.” The same tunings lifted a newly added 707B-total 4-of-256 two-rack option from 18.6% to 19.6%, and David Hall landed heterogeneous per-layer KV heads — 16 local / 8 global — at 21.553% by repeating logical KV heads into one static FA4 shape instead of differentiating through a shape-changing conditional, 4.7% faster than the two-stack version and clean over all 30 steps.

Expert parallelism (EP) closed the gap and then passed the FSDP line. Rafal Wojdyla took EP64 from ~13% typical to 17.8% MFU / 276.1K tokens/s by replacing the ragged collective with a fixed-capacity jax.lax.all_to_all and landing the three sharding fixes filed as #7512, #7513 and #7514 — preserving expert sharding through MuonH's stacked 4D Newton–Schulz, sharding non-expert weights and optimizer state over data × expert, and exposing the receive capacity factor — then to 20.558% by rebuilding dispatch as an int32 assignment scatter plus an activation gather rather than an 8× bf16 row repeat. Matt Wittmann then wrote a custom adjoint for both gathers, whose transposes have exact structured forms computable from the same int32 index composition the forward already builds: the backward HLO drops from 544 scatter ops to zero and one-rack p50 MFU goes 20.61% → 24.04% on a matched 120-step A/B, with leg-batched expert GEMMs reaching 25.39% separately #7279. On both architectures measured, EP64 now beats the best previously reported FSDP figure — 24.594% against 23.1% at d6144 4-of-128, and 20.708% against 19.2% at d5120 8-of-256. The more consequential result is the audit Wittmann ran on his own numbers. Those benchmark configs route with aux-loss-free load balancing off, and an exact drop counter shows the routed experts collapsing: 85–89% of assignments dropped early in training, oscillating 17–79% over a full run, invisible in the loss curve because the always-on shared expert keeps loss descending. Because dropped assignments gather a zero pad row and do less real work, a run that drops more reads higher MFU, so every cross-regime MFU gap is an upper bound. Turning balancing on costs 1.44 points; the fix that recovers most of it is same-step spill — re-offer a blocked assignment to the next-ranked expert that token already chose — which halves drops for 0.213 points, cheap for a structural reason: expert GEMMs run on capacity-sized buffers whether or not a slot is filled, so spill adds index work and never matmul work. Best measured compliant point is spill m=3 at capacity factor 1.0625: 20.708% at 1.44% drops. Two of Wittmann's own earlier claims were corrected in the same thread — a compliance conclusion that turned out true only by accident, and a drop difference reported as draw variance that was actually LR-schedule position — with the rule now adopted that drop fractions are only comparable at the same fraction of the schedule. A second measurement closed off a whole family of ideas: with collectives moved onto the async stream, exposed collective time (4.29 s) almost exactly fills compute idle (4.44 s of a 33.2 s span), so the step is collective-volume-bound, not schedule-bound — rotation, prefetch, token-chunk pipelining and the overlap limit all measured null or negative, leaving fewer collective bytes as the only remaining lever.

The intermittent GB200 CUBIN loader failure that has been taxing this work for weeks is now root-caused and fixed, and #7421 is closed. Wittmann first ruled out the obvious: 1.3M exact module loads across 16 nodes with live NCCL passed clean, serializing every cuModuleLoadFatBinary call behind one mutex still failed, and one-process-per-GPU with strict CUDA_VISIBLE_DEVICES isolation still failed — while a null-classification probe showed all 64 failing calls received a null image, meaning the driver was rejecting the argument, not the binary. Decoding the persisted executables found exactly one zero-byte custom-kernel CUBIN, a Triton fusion normalizing the MoE routing weights, present in both failing artifacts and absent from both passing controls. The cause is an OpenXLA kernel-reuse cache collision: TritonFusion and MlirKernelFusion shared an empty discriminator, so an MLIR fusion could hit a Triton entry — whose binary is intentionally empty because it compiles later — and serialize that emptiness as an owned CUBIN. The discriminator fix is OpenXLA 4c1b005, which Wittmann confirmed fixes every existing reproducer and which arrived in-tree via the JAX 0.11.0 bump #7436; Dial's reaction on the issue was “Another one down :)”. Removing the mask promptly clarified what it was hiding: the hetero-KV nested scan in #7407 now fails with an honest NCCL out-of-memory, and peaks 18.7 GiB/device above the uniform single-scan path after rematerialization. Related stack work: #7587 moved CUTLASS DSL to 4.6.0 with a 64-GPU full-stack run at 23.106% MFU, superseding the wheel-shadowing patch #7491; #7483 traced THD FA4's full-causal and sliding-window failures to the CuTe DSL tracing both arms of a plain if, fixed in #7630 by resolving window sizes before tracing; and Hall's JAX-Toolbox evaluation #7519 / #7524 measured 25.60% against 25.83% and 25.71% controls — a 0.505% time-adjusted regression that passes the proposed 2% adoption gate but gives no performance reason to switch, while the unrolled 48-layer model still asks XLA for a 905 GiB temporary arena that block-boundary optimization barriers move by 128 KiB. On the operational side, Wojdyla fixed #7431 in #7432: gang jobs on the k8s backend had no wall-clock timeout enforcement at all, so gangs wedged in uninterruptible NCCL waits held whole GB200 racks for hours and fragmented the NVLink domains until new two-rack jobs could not schedule.

On precision, the preregistered MXFP8 quality gate #7271 completed and answered its stated hypothesis in the negative. Its registered claim was that MXFP8 training on Blackwell is quality-neutral; the matched pair — 31,474 optimizer steps and 66.006B tokens per arm, same seed, data order, schedule, optimizer and 32×GB200 topology — delivered +7.22% throughput and 91 minutes of wall time, but also +0.056% aggregate eval, +0.110% Paloma and +0.209% uncheatable loss, with aggregate eval favoring BF16 at all 32 paired evaluations. The honest verdict is a speed-quality tradeoff, not a free lunch, and the recipe is hybrid by necessity: uniform MXFP8 for dense GEMMs misses the throughput gate in Marin's implementation and aborts before execution in TransformerEngine's. Getting there required fixing a first-backward NaN that only appeared when grouped MXFP8 and dense per-tensor FP8 were combined #7480. Wittmann also retracted this epic's headline MXFP8 number from two weeks ago: a same-commit EP1 pair on the production d6144 config measured MXFP8 at 0.749× BF16 with the wire disabled, so the earlier +34.8% was confounded on expert-parallel topology and on Newton–Schulz being disabled in the control #7282. Reconciling that against the FP8 wire result in #7279 produced the week's new preregistered experiment #7665: the two threads never conflicted, because each disabled exactly the half the other tested — quantizing is pure overhead when nothing downstream consumes the FP8, and pays for itself when an MXFP8 grouped GEMM does. Its declared success criterion is a positive layer-level A/B inside the real step with rematerialization on; early results show the fused wire running on GB200 with byte-identical forward operands, plus two silent-corruption traps found and pinned as tests — a uint8 payload has a float0 tangent, so JAX drops the dispatch cotangent without raising, and a float8 payload downcasts a bf16 cotangent to unscaled e4m3, flushing small gradients to exactly zero. Finally, the TransformerEngine NCCL_EP evaluation #7331 reached its conclusion after a fairness correction: NVIDIA's recommended full fused MoE block ties Marin's own transport seam at 16.94% versus 17.15%, both ~1.1–1.3 points behind the incumbent a2a_cute, and every remaining knob — scoped command-buffer capture, the collective-overlap flag, an SM-budget sweep — was a wash or a loss, so a2a_cute stays incumbent for bf16 while NCCL_EP is preserved as a derisked path.

Larry Dial spent the week looking for a setting in which multi-head latent attention (MLA) beats grouped-query attention (GQA) on the d1024/d2048 ablation ladder under parent #7374, and did not find one. The learning-rate sweep #7381 closed out its d2048 arm: log-quadratic fits put GQA's optimum at 0.98× the heuristic peak LR and MLA's at 0.91×, so MLA consistently wants a step about 7% smaller, and the heuristic itself slightly overshoots at both widths. GQA edged MLA on Paloma macro loss at both sizes; the quality gap did narrow with scale (+0.0068 at d1024 → +0.0028 at d2048), but MLA's throughput penalty from its 192-dim query–key heads widened over the same span (7.9% → 16.1%), so GQA's real wall-clock advantage grew, +12.5% → +19.8%. The GQA→MLA transition staircase #7423 then walked one baseline into the other across nine runs, one architectural change per step, to localize where the loss actually lives: dropping QK-norm (query–key normalization) costs +0.0199 Paloma and dropping rotary position embeddings entirely (NoPE) costs a further +0.0653, while re-introducing MLA's decoupled rotary component and 192-dim queries at step 7 buys back −0.0891 in a single move. The uncomfortable part is the middle: inserting the KV latent and then widening to full multi-head attention each hurt slightly despite strictly adding expressiveness, which Larry Dial read as a failure of the optimization process rather than of the architecture.

Two follow-ups interrogated the latent directly. A weight-space analysis of the learned KV down-projection #7424 found effective rank below a random-Gaussian control at every layer, U-shaped in depth (most compressed at the first and last layers, near-full-rank mid-network), with layers sharing their dominant read directions at roughly 5.5× chance and an overlap matrix that is band-diagonal — real structure, but mild. The analysis also surfaced a checkpoint-loading trap worth knowing about: the standard full-state restore hard-fails on optimizer-state mismatch and silently falls back to random init, so the first pass characterized untrained weights that looked exactly like the Gaussian control; the fix was a partial param load plus a per-leaf probe. The ablation that followed #7425 replaced the down-projection with a fixed slice of the residual stream — no matrix at all — and matched the baseline on train loss (2.1569 vs 2.1570) for +0.0031 Paloma, a generalization gap rather than a fitting one, while a frozen orthonormal projection cost +0.0042 and a frozen Gaussian +0.0087. At this compression ratio and scale the learned latent is not earning its keep, and Larry Dial's own reading is that d1024 may be too narrow to get MLA signal at all. The next item on the same runbook is a learned content-conditioned relative-position bias replacing partial RoPE #7426, staged prototype-first because the real run needs the CuTe FA4 kernel extended to accept a banded additive bias inside the flash loop.

The week's clear positive came from the other direction. The Inkling-style short causal convolution (SConv) sweep #7585 completed its full d1024 matrix on the datakit mix, and every conv variant beat the no-conv baseline. Full Inkling — depthwise kernel-4 convs on K, V, attention output and MLP output — reached 3.0378 Paloma macro against 3.0610, which the scaling law turns into a 1.186× equal-throughput effective speedup, or roughly 1.11× net once the measured ~1.5 MFU-point throughput cost is charged. Depth 3 is effectively tied with depth 4, depth 2 still beats every single-site arm, and among single sites the MLP-branch conv carries the most (+7.9%) ahead of K (+5.1%), contrary to Inkling's framing that the K conv is the load-bearing one; convs restricted to the global layers capture about half the gain at a fraction of the cost. The same batch retested Partial Key Offset (PKO) on the new datamix and found it context-dependent — +5.0% with global-every-4 layers and +5.9% at sliding-window 2048, but −2.9% in the pure all-local partial-RoPE config. Larry Dial's reading is that PKO's earlier gain was largely an induction-bias artifact of the old eval/train mismatch, and that convolution captures a similar benefit more robustly; initializing the conv to mimic PKO does make induction heads form by around step 100, but the head start washes out and init stops mattering at scale.

On the TPU-side agent ladder, Kaiyue Wen converted two of last week's open questions into sealed results. Over-Encoding's Gate 2 formulation #7368 — lower table LR, full-rank SparseCore gradients and a norm-summed embedding combine, all measured against an exactly reproduced canonical July baseline — held its lead the whole way to step 16,079 through a mid-run TPU preemption and automatic recovery, finishing at 3.0084 Paloma vs 3.0234 at d1024 for 1.090× effective speed after its 2.7% throughput cost (1.168× at d512, 1.131× at d768), reversing the original Gate 1 negative. Identity Hyper-Connections #7409 also cleared Gate 1 once coefficient-only telemetry overhead was stripped: accepted profile overheads of 7.04% at d512 and 10.20% at d768 under an 11% ceiling, terminal Paloma improvements of −0.0225 and −0.0183, and net wall-clock speedups of 1.0515× and 1.0205×. She traced the idea to a Chinese technical blog in the architecture channel and later noted that Qwen appears to be using it. The SiTU activation #7255 was formally written up as a negative gate at 0.76× and 0.72× effective speedup. Around the experiments, the scoring machinery itself got attention: David Hall opened a PR publishing a generated Agent MoE experiment digest #7623 that deliberately separates loss-only model-FLOPs speedup from measured wall-clock speedup so throughput regressions stay visible, and Will Held pointed newcomers at the same agent.md procedure the ladder runs on. The native FP4 pretraining proposal #7403 was closed after David Hall pushed back hard on taking NVFP4 risk and Jeff Hammerbacher noted his agent had filed it over-eagerly alongside the intended quantization-aware-training issue #7404. Kaiyue Wen also froze the archived Qwen3 speedrun ladder into a standing optimizer benchmark #4919, with published submissions for AdamC #4921, AdamH #4922, MuonC #4923, MuonH #4924, MuonRemez #4925 and PRISM-Berkeley #4926 — the MuonC boundary sweep extended upward and confirmed lr=0.012 still wins. Older tracker items were retired in the same pass: paired-head attention #4907, depth-MuP LR sensitivity #5178, ArrayStacked backward-flow logging #5030 and downstream-eval scaling #4550. Separately, Russell Power's Qwen 32B→0.6B distillation screen #7656 finished all 18 cells at a 100M-token budget, with factorized teacher initialization the only research treatment to clear the promotion rule (2.2979 mean held-out NLL vs 2.3315 for scratch forward KL); ten 1.8B-token extension runs are now going on CoreWeave.

The infrastructure-as-code (IaC) migration that started last week finished its first phase: all four CoreWeave clusters are now owned by Pulumi. Will Moss got cw-us-east-02a, cw-rno2a, and cw-us-east-08a previewing clean and stripped the Iris controller of the code that recreated role-based access control (RBAC) objects and NodePools on every start #7471, so the two managers can never fight over ownership again — the controller now only verifies those resources exist and fails fast naming what is missing. Adoption itself landed in #7502, closing both the original "should we try Pulumi" question #6879 and the umbrella issue #7213 that tracked collapsing a nine-step manual runbook down to pulumi up. Adoption also surfaced a live production bug: cw-rno2a's Kueue controller manager was being OOMKilled against the upstream chart's fixed 512 MiB default, which does not scale to that cluster's roughly 2,000 Kueue Workload objects — fixed first as a per-cluster opt-in, then as a fleet-wide 2 GiB reservation #7602. Pull requests touching infra now get a pulumi preview plan posted back as a comment #7532, under a dedicated identity that can read state and decrypt stack secrets but write neither; CI never runs pulumi up. Loom, now serving as Marin's agent router, moved onto the same footing #7528, and the remaining hand-managed surface is enumerated rather than tribal: GCP identity and access management #7576, storage buckets and lifecycle rules #7580, GCE firewalls #7579, the finelog deploy #7571, and a catalog of the four genuinely different Pulumi patterns infra/ has accumulated #7566.

Two concrete control-plane failures drove the reliability work. On July 22 Benjamin Feuer asked in Discord whether finelog was down; Russell Power replied that the cw-rno2a Iris proxy was overloaded and rebooted it with a fix. The diagnosis in #7533 found it was not an out-of-memory kill at all: event writes were waiting more than 11 seconds during a resync of roughly 4,843 pods and 3,080 Kueue Workloads, Kueue missed its 10-second leader-renewal deadline, and the process exited. Kueue's Kubernetes client had been running on its stock 20-queries-per-second limit; the Iris-wide default is now 100. The same change caught a quieter regression — the launcher installing the controller's 64-thread RPC executor built a selector event loop directly, silently disabling the uvloop already present in the controller image. On cw-us-east-02a roughly 180 long-lived inference streams shared that process with Kubernetes list decoding that reached 6.1 seconds, and even loopback health checks timed out; restoring the configured loop factory cut health-probe p99 at 1,600 concurrent streams from 8.85 to 2.19 seconds. Finelog's Kubernetes mirrors had a related pathology #7540: liveness began after 15 seconds while reopening a network-backed store could take over a minute, so one slow recovery became a restart loop — cw-rno2a logged six liveness restarts and 79 readiness failures in about three hours while row writes stretched to 8–24 seconds under multi-million-row compactions. Reserved CPU and memory plus a five-minute startup probe fixed it, with a companion alert now watching mirror readiness and hub query latency #7536. Grafana absorbed most of this signal: a restored high-density infra cockpit #7469, GB200 rack-tray inventory that alerts when a rack drops below 16 trays #7521, and a fleet jobs dashboard #7609 that also fixed a finelog federation bug leaving CoreWeave-forwarded Iris stats with no origin cluster attached. An apparently stalled GB200 controller on cw-us-east-08a turned out to be healthy behind 6.2–6.9-second Kubernetes node-list calls and a NodePool exposing 201 of 216 nodes; that one is recorded rather than resolved #7608.

The controller's endpoint proxy was rewritten in Rust. Benjamin Feuer reported that federated CoreWeave capability URLs — the per-endpoint links an agent running outside the cluster uses to reach a job's served model — had started hanging while the backend was demonstrably healthy in-cluster #7448. The cause was a single 100-connection HTTP pool shared across every peer and endpoint: long-running inference streams held the slots and unrelated capability requests queued until the 600-second budget expired. Removing that cap #7479 was the immediate fix, and Benjamin Feuer noted a typical RL run can have around 1,024 requests alive at once. The structural fix replaced the Python proxy with a Rust listener that owns the public port outright #7537, with Python publishing one generation-checked endpoint delta per committed database transaction instead of copying thousands of unchanged mappings; at 1,600 concurrent streams the controller's own CPU time fell from 5.2 seconds to effectively zero. Two follow-ups made the federated path usable from outside the fleet: a child cluster's minted capability URL now relays through its public parent #7627, under a /proxy/t/cluster=<peer>/ grammar that stays inside the existing unauthenticated ingress rule #7634 rather than bouncing headless callers into a Google sign-in flow — validated with a literal credential-free curl reaching an H100-served model. Capability tokens can now be scoped up to a week #7561, answering Benjamin Feuer's ask for datagen and eval jobs that outlive the 24-hour default #7551. The native extension also got a real release path: nightly wheels and fast dev rollouts #7573, and one workflow publishing all twelve Marin distributions #7664 so Python CI can no longer sit on an older native implementation than the source it claims to test.

The week's other theme was making failures explain themselves. Debugging a cw-us-east-02a job loss took Russell Power about an hour, almost none of it inside the Iris CLI: job summary reported failed exit=0 failures=1 with an empty error column, while the real cause was a 404 bundle fetch in the pod's init container, reachable only by reading Python to derive the pod name and then running kubectl. #7543 turned that into a proposal for a fixed operator vocabulary — eight nouns, eight verbs, each verb meaning one thing everywhere — and #7546 landed the incident slice: task describe and attempt describe, per-attempt pod identity and a bounded terminal reason persisted at creation so a failed-in-init attempt still explains itself after its pod is gone, and read-only commands that no longer require the controller extras. The rest is tracked in #7560. A second incident sharpened the same point: Benjamin Feuer's four-node gang ran healthy for 4.9 hours and then went terminally FAILED #7652 because Kueue had preempted it to admit an interactive job, stamping WorkloadEvictedDueToPreempted; Iris recognized only Kubernetes' own DisruptionTarget condition, so the resulting exit 137 was charged as an application failure and the job finalized before a retry could start. #7661 classifies Kueue evictions as worker failures and records controller decisions in a durable seven-day task-event stream. Smaller sharp edges went the same way: job kill now matches exactly by default after a prefix match took out a bystander run #7478, and a resubmitted job no longer adopts the previous attempt's terminal pod and reports its stale log as the new failure #7518. After Larry Dial's v4-2048 MoE run died three times at random multi-hour marks on a coordination-service socket close #7430 and he asked for something that auto-retries rather than losing seven hours whenever nobody notices a run has stopped, the answer turned out to be simpler than a supervisor process: stop running with a zero failure budget. Shared Grug runs now default to ten task failures and the v4-2048 hero runs to a hundred #7451, letting Iris's existing retry path reschedule the gang.

Serving collapsed into a single lifecycle this week. Russell Power replaced quick-serve and the separate brokered vLLM launcher with one path for both vLLM and Levanter #7452: marin-serve local starts the selected backend on the current host, and marin-serve iris submits it through Iris, choosing direct transport for a single instance and brokered transport for several. Broker transport now carries raw request bodies, query strings, allowed headers, and response status and headers across arbitrary /v1 routes; brokered requests stay buffered and reject streaming outright, with the shared dashboard switching clients based on endpoint metadata. Two scheduling details matter more than they sound: broker coordinators drop the CLI region constraint and broker workers carry Fray's explicit ANY_REGION marker, so an accelerator worker lands wherever matching capacity exists rather than inheriting the coordinator's eventual region, while direct serves stay region-pinned. A day later he traced the first CoreWeave H100 failure to object-store addressing — the default 14-day model cache rewrites Hugging Face ids into regional s3:// paths, and CoreWeave rejects Run:ai's default path-style ListObjectsV2 with PathStyleRequestNotAllowed — and forcing virtual-hosted addressing #7487 brought Qwen/Qwen3-0.6B to READY on a live H100.

The larger speed win was refusing to recompile. Persisting vLLM compilation artifacts in a manager-owned archive backed by region-local 30-day temp storage #7615 gives workspace, TPU, and CUDA launchers the same JAX/XLA, vLLM, TorchInductor, Triton, and CUDA cache environment. Restoring validates a content-addressed generation before extracting into a deterministic local path — necessary because vLLM folds its compiler-visible cache root into internal Torch cache keys — with a host-local lock against concurrent servers and a 120-second deadline that degrades to a cache miss so serving stays available. On an H100 the cold run published 509 files (42.6 MB, 5.8 MB archived) and the warm run restored them in 2.0 seconds, cutting torch.compile from 12.78 to 2.32 seconds, engine initialization from 26.69 to 6.37 seconds, and manager start-to-ready from 59 to 30 seconds. A region-pinned v5litepod-4 comparison in europe-west4 published 219 XLA artifacts and saw the sharper end of it: vLLM-attributed compilation fell from 73.78 to 23.56 seconds, engine initialization from 118.15 to 27.30 seconds, and start-to-ready from 199 to 95 seconds. For a fleet that stands endpoints up and tears them down per eval or per rollout batch, that is most of the fixed cost of a serve.

Weight loading was the week's reliability theme. Benjamin Feuer filed the failure mode #7525: concurrent CoreWeave evals loading an s3:// checkpoint through the Run:ai model streamer hit a mid-load File access error, the vLLM subprocess exited, and the eval failed permanently — while staggering the same launches succeeded. Romain Yon picked it up and measured it before fixing it: 41 of 213 eight-way tensor-parallel (TP=8) loads on cw-us-east-02a failed that way over four days, about 19% per attempt. Since libstreamer reports no status code, #7545 retries startup up to three times with jittered backoff only when the child's logs show that specific read fault — a hang, an OOM, or a bad config still fails on the first attempt — with all attempts sharing the existing startup-timeout budget, leaving under 1% residual failure. A parallel thread went after the same errors from the storage side, defaulting GPU serves to materialize weights on pod-local NVMe rather than the regional mirror while TPU keeps its 14-day cache #7548 #7549; both are folding into the consolidated #7552. An idle-timeout option for Iris endpoints, counting only non-GET inference requests so readiness and /v1/models probes cannot keep a slice alive, is parked on a retained branch after the same consolidation #7553.

Two correctness bugs in the cache layer surfaced through the eval lane. A nightly caught that marin-serve had begun advertising the served model under its resolved weights path instead of the requested id, so a client asking for the name it asked to serve got a 404 #7636; the regression window was a single nightly build, where a refactor collapsed the old model and model_path fields into one weights field that resolution then overwrote. The fix captures the requested id before resolution and was checked live on a v5litepod-4 both cold and on a mirror cache hit, with vLLM loading from the cache path and answering to the friendly id #7637. Still open is the subtler one: the model cache keys on the reference string, so a bare org/model id never notices that the repository's main moved #7632 — one reproduction served a pod whose staged files lacked the repo's current generation_config.json, and vLLM generated straight past the literal end-of-turn token into another assistant turn. Alongside those, a skill for querying a finished serve's durable metrics landed #7577, giving two queries — lifetime requests, tokens, throughput, and mean latency, and whether requests queued or the KV cache filled — plus the three ways those numbers come back plausible and wrong: series identity is job, worker, and attempt rather than job alone; vLLM counters are cumulative per replica and must be differenced before summing; and histograms arrive as separate bucket, count, and sum families. The stack it all serves is holding up: reporting on the Delphi 1e23 runs, Benjamin Feuer noted they ran fully on Levanter, Iris, and the Marin TPU and GPU clusters at 512 rollouts per step across 32 H100s, generating roughly 2.2–4.8k tokens per second whole-step.

Last week Snowball served on both GPU and TPU; this week the work turned to proving that what it serves is numerically the same model on each. Romain Yon opened a digest-bound Snowball parity matrix #7534 that takes the 64 representative Levanter-GPU logprob cases from #7314 as the canonical reference and scores six serving cells against it — native and export-loaded Levanter on GPU and TPU, plus vLLM on each — through 32K contexts, while making Snowball checkpoint restore and BF16 export fit on a v6e-8 and verifying byte-identical GPU/TPU weight exports. It fails closed on drift in prompts, checkpoint, config, fork source, capture code, dependencies, runtime, and compiler policy. That draft was replaced the same day by #7556, which rebuilds the same work into a reviewable commit stack. Requalified on real hardware, five of the six cells agree closely at every context length: native GPU against the GPU golden and export-loaded TPU against the TPU golden are exactly zero, and export-loaded GPU, vLLM GPU, and native TPU all stay within 0.028–0.073 maximum probability error. The sixth — TPU vLLM against the TPU golden — is 0.030 / 0.044 / 0.083 at 256 / 1K / 4K but jumps to 0.586 at 16K and 0.437 at 32K. By the end of the week #7556 had been deliberately narrowed to just the two gates that are ready to defend — vendored June Levanter TPU golden revalidation, and TPU vLLM served against that golden with tensor parallelism (TP) 8 — setting aside the matrix, the exported-Levanter path, and the levanter.models.snowball memory work for later. One qualification failure is worth recording as a pattern: the first distinct-slice holdout reproduced every discovery maximum exactly and still failed closed, because the frozen contract had captured the v6e-8 resource shape while JAX reports the hardware identity as TPU v6 lite, and the manual invocation omitted three production JAX/PJRT selectors. Adding them reproduced the runtime digest bit-for-bit.

The deeper result is #7554, which explains why the GPU gate's rank sentinel has been flaky since #7354: the gate asserts a quantity the stack does not hold stable. Serving Snowball under data parallelism (DP) 8 with expert parallelism (EP), the same prompt gets a different next-token distribution depending on which rank serves it. A model-free reproducer on the production communicator shows the ring reduce-scatter returning eight different results for one identical sum — each destination sums its chunk in a rank-rotated order and rounds on the bf16 store, so about one unit in the last place (ulp), confirmed by max deviation scaling exactly 1×/8×/32× with operand magnitude. That rounding is then amplified by routing: a ~1 ulp perturbation flips the router's top-4 by layer 14 and reaches five distinct expert sets by layer 25, which is how a rounding difference becomes a 0.03–0.29 probability swing. A destination-independent combine (all-gather partials, sum in rank order in fp32) removes 85% of the spread at 128 tokens but as little as 5% at 15,025, because multi-chunk prefills already diverge upstream of the combine — and in one arm the better-conditioned arithmetic moved the sentinel further from the cross-hardware golden. A separate launch-scoped effect shifts per-rank values between a few discrete states that recur to six decimals across jobs, nodes, and days; of three observed states, two fail the 0.075 bound and one passes. The proposed landing is a test-contract change rather than a kernel fix: keep the 64-case golden assertion, replace the sentinel's cross-rank golden assertion with a same-rank determinism check, and log rank spread instead of asserting it, with the collective and local-kernel behaviors filed upstream. The write-up ships with its jobs, an R2 log archive, and an analyzer script so every table can be recomputed from raw blocks.

The TPU long-context divergence is a different animal and got its own issue #7555. Three clean v6e-8 processes produced bitwise-identical observations for all 64 prompts under DP1, with greedy agreement 16/16 at 256 and 4K but only 3/8 at 16K and 4/8 at 32K — repeatable across processes and slices, where the GPU path varies by rank and launch, so a shared root cause has not been established even though MoE routing plausibly amplifies small numerical differences in both. The 0.70 and 0.50 bounds the parity gate carries at 16K and 32K are explicitly compatibility smoke, not a parity claim: they catch regressions while #7555 works out where the first material divergence appears. Elsewhere, David Hall noted on the quantization-aware training proposal #7404 that collaborators at Bonsai could take Marin down to 1-bit or ternary weights, a more aggressive target than the W4A16 default that issue recommends for a bandwidth-bound MoE decode. And in #code-talk, Rohith Kuditipudi asked whether there is a canonical codepath for logprob scoring, observing that Levanter's support lacks batching and the older save_logprobs path looks stale — the same currency this epic's parity gates are denominated in. The three sub-issues stay open: correct TPU inference #6868 now has a gate around it, correct GPU inference #6869 has a gate whose contract is being rewritten, and the GPU performance target #6870 is still waiting on a defined "fast enough."

Marin now has one evaluation launcher. #7570 put Harbor and evalchemy behind the same group launcher: serve a model once, run either mechanism against the served OpenAI-compatible endpoint, and write one EvalRunRecord per eval plus one columnar sample file, so an agentic trial and a multiple-choice question land in the same schema and both reach the evaldash dashboard. Harbor runs as an isolated uv subprocess because its Daytona pins drag in pre-release transitive dependencies the marin lock cannot resolve. #7575 then absorbed the OpenThoughts-Agent evaluation stack — 43 model definitions plus the Terminal-Bench 2, SWE-bench, GAIA, BFCL, Aider, MedAgentBench, and FinanceAgent presets — onto a shared ModelConfig / ServeConfig protocol, retiring a parallel YAML serve schema, vLLM lifecycle, and Hugging Face upload path; the common runner opens one inference context and hands the same RunningModel to every executor. #7578 gave the dashboard the matching view: each sample shows its grader method, metric, extraction filter, score, and the verifier's verbatim detail, and agentic samples lazy-load a trajectory of turns, tool calls, and observations through a path-scoped, size-capped artifact endpoint. A single Qwen3-0.6B batch served once on a v5litepod-4 then ran both an MMLU smoke and a two-task Terminal-Bench 2 run through Daytona, which is the evidence Romain Yon cited when closing this epic.

Getting there meant collapsing four parallel eval branches rather than merging them. #7476 and #7552, the backend-agnostic and CoreWeave GPU evalchemy launchers, were folded into #7570 — their model-config intelligence survives as the logic that reads a checkpoint's own config.json to pick a triton prefill backend for linear-attention models, a zero multimodal limit for vision wrappers, a Qwen reasoning parser for thinking models, and a max_model_len clamped to the native context. Benjamin Feuer's standalone agentic-evals package #7246 became the input to #7575, and his Daytona and Harbor configuration branches #7517, #7594 and #7595 were retired in favour of the shared path. Will Held's #7450, which built Dockerfile-defined Harbor task environments on the worker's own docker daemon, produced the most useful argument of the week before being closed: an in-trial build occupies a concurrency slot for its whole duration — up to about eight minutes on apt-heavy environments — so tail builds head-of-line block long runs, and a fresh worker rebuilds because the content-keyed tag caches per worker. Russell Power pushed toward Cloud Build with Artifact Registry, Benjamin Feuer pushed back that SWE-bench-scale agentic data generation implies tens of thousands of unique images, and the agreed shape — resolve every task's environment hash up front, build the missing images off-slot in parallel, and open the trial queue only against ready images — is now #7462. The same thread surfaced the coordination question directly, Benjamin Feuer asking where eval code is going to be centralized when it keeps appearing across PRs; Russell Power answered with a six-point contract (drive everything from the marin repo, separate serving from evaluating, keep Daytona pluggable, standardize output formats, share model definitions across Harbor and evalchemy, and make "run an eval on model X" reliably fan out to the right hardware) and #7345 closed against it.

Running that launcher against real models was a grind of unglamorous, specific failures. The Qwen3-32B Terminal-Bench 2 acceptance in #6865 first found the catalog routing the model to H100x2 rather than TPU, then the pinned CUDA vLLM fork rejecting a catalog-generated --swap-space 32, which #7624 cleaned out of the serve YAMLs; a retry was SIGKILLed after loading 47 of 707 weight shards on an inherited 64 GiB host-memory request, and #7640 sized it at 128 GiB, after which all 707 shards loaded in 38.4 seconds at 30.59 GiB per GPU. Daytona sandboxes were getting HTTP 403 from the minted capability URL because the job had been launched directly on a child cluster whose origin is not externally reachable #7607; the fix routes the mint and proxy through the top-level Iris cluster #7627, with #7634 making Marin consume the controller-minted relay URL. Harbor then failed uploading its artifacts because its GCS helper parsed a CoreWeave s3:// destination as a bucket named s3:, fixed in #7625, which also decided that a trial carrying exception_info is a failed evaluation while a verifier-scored zero is a successful one. The final run kept 2-3 requests in flight at 198 generated tokens per second and persisted 30 files across 41 S3 paths, and its record reads failed: nothing solved out of two, one task returning AgentTimeoutError after 900 seconds. Probing why, thinking-enabled generations exhausted 32- and 256-token budgets with finish_reason=length and no answer at all, while disabling thinking returned 391 in four tokens — which is why #7640 also put the max-token budget in the Marin Serve toolbar. Benjamin Feuer's Grug OpenCode evaluation came back on the new launcher in #7606, streaming a 124.9 GiB export onto eight data-parallel ranks in 140-155 seconds and solving one of three capped attempts.

Alongside the plumbing, a run of work went into whether the numbers can be trusted at all. David Hall flagged in the evals channel that evalchemy's GPQA-Diamond should not be believed: #7527 shows the task constructs random.Random(42) inside its per-question shuffle, so all 198 gold labels land at option D, and the 45.286% Grug score is exactly the rate at which the model picked D. Its answer extractor takes the last standalone A-D letter anywhere in the completion rather than the boxed choice the prompt asks for, so 78 of 521 boxed outputs are misread; the upstream shuffle fix has been sitting in an evalchemy PR for about four months. Russell Power's #7449 found two more of these while validating a new baseline suite on the cluster: --num_fewshot was only forwarded when truthy, so a requested 0-shot GSM8K silently ran at the task YAML's 5-shot default, and an oversized generation budget let a weak model generate to the cap on every unsolved problem until the client timeout turned into a retry storm. The same PR added a version and description to every run so evaldash's headline matrix shows one model's latest version cohort instead of a union across all time. On the harness question, #7626 got RULER's 13-task long-context group running by routing metadata-carrying tasks through lm-eval's own CLI, because the evalchemy fork's entrypoint builds its task manager without metadata and drops the served tokenizer RULER needs to synthesize haystack prompts; Russell Power asked for one source of truth, Will Held and Benjamin Feuer both agreed it should be the fork Marin owns, and the PR is being closed in favour of fixing eval.eval upstream under #7638. Rohith Kuditipudi opened grading speed as a CPU-only entry point for new contributors in #7463, and the first answer landed: #7657 ports the MATH graders to match lm-eval verbatim while skipping sympy's LaTeX parser wherever the answer shape already settles the question — 74 of 100 benchmark problems need no parse at all — for a 3.7x cold and 10.1x warm speedup with 100/100 agreement and no disagreement across 34,624 differential pairs. What remains for the automated eval system in #6503 is now narrow and named: a checkpoint-eligibility trigger #7628, a maintained playbook for registering a new benchmark #7629, regional caching so a two-task smoke stops pulling 861 task files from Hugging Face #7596, and an appendable columnar archive to replace the current spread of records, JSONL, parquet, and per-trajectory objects #7584. The eval set itself was settled in an evals thread — OT-TBLite, Terminal-Bench 2 and SWE-bench for agentic work, MATH500, HumanEval+, MBPP+, MMLU, GSM8K, AIME24, GPQA-Diamond and IFEval for the rest — after Percy Liang asked for a canonical number to keep in his head for each new pretrained checkpoint.

The dedup investigation found its root cause, and it was not the one everyone assumed. Fuzzy dedup had been removing a median of 76% of documents per source, taking massive_function_calling from 859,092 documents to 2 and starcoder2/ir_python from 154,406 to 2,613, and the leading hypothesis in #6854 was that a single global canonical per cluster stranded whole sources. Rafal Wojdyla rejected his own agent's per-source-canonical PR #7197 on exactly that ground — preserving one row from a false cluster does not fix the cluster. The real defect is that dupekit interpreted ngram_size=5 as character 5-grams, so shared English and code substrings make unrelated long documents look similar: across eight sampled Project Gutenberg members, pairwise character-5 Jaccard similarity reached 0.36 while word-5 Jaccard was at most 0.00043, and 51,160 unrelated books had been swept into one cross-source cluster. Compounding it, every shared locality-sensitive hashing (LSH) band counted as a duplicate edge and connected components took the full transitive closure, so sparse false links chained documents that never shared a band. #7591 moves document MinHash to word 5-grams and emits duplicate markers only for a cluster's canonical and the members directly adjacent to it in an LSH bucket, retaining transitive-only members; it also makes the shingle unit explicit in the native and artifact APIs so stale character-gram caches cannot be silently reused. It closes the templated-synthetic over-merge of #6851 alongside #6854. Rather than assert the fix, the agent is A/B-ing it at scale on the immutable 100B Datakit testbed — 115 sources, 768 shards, 103,716,988 documents per arm, matched at 286 permutations, 26 bands, a 500,000-character cap, seed 42, and identical non-preemptible resources. The infrastructure result is already settled: MinHash CPU fell from 84,007.55s to 24,982.35s, a 70.26% reduction, at similar peak memory, and connected components converged in 24 rounds rather than 53. The precision comparison is the slow part. Of 1,534,372 emitted markers and canonical references, machine checks classified 358 exact duplicates and 407,207 low-overlap baseline false positives outright, leaving 755,281 pairs to semantic review running on four 2×H100 batch-priority workers. By the week's last checkpoint, 185,235 pairs (24.52%) were independently verified across 1,458 immutable checkpoints, with all 210 model-unresolved cases read in full text and bound to source and judgment hashes — interim false-positive rates of 63.55% for the baseline against 51.72% for the treatment, flagged in every update as order-biased and not the arm-level verdict. The audit is honest about the tradeoff it is measuring: the treatment missed a genuine truncated-prefix pair and still merges multilingual function-calling examples whose tool catalogs dominate their text, so fewer drops alone is not the success criterion.

Decontamination got its complementary precision fix. The per-source common-n-gram filter added earlier removes boilerplate concentrated inside a single source, but text shared diffusely across sources — the Gettysburg address, an MMLU instruction template, the arithmetic-series formula — still cleared the 0.5 paragraph-overlap threshold and flagged 40 of 40 hard negatives, as documented in #7126. #7635 adds a global pass that aggregates eval n-gram document frequency across source samples and drops a hash only when it clears both a corpus-wide frequency floor and a minimum distinct-source count, so a high-frequency match confined to one source stays eligible for marking; in a four-source repro the repeated public quote falls from 1.0 overlap to 0.0 while an injected single-source eval item holds at 1.0. Weighting by eval-record frequency was tried and rejected at a 45% recall cost. The recall side, #6852, remains a posed question rather than a bug: short-line and embedded contamination escape by design, and the team has yet to say whether ~0% flagging is the intended operating point. On the store, #6725 merged the shuffle-based clustered-store build that closes #6687, replacing roughly 14.2M tiny leaf caches with about 84K scatter files and a store born compact, at an estimated 17 TB of in-region scatter and about two hours at 2k workers; #7434 now proposes dropping its dependence on a prior store's token mass in favor of a self-contained counting pre-pass, since no constant subshard default serves both the 651B-token hot bucket and the small ones. #6895 closed a quieter hazard: the reference pipeline's minhash step had no hash attributes at all, so changing permutations, bands, n-gram size, character cap, or seed served a stale cache, and external inputs now enter the hash as version tags instead of region-specific paths. #7461 wired a canonical end-to-end driver — datakit processing, a tiny pretrain over the resulting per-bucket store, then evals — where a 10-source sample builds a 36-bucket store and a nano model trains from 11.76 to 7.93 loss, confirming the flat-cache path actually feeds training. The quality-classifier cluster from the store audit — domain-not-quality sorting #6849, the truncated 4KB lead #6859, and near-constant per-source scores #6860 — is now closed, along with the clustered-store determinism question #6798 and the store visualizer #6856. Still open ahead of the mix: how to bucketize the calibrated quality score #7124, the million-shard fragmentation of #6858, and the lexical embedding that collapses all code into one domain cluster #6850 and all non-English into another #6855. Rafal Wojdyla sketched the plan that consumes all of it: new data buckets from the new sources and classifiers, sampled as widely as capacity allows on top of the existing 840 mixture runs, aiming at a new mix around mid-August.

On the corpus itself, HuggingFace released Stack v3 mid-week; David Hall posted it in data-curation within hours, Rafal Wojdyla filed #7558, and Will Held picked it up the same afternoon, noting that Marin would want to drop the Common Pile Stack v2 data and likely the Dolma 3.5 code derived from it. #7564 landed that swap, serializing each upstream repository as one training document with plain-text repository and file headers in deterministic directory order, preserving repo_path, commit, license, and file-level provenance, and measuring 4,568,429,666,429 tokens over 172,898,790 repositories; repository-scale records needed 64 GiB workers after 32 GiB ones died in external-sort pass two on CoreWeave. #7119 registered three components of AI2's Dolma 3.5 pool — dolma4pdfs at an exact 1,804,002,448,556 tokens over 137,132,279 documents, plus the two code subsets — splitting PDF normalization from code so the separately-ingested FinePDFs component is not double-counted. Staging SEC-EDGAR on CoreWeave took three fixes: #7499 routes shard writes through a virtual-host S3 filesystem, since CoreWeave rejects the path-style multipart upload that worked on GCS, #7520 reads shards with DuckDB because PyArrow's Thrift decoder cannot deserialize the page headers that multi-MB filings produce, and #7522 moves that DuckDB dependency out of core and into the datakit extra. #7567 added a source filter to trigger_sources so one failed source can be re-driven or moved to another cluster without rewalking every chain.

The next new capability is PDFs, the one common data type Marin currently drops. #7616 lays out a four-step pipeline over the science Focus Crawl, whose 25 MB content limit (raised from 5 MB) makes it an unusually good PDF source: #7617 pins the input manifest and the output document shape, #7618 builds the extraction step that routes born-digital versus scanned pages through Docling and AI2's olmOCR-2 optical character recognition (OCR) model behind a swappable interface, #7619 scores and filters with the released FinePDFs quality classifiers through Marin's quality tagger, and #7620 runs the existing dedup and decontamination tooling over the result. #7621 closes the loop with a token-matched side-by-side against a released FinePDFs sample, the same shape as the earlier comparison against Common Crawl's own WET plain-text extraction. The framing is deliberately modest — most of the pieces exist, and the plan is to prove quality at training-run scale before extracting the whole crawl. The jusText extraction that preceded it was marked complete and archived in #7397. Timing helped: an engineer from the Common Crawl Foundation who maintains Grobid, the open-source tool for structuring scholarly PDFs, introduced themselves in Discord the same week, offering to help harvest pretraining data from scientific publications and patents. Elsewhere, Ahmed Ahmed asked whether Nemotron-CC and DCLM had been decontaminated against Paloma, and Will Held confirmed all production data is.

The mixture-surrogate form-selection campaign settled this week, and it settled on a result that changes what the epoch penalty means in production. Rafal Wojdyla posted the final write-up on #7067, the issue arguing that bucket-indexed sweep evidence is stranded by any change to the bucket layer. The chosen model is a bucket-independent content surrogate — a Hellinger-kernel ridge over per-phase embedding histograms — plus a linear-past-threshold epoch-harm term, so a new source enters as a new column and gets priced without re-sweeping. The kernel held Spearman 0.720 on a pre-registered 40-run quarantined test set, about 79% of the seed-noise ceiling, beating weights-ridge, gradient-boosted trees, random features, kernel mean embedding, and neural baselines in the bake-off. The load-bearing finding is that repetition harm collapses with budget: on a matched-mechanism run at 100B tokens the four-to-sixteen-epoch rise measured −0.003 bits per byte (bpb), 0.4σ from zero, against a budget-transfer curve of {0.688, 0.384, 0.092, −0.003} bpb at {2.5, 10, 40, 100}B and a fitted exponent of −1.15. The severe harm seen at 10B was a small-budget artifact; at production scale the kernel alone suffices and the harm term is a guardrail for extreme (16–32 epoch) repetition. The committed swoosh-shaped harm curve was wrong in magnitude, shape, and onset, and the threshold turns out to be per-bucket (τ ≈ 8.85 for code, ≈ 11.66 for math).

The follow-on argument is about how far to take the de-bucketing. Percy Liang proposed dropping buckets entirely: represent each document as a vector, a domain as the average of its document vectors, and learn loss directly from the weighted average — at which point querying a chosen point means solving a linear program to construct a document distribution whose mean matches it. Rafal Wojdyla was happy with any form that gets rid of the opaque buckets, and plans to build new buckets with new data and classifiers, sample the space as far as capacity allows on top of the existing 840 samples, and aim a fresh mix at mid-August. Percy Liang pushed on how much extrapolation is really expected and asked for the plain argmax-over-the-swarm baseline; a leave-cluster-out check came back at ρ 0.787 for zmacro and 0.918 for HumanEval versus 0.817 / 0.938 under random folds, and a head-to-head write-up found kernel ridge and the effective-exposure parametric surrogate close on shared held-out rows, with both badly underpredicting the deliberately out-of-support policies Calvin Xu supplied as an adversarial stress test. Calvin Xu also flagged the open problem that the predicted two-phase optimum still barely beats the single-phase one at equal swarm budget, despite evidence the true gap is larger — more degrees of freedom, harder fit. Meanwhile Will Held noted that none of the swarm evidence yet covers how sparsity changes the epoching penalty, with Larry Dial expecting the hero-run architecture to land near 4-of-128 experts. On the plumbing side, a CoreWeave swarm launcher #7541 retargets the datakit mixture runs from a v4-8 TPU to H100s, one run per candidate mixture over the real 168-bucket store, after fixing a store-relative path that levanter opened as a local file and sizing runs around a 32 GiB fully-replicated activation tile.

The largest concrete change to the pool was a code swap. David Hall flagged the Stack v3 release the day it appeared, and Will Held — who wanted to retire the Common Pile Stack v2 data and the Dolma 3.5 code derived from it — landed the replacement in #7564 two days later, tracked in #7558. Stack v3 enters as 4.57T tokens over 172.9M repositories, serialized one training document per repository with plain-text Repository: and File: headers, direct files emitted before a deterministic recursion into child directories, and explicit nullable PyArrow schemas so batch-dependent type inference stops producing incompatible Parquet. Repository-scale records needed 64 GiB normalization workers after 32 GiB ones were killed mid-external-sort on CoreWeave. The raw pool itself sat flat at 23.2T tokens pre-dedupe across 139 datasets, 27.7% synthetic; Percy Liang's weekly update put the post-dedupe expectation nearer 15T. Ahmed Ahmed asked whether Nemotron-CC and DCLM had been decontaminated against Paloma; Will Held confirmed all production data is.

Michael Ryan posted his weekly sync on the LLM extraction recipe #2351 with a structural change: splitting the single-prompt spec into a cheap filter call followed by a separate extraction call for surviving pages runs roughly 3× faster, since reasoning only has to stay on for the gate. That buys scale, and he is taking the sweep from 300 raw crawl archives (WARCs) to 3,000 over the weekend — the point where the trends previously seen at 10k WARCs start to appear. His own read is that the benchmark deltas at 300 WARCs do not match what shows up at larger scale, so he is weighting direct inspection of the extracted text (via a cluster-backed spec explorer dashboard) over the scaling plots for now. His second conclusion is that data mixing is the untouched lever: he wants to run RegMix over DCLM plus his own extractions with DCLM and OLMo evals split as val and dev, on the argument that DCLM incidentally mixed itself through aggressive filtering, and that filtering and mixing are better kept as separate stages. Elsewhere, Jeff Hammerbacher revived the metadata-conditioning evaluation #5197, and a batch of long-dormant sourcing proposals was closed as stale — the long-doc quality audit #4735, the raw technical-text and raw-HTML tranches #4961 and #5012, and the diff/patch, paraphrase, and ASR/OCR inventories #5100, #5101, and #5102 — leaving the queue pointed at the mix work actually in flight.

The bucket-indexed-evidence problem in #7067 — that a mixture sweep learns a map from a bucket's name to loss, so any re-bucketing or corpus refresh strands the evidence — got its model form settled this week. Working through his agent session, Rafal Wojdyla posted the close-out of the form-selection campaign: a bucket-independent content surrogate, a Hellinger-kernel ridge over per-phase embedding histograms, plus a separate term for the damage done by repeating data. A new source becomes a new column of the embedding matrix and can be priced without re-sweeping. The surrogate held at Spearman 0.720 on the 40-run quarantined test set, roughly 79% of the seed-noise ceiling, and beat the weights-ridge, gradient-boosted, random-feature, kernel-mean-embedding, and neural entrants in the bake-off. The repetition term's shape was decided by leave-one-epoch-out cross-validation, which came out unambiguous: a linear-past-threshold form gives 0.056 bits-per-byte cross-validated RMSE on code against 0.246 for a quadratic and 0.260 for softplus-squared, because the curved forms over-extrapolate the 32-epoch point by 10–30σ. The onset threshold turns out to be per-bucket rather than global — about 8.85 epochs for code against 11.66 for math, with a confidence interval on the latter that excludes the former — and per-epoch damage ranks math above code above web. The previously committed “swoosh” curve was wrong in magnitude by roughly 119×, and wrong in shape and onset besides.

The finding that changes the recipe is the production-scale gate. A matched-mechanism run at 100B tokens — real subset-repeat through max_train_batches, the same mechanism as the 10B harm experiments, so only the budget changed — moved loss by −0.003 bits per byte going from 4 to 16 epochs, 0.4σ from zero and 9.3σ below what the power law fitted at smaller budgets predicted. Across the four-point budget-transfer curve at 16 epochs, harm runs 0.688, 0.384, 0.092, −0.003 bits per byte at 2.5B, 10B, 40B, and 100B, collapsing with a fitted exponent of −1.15, steeper than the −0.73 mid-range fit. At production scale the content kernel alone suffices, and the repetition term is a guardrail for proxy-scale work and extreme repetition past roughly 16 to 32 epochs rather than a core production requirement; the severe harm measured at 10B was a small-budget artifact. The caveats are on the record too: a single seed per point, and a small residual that survives only at 32 epochs, about 3.4× weaker than the 10B fit. The test-set protocol was pre-registered before any look and the 40-run quarantine held throughout.

Posting the write-up in the data-mixing channel opened a design argument about what the surrogate should be. Percy Liang laid out the general form: represent each document as a vector and each domain as the average of its document vectors, at which point the bucket is superfluous and the object being learned is a map from a distribution over documents to a loss. Choosing the most informative point to query then means constructing a real document distribution whose weighted average hits the queried vector, a linear program min_w ‖Dw − v‖₁ with an L1 penalty for sparsity — the hard part being to make that efficient over billions of documents. He also flagged the inductive-bias question, that carefully constructed parametric forms should be preferable in the low-sample regime unless a non-parametric fit models its uncertainty well, and asked for the two approaches compared on the old buckets alone with function projections overlaid on Calvin Xu's 2-D contour plots, plus a plain argmax-over-the-swarm baseline. Rafal Wojdyla's position was that he is not married to any particular functional form, only to getting rid of the opaque buckets. Calvin Xu answered the baseline question — RegMix-style interpolation usually does improve on the best run in a swarm, and parametric modeling was adopted to extrapolate further and to carry inductive bias into the low-sample regime — and described where his own work is stuck: the predicted two-phase optimum is not yet much better than the single-phase one despite evidence that a larger gap exists, which he reads as the extra degrees of freedom making the fit harder at a fixed swarm budget rather than a weakness of the two-phase policy class.

Both requests were answered within days. A leave-cluster-out check, holding out whole clusters and predicting the runs that used them, cost little rank correlation: 0.817 to 0.787 on the aggregate z-score target and 0.938 to 0.918 on HumanEval. The head-to-head write-up against Calvin Xu's effective-exposure surrogate found the two close on RMSE and rank correlation over shared held-out rows, with kernel ridge better on mean absolute error and Regret@1, and both badly underpredicting the adversarially out-of-support policies Calvin Xu had picked by total-variation distance from his training rows. The practical plan for August is deliberately unambitious about extrapolation: sample the space as widely as capacity allows and interpolate inside the sampled regime, adding new buckets built from new data and new classifiers on top of the existing 840 runs, with a refreshed mix targeted around mid-August. Will Held is firing swarm runs on the natural source buckets in the meantime, and Larry Dial pinned down the sparsity that the repetition penalty is sensitive to — likely between 4-of-128 and 4-of-256 experts, since sparser models memorize faster and tolerate fewer epochs. Separately, the dormant backlog of held-out perplexity slice proposals under #5005 was swept closed as stale rather than built — synthetic reasoning #5052, LM-eval dev splits #5053, raw web and markup #5056, binary and security artifacts #5057, bio and chemistry notation #5058, time-series and geospatial #5059, formal methods and hardware #5060, diagnostic log streams #5093, diffs and patches #5095, and paraphrase and translation robustness #5096 — as was the Luxical embedding exploration #3049, whose recorded conclusion is that it produced an experiment scaffold rather than a verdict, and that provenance stays the fallback for topic mixing until an embedding signal clearly beats it.

The week's work moved the agentic (tool-calling) SFT data path into the Marin repo, under #7098's goal of running post-training experiments from Marin rather than from a scattering of forks. Benjamin Feuer split it in two. #7454 covers the conversation-preprocessing half: the SINGLE_COLUMN_MULTI_TURN adapter now passes through assistant tool_calls and the tool_call_id/name on tool turns, which the plain role/content mapping had been dropping along with the whole <tool_call>/<tool_response> structure, plus a reusable tools_column_to_chat_template_kwargs that relocates a per-row tools column into chat_template_kwargs because Levanter has no tools column wiring of its own. #7455 merged the other half: a {% generation %}-annotated tools-aware Qwen3 chat-template resource plus a ChatLmDatasetFormat builder, with the generation markers covering assistant reasoning, content, tool calls, and <|im_end|> while leaving the assistant header prompt-side, so mask_user_turns=True works without changing the rendered token ids. Russell Power asked for the docstrings to be compacted and the string-asserting tests dropped before it landed.

The correctness claim behind that is unusually concrete. A harness ran each framework's own tokenization code over 60 real rows of the laion/nemotron-code-oracle-opencode-sft-serveparity set on Qwen/Qwen3-30B-A3B-Thinking-2507, and Levanter's input_ids came out byte-exact against Axolotl on all 60, covering tool-schema JSON serialization, tool-call argument serialization, role: tool framing, <think>, and BOS/EOS with no double BOS. The loss mask is not bit-identical, but the residual is roughly 0.38% turn-boundary jitter from Axolotl's runtime find_turn diffing rather than anything in the rendered text, and Levanter's static generation-marker mask is the cleaner assistant-only signal. Both the harness and its artifacts were published as re-runnable gists. #7454 and the Vocab-axis padding fix #7234 were both closed at the branch owner's request during a PR consolidation pass, with the branches retained, so that content is queued to come back in a larger batch.

The trace corpus those templates consume kept growing. Benjamin Feuer posted a cohort update to the agentic-trace index #6191: twelve new Qwen3.5-122B-A10B-FP8 at 131k datasets generated through the opencode installed-agent harness with the verifier on, about 64,772 rows added since the 2026-07-09 index, bringing that cohort to 34 datasets on Hugging Face. The sources are a mix of repository-task sets, Nemotron gym task families, and self-instruct sandboxes; three more sweeps are still generating and will be consolidated into -traces repos as they finish.

One decision closed out that had been holding the SFT framework story open. #7427 reports a controlled paired-init A/B on exactly that serve-parity opencode data, toggling only DenseMixer's dense-forward router gradient, and finds it functionally inert for this model and regime: top-8 routing overlap against the shared initialization was 0.859 with it versus 0.860 without, output-distribution Jensen-Shannon divergence averaged 0.0035, none of the 178 near-frozen experts were revived, and the in-distribution agentic eval was nominally slightly worse rather than better. The only signature was a low-rank re-steer of the router that never reached a different top-k selection. That tempers the case for a Levanter port under #7088, which the issue recommends keeping flag-gated and validating first on a regime with real routing demand. Benjamin Feuer read the practical consequence in #midtraining: DenseMixer had been the main reason to keep Axolotl in the picture, so with it inert here, Marin and OpenThoughts-Next SFT and midtraining can unify on Levanter. Separately, #7622 filed an export-portability gap in the same neighborhood as last week's chat-template export fix: Marin's Transformers 5 save path writes a tokenizer_class of TokenizersBackend, which standard Transformers 4 consumers cannot resolve, and it needed a one-file repair on the Grug 67B-A2B thinking export before evaluation could load it.

The week's biggest development for this epic arrived in Discord rather than the repo: SkyRL's core maintainers introduced themselves and offered to help rebase MarinSkyRL onto SkyRL HEAD, shrinking Marin's ongoing fork surface to something minimal. Their argument was that recent upstream has been run at 400B scale — GLM-4.7 and Qwen3.5-397B — at roughly 150k context, and that a good share of those performance improvements should carry over; they asked for a validation target (one small experiment for fast iteration, one larger one for final sign-off), a path to Marin cluster access, which agent harnesses Marin actually runs, and whether vision-language model (VLM) post-training is in scope. Benjamin Feuer answered the harness question — VLMs are not on this year's roadmap, and the move to the opencode harness for RL is mostly done. Russell Power set the validation target against a Marin-shaped MoE: 25B active out of 500B–700B total as the eventual shape, or Snowball's 2B-active / 64B-total in the shorter term. He was candid that cluster capacity for upstream's own iteration is not assured in the short or medium term, and framed the fork as a comfortable position rather than a problem — Marin is not unhappy on it, but would welcome having Benjamin Feuer's divergences upstreamed and rebasing on the result. A scoping doc from the SkyRL side is the next step; nothing has been committed yet.

On the code side, Romain Yon merged a Levanter oracle for Grug training parity #7420, giving MarinSkyRL's PyTorch implementation something exact to gate against. Marin emits a tiny Hugging Face checkpoint plus a deterministic NumPy .npz of observations from Levanter's JAX Grug model — routing, hidden states, logits, loss, four selected gradients, and one query-bias update — while MarinSkyRL consumes only ordinary Hugging Face and NumPy artifacts, keeping the JAX and Levanter environment on the Marin side of the boundary. Generation is pinned to one local JAX device so the fixture stays byte-identical on hosts that expose several; regenerating on current main after #7458 renamed canonical config fields changed only config.json, leaving model.safetensors and observations.npz untouched. The limitation is stated plainly in the PR: refreshing the downstream fixture is manual, so a future Levanter behavior change will not fail MarinSkyRL until someone regenerates and reviews it.

Russell Power's standing goal of driving every post-training experiment from the Marin repo #7098 collected two reports from Benjamin Feuer. #7454 and #7455 put the opencode agentic (tool-calling) SFT preprocessing in the Marin repo — a conversation adapter that passes assistant tool_calls and tool-turn ids through instead of flattening them to role and content, plus a {% generation %}-annotated tools-aware Qwen3 chat template wired into Levanter's chat format — checked by a byte-exact Levanter/Axolotl tokenization-parity harness that matched input_ids on all 60 sampled rows for Qwen3-30B-A3B-Thinking-2507; the loss mask is not bit-identical, which he traces to Axolotl's runtime find_turn boundary handling rather than to Levanter's static assistant span. Draft #7510 stakes out the Iris diagnostic-metadata and configuration boundary, deliberately leaving the watcher, collector, analyzer, and mirror ports, plus a 14-script migration plan, for follow-up. Separately, in his write-up of the 1e23 Delphi midtraining run, Benjamin Feuer noted the RL portion ran entirely on the Marin stack — Levanter, Iris, and Marin's own TPU and GPU clusters — at 100–121 s/step and 512 rollouts per step on 32 H100s, which is the clearest evidence so far that the current framework holds together end to end on Marin infrastructure.

Finally, Benjamin Feuer's #7435 — Harbor's per-trial trace layout writes roughly 70–120 small files per trial, so a 30k-trial job produces on the order of two to three million filesystem entries, a pathological case for both parallel filesystems with inode quotas and object stores with per-object latency — closed in favor of the shared appendable columnar eval archive #7584. The closing note carries the Harbor-specific acceptance criteria forward as requirements of the shared archive rather than a separate project: separate the compact runtime signal RL actually needs, rewards and per-trial status, from the full trajectory record; support concurrent trial finalization; preserve single-trial debugging; and drop the per-file upload, list, and delete behavior.

The Delphi RL scaling-law thread #6279 closed with a top-of-ladder addendum that is the most informative ablation this epic has to work from. Benjamin Feuer carried the ladder's method up to the 25B delphi-1e23 checkpoint and ran the same controlled group relative policy optimization (GRPO) probe — 100 steps, lr 3e-6, entropy and KL off, 4k context, 512 rollouts per step, one cell per RL dataset on 32 H100 — this time end to end on the Marin stack, with the SFT stage on Levanter and the Executor and RL launched onto CoreWeave GPU through the MarinSkyRL Iris launcher instead of Leonardo SLURM. Against an SFT baseline of 0.536 on MATH-500, RLVR-MATH finished at 0.544 and DAPO-Math-17k at 0.518, with every gsm8k delta inside roughly one standard error; the only cell that moved the selector was the deliberate train-on-test control at 0.592, logged as a contamination upper bound rather than a result. The sharpest reading is negative: RL on RLVR-IFeval pulled MATH-500 down to 0.278 while barely touching gsm8k, a clean cross-domain forgetting signal from optimizing against instruction-following constraints. Training was stable throughout — entropy sat around 0.11–0.14 on every cell, in contrast to the 1e22 rung where five of six cells entropy-exploded under the identical probe, so the stabilized recipe holds at 25B. Two caveats travel with the numbers and were stated up front: the midtrain seeding the chain stopped at 38% of its token budget under preemption and can be resumed when compute allows, so this rung does not support scaling-law claims on its own, and a gsm8k harness at 3072 generation tokens turned out to be model-blind against the 4k context, scoring three cells identically to 16 decimal places until it was re-run at 2048.

On the curation side, the concrete ask went public. #7418 reconciles the 95 SFT task-generation sources ablated in the OpenThoughts-Agent work against TaskTrove's verifier coverage and finds 43 that still cannot be pulled in for verified RL and datagen. The framing matters: this is a verifier gap, not an availability gap — nearly all 43 already have a public task set in Harbor format under the canonical DCAgent/a1_* naming, and what is missing is the pass/fail gate, whether because no verifier ships, the shipped one only checks that an answer file exists, or the real check needs an interactive environment. The triage splits into 28 sources where a deterministic gate is available at no per-rollout cost (compiler and test-runner exit codes for the Rust, C#, pytest, RSpec and bash slices; I/O exact-match for CodeElo and Codeforces), 13 that need an LLM judge, and 7 where gold exists but is awkward to operationalize. Five sandboxes were built and registered ahead of the tracker — three StackExchange families and glaive-code-assistant on an adapted nemotron_gym judge, plus a no-API-cost deterministic math verifier for tulu3-sft-personas-math. Benjamin Feuer then opened the list to the community in #data-curation, pointing contributors at the ground-truth tests that were simply dropped from the original Harbor releases.

What this RL data has to move got sharper this week too. The non-agentic evals for the 67B-A2B 2T cooldown, shared in #general, put MATH500, HumanEval+ and MBPP+ on a positive trend that dedicated post-training will have to finish, and name IFEval, IFBench, GPQA-Diamond and AIME24 as outright gaps — the same instruction-following axis where the 25B probe showed a math trade-off, which makes mixture design rather than single-domain RL the open question. The older IFBench preference work #5244 was given a managed summary that consolidates where it actually stands: on Tulu-3 8B, all three DPO arms beat the base on IFBench strict pass-all, the loose arm by 9.33 points, with IFEval flat to slightly up, while the paper-style full-train strict dataset and its LR sweep have not yet had results written back. And Percy Liang asked for the missing yardstick — a canonical way to score an incoming pretrained checkpoint, along the lines of MATH500 after minimal SFT plus a fixed budget of RL steps — which is the number this epic's ablations would otherwise keep re-deriving per run.

The largest thread outside the milestone epics was Echo, a shared context store for agents. Rafal Wojdyla landed the context database on the shared Cloud SQL instance along with a sync job that mirrors the marinmirror corpus into it every ten minutes — roughly 73k issues, pull requests, comments and Discord messages, embedded for semantic search with a citable URL on every row #7530 — and a follow-up adds a work-log skill and command-line tool so agents can check whether someone is already on a task and record results, decisions, and handoffs at milestone boundaries #7531. Russell Power opened database and Identity-Aware Proxy access to the wider Open Athena group and consolidated the deployment docs #7662. The other agent-tooling story was less happy: the automated prose-cleanup workflow introduced this week #7457 was passed instructions but not the description it was meant to rewrite, so it published its own "no source description was provided" error over real issue bodies, reported by Benjamin Feuer #7506 and Will Moss #7503. The body is now handed to the model as delimited untrusted text #7508, and Russell Power disabled the workflow pending review #7509. Separately, Romain Yon found that every Claude-driven workflow rendered quota exhaustion as a build failure, indistinguishable from a real one #7486; exhaustion is now classified as a skipped agent run #7496, and the invocation and Slack-notification steps moved onto shared marin-style actions #7515.

Three standalone fixes are worth noting. Romain Yon tracked down a Levanter inference test that had never passed: it sized a mesh to the host device count but pinned a fixed two-element key/value cache axis to it, so it failed on any host with more than two accelerators, and because the job is path-gated it turned a required check red on every Levanter pull request while main stayed green #7467; sizing the stub to one key/value head per device keeps the packed axis divisible at any device count #7468. Benjamin Feuer rebound the vocabulary axis to the padded tokenizer under pad_tokenizer_to_match_model, which had been sized from the unpadded tokenizer and so mismatched the loaded embedding for models such as Qwen3 whose vocabulary is padded past the tokenizer #7234. Eric Czech generalized coral's TPU batch calibration to Levanter's full parallelism model, deriving data-parallel and tensor-parallel degrees, per-device parallelism, and gradient accumulation from the topology, global batch size, context parallelism, and a high-bandwidth-memory estimate, with a policy that maximizes data parallelism and uses tensor parallelism only to occupy leftover chips #7380. The remainder was routine: dependency and security bumps, docs and build chores, Will Moss fixing two flaky-test races in shared test fixtures #7443, and the nightly cleanup sweeps, which alongside the usual deduplication corrected Mistral's FLOPs-per-token accounting — it ignored grouped-query attention and the gated MLP, skewing throughput and MFU estimates #7428 — and a Kubernetes quantity parse that truncated a node advertising 0.5 allocatable cores to zero millicores #7550.

Outside the core team, Matt Wittmann again carried the most, much of it on the toolchain the Blackwell work sits on: GPU JAX to 0.11.0 #7436, a CUTLASS DSL CUDA wheel no longer shadowing the pinned build #7491, and a gradient-accumulation bug that drops the FP8 amax history to the last microbatch #7659. He also identified the OpenXLA commit that fixes every existing reproducer of the intermittent GB200 CUBIN load failures in #7421. Will Moss drove the Pulumi adoption covered above and cleared two flaky-test races #7443. A first-time contributor, ayush1298, took up Rohith Kuditipudi’s call in #evals for CPU-only work with #7657 and a SciCode difficulty-proxy proposal #7658; znichols opened a durable JSONL GPU telemetry writer #7641.

On Discord, Kaiyue Wen traced the identity hyper-connections variant behind #7409 to a Chinese technical blog post and, asked which labs use it, named Qwen from earlier correspondence. Mayank offered a hybrid Newton–Schulz variant he thinks improves on the five-step MuonH in use today and asked how to run it; Will Held pointed at the agent.md procedure the team uses to assess architecture proposals. An announcement in #documentation open-sourced Open Athena’s mumwelt, agent skills plus a CLI for navigating Marin’s context: full-text and semantic search over code, including active branches, and over issues, comments, pull requests, Discord, W&B and these weekly summaries, against an hourly-refreshed corpus available by application.

Four people introduced themselves: Luca, at the Common Crawl Foundation and a GROBID maintainer, who wants to harvest pre-training data from scientific publications and patents; Alexandra Arriagada, an ML engineer at Cenia working on LatamGPT; Jessica Chudnovsky, a Stanford coterm student researching repeat data in pretraining after training multimodal models at Mistral; and Anni, a Stanford MSCS/MBA student working on multimodal classification and on evaluating agent behavior. Chudnovsky’s repeat-data work intersects with this week’s epoch-penalty result #7067. Three arrivals posted nothing: Pedro Ortiz Suarez, founder of the OSCAR corpus project and now a principal research scientist at the Common Crawl Foundation, whose Common Crawl annotation pipelines intersect with the focused-crawl and web-curation threads; davidheineman, who works on Ai2’s OLMES evaluation framework and DataDecide and brings context on the eval-standardization work; and Craig McLuckie, a Kubernetes co-creator, arriving the week all four CoreWeave Kubernetes clusters came under Pulumi.

What the community shared ran to corpora rather than papers — the Stack v3 release, the distilled ESM Atlas protein data feeding MarinFold, the Delphi write-up — with the week’s one arXiv link asking how pretraining choices shape the returns to RL compute.