Marin: Week of July 27th summary

The week's decisive result on the 2T cut is a clean negative. #7743 closes the loop on the Grug 67B agentic SFT run — five epochs of a serve-shaped agentic mixture on top of the stage-2 checkpoint, 1,903 steps on 64 H100s with AdamH, peak learning rate 5e-6, sequence length 32,768, final training loss 0.199, over one equal-weight mixture of 29 cache-backed components totalling 77,012 converted rows and 797,783,562 tokens. The first evaluation cohort was thrown out for a real export defect: config.json listed only EOS token 128001, generation_config.json was missing, and tokenizer_config.json declared the wrong tokenizer class. Benjamin Feuer republished a repaired checkpoint under a fresh repository identity with both Grug EOS IDs, a PreTrainedTokenizerFast declaration, and an embedded chat template whose SHA-256 matches chat_template.jinja exactly, then relaunched all three suites — in-distribution dev_set_v2, SWE-bench Verified, and Terminal-Bench 2 — against a new local asset identity with valid Daytona credentials. The repair worked and changed nothing. Of 600 terminal attempts, 506 scored, mean reward 0.005767, four attempts with any reward and two with full reward. Every one of the 506 ended abnormally: 495 through AgentTimeoutError and 11 through AgentKilledBySignalError.

The trace analysis names the failure precisely, and it is not knowledge. Median tool-call counts were 806.5 on the in-distribution set, 2,717.5 on SWE-bench and 1,495.5 on Terminal-Bench, with exact-duplicate call fractions of 33.9%, 20.3% and 36.9% — statistically indistinguishable from the discarded cohort's 994, 2,528 and 1,538. The individual traces are the clearest evidence: basic-message-queue produces a legitimate full verifier pass in one write and 13 calls, then stays alive until the deadline signal; broken-python also passes its verifier and then continues to 1,540 calls; legal-summary-extraction makes 8,473 calls including 1,262 identical shell reads of the same CSV; tsl-test-case-generation reads one input file 962 times before timing out at 4,400 calls. This is the same root cause David Hall identified in last week's sample-level analysis of the chat evals — the model does not have a real stop condition — now reproduced in an agentic scaffold where it costs everything. The issue's recommendation is not more evaluation of this checkpoint but a small diagnostic comparison against the repaired stage-2 base and a known-capable control before any further Grug agentic SFT. The full 587-row repaired trace set, the exact SFT data and token caches, and the checkpoint are all public on Hugging Face.

The grader-correction thread from last week reached its natural end rather than continuing: #7527, the GPQA-Diamond all-D gold-label and boxed-answer misparse report, closed on 23 July, and the canonical post-SFT eval sheet #7505 saw no new numbers this week. What eval work there was went into making expensive agentic cohorts survivable instead of re-scoring old ones — #7789 reports that Harbor resume cannot selectively retry unscored trials, so a run with 297 scored results and three that died to upstream 504s has to choose between rerunning everything and abandoning the three; #7835 reports that completed trials sitting in the eval pod's working directory are discarded if the driver stops before the post-run sweep, which cost one multi-hour selective resume five finished trials; and #7848 seeds the restored job config so Harbor's resume scan recognises trials it already has. Separately, #7830 pins all 29 Hugging Face sources and the teacher tokenizer behind this run's agentic SFT data into a reproducible Datakit recipe, and in doing so caught a source-provenance error — one component had been pinned to a later rewrite rather than the commit actually trained on, and pinning the historical source reproduced all 451 archived records byte-for-byte.

The forward move is #7786, filed by Benjamin Feuer on 30 July: non-agentic reinforcement learning with verifiable rewards (RLVR) on Snowball, joining the SFT checkpoint proven end-to-end in #7321 to the datasets established as known-good on Delphi in #6279. It is written as a set of hard constraints rather than a wish: the Grug path in MarinSkyRL is correctness-first, policy-only and expert parallelism (EP) 1, and rejects non-FSDP2 (Fully Sharded Data Parallel 2) strategies, router replay, sample packing, FlashAttention, context parallelism, grouped or fused MoE, LoRA and 4-bit loading before GPU allocation, so a misconfigured launch fails fast rather than producing a subtly wrong run; serving requires the Marin vLLM fork at a named commit, passed by immutable digest until the cluster's standard image is verified to carry it; and the query bias is FP32 state that participates in correctness, applied only after a successful optimizer step and preserved across skipped non-finite steps, so any resume path has to be shown to round-trip it. Node geometry is deliberately left unset — Delphi's two policy nodes and eight tensor-parallel inference engines is a 2B-active geometry, and Snowball is 67B total at roughly 2B active with 256 experts under EP1, so the memory profile has to be derived from a short run rather than copied. Two of the definition-of-done items are unusually strict: a detailed assessment of RL challenges specific to Snowball is named a primary deliverable, and performance metrics must be reported in a form that survives the confound that sank #7784, where cross-arm step time correlated with rollout length at Pearson 0.809 and produced a "5× faster" claim that had to be retracted. The plan also flags that use_kl_loss: false and use_entropy_loss: false, inherited from the Delphi math config, should be revisited rather than copied, since an unanchored objective was that campaign's dominant failure mode.

Carrying the Delphi recipe forward immediately surfaced a reproducibility problem worth recording. Benjamin Feuer went back to #6279 and documented the dataset provenance that had only ever existed in an archived experiment directory: rlvr7500 is allenai/RLVR-MATH at 7,500 rows and dapo17k is BytedTsinghua-SIA/DAPO-Math-17k re-wrapped from its native Answer: prompt to boxed. More usefully, he verified that the prepped parquets sitting under iris/rl-data/ are the 500-prompt probe subsets, not the full-scale sets — the parquets behind the headline MATH500 53.4 and 51.6 numbers are not there and need locating or regenerating before anyone tries to reproduce them. #7794 proposes maintained profiles for the common public math and instruction-following RL sources, recording revision, split, prompt field, ground-truth field and reward contract, so this run can cite a pinned source rather than hand-convert. On the systems side, Romain Yon had the Grug FSDP2 training path in MarinSkyRL up for review at the start of the week, having ported the vLLM fork commits MarinSkyRL depends on into Marin's own fork, with the public safetensors export for Hugging Face still waiting on legal. His draft #7878 supplies the number that decides whether a full RL run is affordable at all: on a frozen 4,096-sequence replay across four hosts and 32 H100s, Levanter's fastest valid update takes 255.859 seconds against MarinSkyRL's 6,991.670, a 27.3× gap, and a matched cross-entropy run instrumenting all 26 routed expert modules attributes 5,583.786 seconds — 81.466% — of its critical-rank wall to routed experts. The report is careful about what that does and does not prove, rejecting the old cross-stack causal label because Levanter starts from FP32 masters while MarinSkyRL starts from stored BF16 parameters and their full-replay losses differ by 0.95%, and naming the cheapest discriminator: the owner's parity-passed grouped experts against its eager expert loop on the same replay with identical represented values. A related fix, #7894, carries the next valid query's causal lower bound backward through leading padding in packed segments, since FA4 was pruning key tiles that later valid queries in the same tile still needed.

No bytes moved to the Hub this week. The release plan #7495 drew exactly one comment — Romain Yon asking for a read on 27 July — and the reason it is parked shows up in the week's standup #7669, where Romain Yon lists cleaning up the Grug/Snowball safetensors export for public consumption and upload to Hugging Face as blocked on legal. Everything upstream of that block is already checked off: the single-name config contract #7447 / #7458 merged and the bf16 tree re-cut, the snowball-v1 serving-fork tag created, and the model card's commands tested end to end on a driver-only 8×H100 node. What is still open is the human part — the repo id, whether to ship or strip the untrained chat template, the apache-2.0 approval, and a paragraph describing the full 2.3T-token data lineage and decontamination status. Work continues; the artifact itself has been verification-ready since the export was re-pinned.

The serving side moved without waiting. Romain Yon opened #7758 as a draft, running GrugMoE through native vLLM pipeline parallelism (PP) on every Iris task, and used Snowball as the correctness oracle: all 64 representative goldens scored on one node at PP1 and again on two gang-scheduled 8×H100 tasks at PP2, with every DP rank recomputing twice and matching itself exactly. Worst main deviations were a greedy gap of 0.0549 and probability error 0.0555 at PP1, and 0.0129 / 0.0838 at PP2, inside the 0.125 vLLM-only bound. The PP2 run split layers [0,13) and [13,26) across the two tasks — the exact shape #7706 recommends for the ~360B GrugMoE candidate, which Romain Yon had floated in inference two days earlier. Letting native vLLM own pipeline communication and stage-local loading instead of a custom Fray named-port path cut the Marin diff from 21 files and 1,609 additions to 15 and 1,073, and 36% across Marin and the fork together. That the same 64-case set now passes across a node boundary matters directly to #7495, since rerunning it is one of the five pass/fail items in the release verification gate.

The checkpoint's downstream reach kept growing. Benjamin Feuer filed #7786 to run non-agentic RLVR on Snowball, joining the two halves that had never been connected: the SFT checkpoint proven end to end in #7321 and the RL datasets established as known-good in #6279, where rlvr7500 reached MATH500 53.4 (+8.4) and dapo17k 51.6 (+6.6). Its definition of done asks for a full run rather than a smoke test, a written assessment of RL challenges specific to this architecture, and throughput numbers matched on output-token count — the confound that produced a retracted “5× faster” claim in #7784, where step time tracked rollout length at pearson 0.809. Snowball is also becoming the reference point for eval design: #7776 proposes perturbing GSM8K with HELM invariance transforms and irrelevant chat history precisely because the post-SFT checkpoint's 77.1 is the strongest clean baseline in the canonical table.

Two findings this week are about the seams around the checkpoint rather than the weights. #7743 re-evaluated the stage-3 agentic SFT checkpoint after repairing a genuine export-metadata defect — both Grug EOS IDs restored, a real PreTrainedTokenizerFast declared, the chat template hash-matched — and it still scored 0.00577 mean reward over 506 attempts, every one ending by timeout or signal. The repair was correct and the conclusion changed anyway: this is agent-control behavior, traces continuing past useful work and duplicating tool calls, not a packaging bug. Separately, the init_from silent no-op — documented as the hook a chained mid-training/SFT/RL phase uses to point at a prior phase's checkpoints, never read by either Grug train loop, and fixed this week in #7791 after #7742 — is the mechanism every downstream continuation of this checkpoint depends on: until it landed, everything chained off Snowball was quietly starting from random initialization.

The multinode GPU proof-of-concept cleared its throughput bar. The JaxPP pipeline-parallel tracker #7024 took 89 entries in a single 27-hour push and ended with the exact 24-layer, d2560, 64-expert top-4, sequence-4096, batch-8192/m256 target running at 20.4067 mean MFU across four H100×8 nodes — 460,234 tokens/s, against a strict target of sustained mean MFU above 20 and an ordinary-Ring baseline of 18.2583 that had been saturated for two weeks. The unlock was replacing ring expert parallelism (EP) with NCCL 2.30.7's UB-X symmetric-memory transport, promoted through four gates in sequence: a direct eight-way EP identity-transport gate 3.17–3.45× faster than ring with bitwise-exact dispatch, a raw one-process JAX FFI at the target 16,384 tokens/rank and hidden 2,560, a full MoE value-and-vector-Jacobian-product gate whose worst output/loss/gradient relative-L2 was 0.0113% against an FP32 transport reference, and then a reduced JaxPP execution gate. The last blocker was structural rather than numerical: the six-layer-per-stage graph kept aborting with XLA seeing only 1 of 8 expected local-device threads at thunk initialization. Compiling and caching every local task up front, then crossing a coordinated barrier before execution, made the rank-skewed lazy initialization go away, and the first exact run under it landed at 19.8996 mean MFU — 8.99% over ring, 0.50% short of target.

Closing that last half-percent produced the week's most instructive detour. A vectorized 128-bit dispatch copy replacing the scalar per-element kernel improved the exact-shape transport microbenchmark by 6.5%, but the exact reruns first died on a 19.51 GiB stage-3 backward allocation, then — once an allocator selector was traced to XLA_PYTHON_CLIENT_ALLOCATOR rather than the TF_GPU_ALLOCATOR knob that JAX 0.11's PJRT plugin ignores — came back at roughly half throughput, ~142s per step. Two hypotheses were tested and discarded: the RNO2A cluster was genuinely emitting InfiniBand port errors and retry-exceeded send completions (21 events in one job against zero in the successful baseline), but moving to cw-us-east-02a recovered only 14% and left the run 40% below the historical number; and a matched scalar-copy control on the clean cluster came in at 11.83 MFU versus the vector run's 11.85, making the copy kernel performance-neutral at full shape. The actual cause was a missing --vocab-size flag in both arms, which silently inherited the launcher's 128,256-token vocabulary instead of the intended 8,192 — the b8192 in the run name denoted batch, not vocab. Restoring it, with the vectorized copy retained, produced the 20.4067 result.

Everything else tried this week was closed as a negative, and the closures are worth recording because they narrow the design space. A fused FP32 data-local expert-gradient accumulator passed numerics and a reduced lifecycle gate and projected 20.014 MFU from direct measurement, but the exact L24 run falsified the projection at 15.95 mean MFU, 12.6% below baseline; profiling attributed the regression to twelve FP32 all-reduces inside every six-layer backward and a 3× growth in send/recv count. Moving the Ring pullback into an explicit custom vector-Jacobian product removed 96.31% of layer-normalized all-reduce time and gained 2.5% at L8, but the exact L24 graph crashed in six-layer backward compilation, and a narrow late outline that did compile needed an unequal 7,6,6,5 split, reached only 16.5155 MFU, and then exhausted rank-0 memory. QuACK 0.6.1's wrong-axis backward concat permutation was found and fixed — token and W13 gradient relative-L2 fell from 128.29% and 141.24% to 0.691% and 0.708% — but that is still above the project's 0.2% ceiling, so the Sonic grouped-MLP path stays closed. The NCCL_EP hybrid, FP8 bulk-ring wire transfer, and both EP4/data2 and EP2/data4 topology remappings were rejected on the same evidence standard.

On the loss-validation half of the epic, the multi-head latent attention (MLA) versus grouped-query attention (GQA) comparison flagged last week resolved in MLA's favor on loss but with a caveat that changes the reading. Larry Dial's MLA arm grug-d2048-L24-mla-500B-r4-b512-datakit completed all 238,418 steps at 21.2% MFU and 2.4657 Paloma macro loss, ahead of the GQA reference grug-d2048-L24-gqa4-500B-r8-nosim-v2 at 2.4787 and 23.7% MFU. The two are not FLOP-matched — MLA runs 32 attention heads against GQA's 16 — so this is MLA buying 0.013 nats with extra attention compute, not beating GQA at equal cost. The cheapest useful answer came from the sibling with the latent-dimension scaling correction enabled, grug-d2048-L24-mla-500B-datakit-qkvscale, which stopped 712 steps from the end at 2.4661: a dead heat with the uncorrected arm, so the correction does not matter at this token budget. The two no-sliding-window arms stopped at 216,297 and 221,655 of 238,418 steps, mid-cooldown, so their readings do not compare cleanly against the finished runs.

The 67.1B-total / 2.01B-active MoE on TPU v4-2048 — tracked here and logged on #6044 — went from 4.98T to 6.35T tokens this week, step 81,740 to 102,192, or 63% of the 10.07T-token horizon. MFU held at 18.6% and train cross-entropy fell from about 1.37 to 1.33. Paloma macro loss dropped from 2.3318 to 2.3000 across the week's eval points (macro bits-per-byte 0.8430 to 0.8319). The stage-1 target was preregistered before the run started — 2.269 Paloma macro loss at 8T tokens, from a three-point Chinchilla fit over d=512/768/1024 anchors with the irreducible loss pinned at 1.4 — and the run is now inside 0.031 of it with 1.65T tokens left to reach the 8T mark. The last 1.37T tokens bought 0.032, so the trajectory is close to exactly on the predicted curve, and the learning rate has still not decayed.

Uptime was the best of the run so far. The reserved v4-2048 slice logged 147,456 chip-hours at 99.7% utilization over the seven days, against multi-hour and in one case 38-hour restart gaps earlier in July; the failure-budget fix that landed last week in #7451 appears to be holding. The one visible interruption was not a training outage at all: Larry Dial reported on Monday that W&B logging had stopped while the iris job kept running and checkpointing normally, and killed and resumed the job to bring telemetry back. The W&B history shows the shape of that: nothing recorded between step 81,740 and step 84,416, roughly 180B tokens trained and unlogged, and the periodic Paloma eval at step 84,000 is simply missing from the series.

The run crossed 6T tokens on 31 July at step 96,919, and Larry Dial said in Discord on Saturday that the second intermediate cooldown — the successor to the 2T cooldown from #6811 that became Snowball — branches at 6T and should finish the next day. Earlier branch points were skipped rather than lost: cooldowns at intermediate token counts can be re-cut later from retained checkpoints as compute allows. He also flagged the follow-on to come back to these checkpoints and take sequence length from 65,536 to 262,144, the long-context extension the architecture was deliberately simplified for at launch — per-key offset and rotary embeddings on the full-attention layers were both disabled so the extension could follow textbook practice.

The Hugging Face release of Snowball, the 2T-token cooldown checkpoint, did not move. Romain Yon asked Isaac Hodes to review the release plan #7495 on Monday, and it still carries its four open decisions (repo id, ship-or-strip the untrained chat template, public-upload-then-verify, and license plus a training-data paragraph covering the full 2.3T-token lineage); his standup note records the export cleanup as blocked on legal, which is decision four. Meanwhile the checkpoint is doing the job #6811 was filed for — exercising the post-training stack ahead of the full 10T model. #7786 was opened to run non-agentic reinforcement learning with verifiable rewards (RLVR) on Snowball from the supervised fine-tuning checkpoint proven end to end in #7321, using the datasets established as known-good in #6279, with a per-architecture assessment of what behaves differently on a 256-expert model as a primary deliverable rather than a footnote.

Nothing landed against the budget itself. #7074's own thread was silent, and so were the two threads that carried last week's action — the on-policy distillation (OPD) teacher-serving RFC #7236 and the quantization-aware-training proposal #7404 — neither of which took a comment this week. Benjamin Feuer's pricing therefore still stands unrevised: roughly one to ten billion post-training tokens, each costing about 700× the hardware FLOPs of a pretraining token. The one genuinely new input came from an adjacent experiment that measured, rather than estimated, what a distilled token costs.

Russell Power's Qwen3 32B-to-0.6B study #7656 finished and reported in #7671. On a four-GB200 cell, exact full-vocabulary forward Kullback–Leibler (KL) distillation from a frozen 32B teacher ran at about 27,900 tokens/s against 95,000–97,000 for hard-label training — 78.6 GPU-hours versus 23.9 for the same 1.8 billion tokens, so a distilled token bills roughly 3.3× a plain supervised one. Cheapening the teacher does not recover it: the Qwen3 4B capacity-gap arm ran 2.1× faster and still failed to beat the scratch forward-KL control, and adding per-layer activation matching cost another 21% for no reliable paired gain. That is the frozen-teacher, off-policy counterpart to the legs #7236 prices, and it makes the teacher-serving question concrete: the multiplier is real and it is not small.

The study is also a clean piece of prespecification. Its promotion rule was fixed before any results were read — at least 0.5% mean terminal negative log-likelihood (NLL) below the scratch forward-KL control, both paired seeds directionally better, and no zero-shot regression outside frozen tolerances — and rank-512 factorized teacher initialization was the only one of five treatments to clear it, running 1.47% better at 450 million tokens. The 1.8-billion-token extension then overturned it: factorization ended at 2.6644 NLL, 0.39% worse than the 2.6541 control, with both paired seeds worse. Starting from the official Qwen3 0.6B Base checkpoint and training with forward KL was the strongest arm at 2.5579 NLL, 3.63% below scratch KL, and led every zero-shot task. For the budget the reading is that teacher-derived initialization does not buy tokens at any horizon worth pricing, while a good public base checkpoint does — and that a short screen cannot be trusted to rank methods for a long run.

Around the edges, the teacher-logprob plumbing settled. Jeff Hammerbacher put the canonical-scoring-path question to mumwelt, and in the thread Rohith Kuditipudi landed on vLLM's prompt_logprobs — noting that vllm-tpu cannot serve prompt logprobs with automatic prefix caching on, which does not matter for scoring fixed documents — and proposed deleting Marin's stale save_logprobs, which Romain Yon and Russell Power agreed to; Romain Yon's standup #7669 keeps TPU-versus-GPU logprob parity on the back burner. Two items bear on the upper bound's biggest unknowns. Jenia Jitsev questioned, in the Kimi K3 report thread, whether separate per-domain RL teachers are needed at all once models are large, rather than pooling domains into one RL run — if pooling holds, the K-teacher multiplier on this epic's estimate falls away. And Benjamin Feuer's agentic-RL campaign wrap-up halved the per-arm footprint from eight nodes and 64 GPUs to four and 32 while raising the context budget from 32,768 request tokens, 4,096 new tokens per turn and 30 turns to 131,072, 16,384 and 90 — a roughly order-of-magnitude rise in tokens per rollout, which is the term that dominates the RL side of the estimate and has not been folded back into it.

The shape question got its clearest answer yet, and it arrived from the schedule side rather than the throughput side. Will Held opened #7673, which runs the candidates from the GB200 Architecture TPS Tracker #7201 through three end-to-end schedules — post-training handing off November 15 or December 1, releasing December 24 or January 15 — and prices each one in pretraining tokens and RL response tokens. The recommendation is to plan around d6144 top-4-of-128 at ~20.0B active / ~355B total: roughly 20.8T pretraining tokens for a December 1 handoff, or 17.4T if the handoff moves to November 15, which trades 3.3T pretraining tokens for 13.6 more effective post-training days. That is not a large prior update against last week's shortlist, but it converts the shortlist into a default. The analysis is explicit about why the fastest option does not win: d5120 4-of-64 gets 27.4T tokens against d6144's 20.8T, and the loss fit still ranks it below, penalizing the smaller active model even with more tokens and more RL cycles. It is equally explicit about where it stops. d8192 4-of-128 has the lowest predicted reducible loss in all three schedules, and Will Held declines to train it — there are no measured throughput points above d6144, the loss fit has no prediction interval, and Marin has no RL scaling curve — so d7168 stays on the table only for a January 15 release and only if its measured pretraining and RL pace turn out close to the forecast. Every RL number in the table is labelled an unmeasured ceiling that reserves no time for SFT, evaluation, verifier work or release packaging, and the 20.0B active-parameter convention is itself still unresolved in #7201. A follow-up comment sets the plan against published models for calibration — MAI-Thinking-1 at 35B active on 30T pretraining plus 3.55T mid-training, OLMo 3 Think 32B on 5.5T, Arcee Trinity-Large-Thinking at 13B-of-398B on 17T, Kimi K2 at 32B-of-1.04T on 15.5T, and DeepSeek V4 Pro at 49B-of-1.6T on 33T.

On the measured side, Larry Dial filed the full d6144 candidate as a single stack rather than a shape: 48 layers, hidden 6144, 48 query heads with heterogeneous grouped-query attention (GQA) at 12 key/value heads on local layers and 6 on global, top-4 of 128 routed experts plus two always-on shared experts, sliding window 512 with global attention every six layers, no positional embeddings (NoPE) on the global layers and half rotary position embeddings (RoPE) on the local ones, a short causal convolution (sconv4), attention gate, XSA, quantile-balancing routing, and a DeepSeek-V3 depth-1 dense multi-token-prediction (MTP) head — 22.24% MFU and 246.0K tok/s on one rack, projecting to ~84 days for 20T tokens. The two real throughput costs are the convolution at −2.0 points and the MTP head at −1.8; partial-RoPE, the 1× MTP MLP and hetero-KV are all roughly MFU-neutral quality plays. The same stack scaled to 256 experts — ~716B total / ~21.5B active on two racks — lands at 18.10% MFU with MTP and 18.66% without, with expert-chunk 4 the knee (chunk2 16.41%, chunk8 16.36%) and MTP nearly free at scale, −0.56 points instead of −1.8, because the cross-rack expert all-to-all dominates the step and shrinks the head's fixed share. That all-to-all is also the ceiling: two racks top out near 18.7% against one rack's ~22%, and chunk tuning cannot fix it — expert parallelism (EP), keeping the exchange on NVLink, is the untested lever there. A long-context sweep at constant ~4.19M tokens/step found 8k context nearly free at −1.8% tok/s, while 65k costs −35% but does fit in HBM with FA4 and sliding-window attention; Larry Dial's read is to limit how much training happens at 65k and above, and that 262k will need a new sharding pattern since it will not fit on a single GPU. The sweep's most useful byproduct is a measurement correction: lm_flops_per_token is window-blind and counts full quadratic attention on all 48 layers when 40 are windowed to 512, overcounting by ×1.08 at 4k, ×1.17 at 8k and ×2.14 at 65k — so reported MFU rises from 22% to 36% across the sweep while true efficiency falls from 20.6% to 17.0%. Trust tok/s; treat MFU as inflated whenever sequence length is not 4k.

A sixteen-row variation sweep on the Fully Sharded Data Parallel (FSDP) d6144 baseline then mapped the space around that stack. The best row is no-conv, sliding-window 512, expert-chunk 4 at 25.5% p50 MFU and 284K tok/s, which Larry Dial calls the ~25.6% ceiling — dropping the convolution is worth about a point, though it is expected to be included anyway on quality grounds. His reading of the sweep is the substantive part: going from 48 to 64 query heads costs only 7% tok/s (262K→243K), barely moves parameter count once sharded over 64 GPUs, and is the same head ratio Inkling uses, so he expects quality to repay more than 7% and will test that separately. The expert side has no comparable room — widening routed experts to 4096, going to 256 experts at top-6 with routed 2048, and stacking 64 heads with batch 1152 all hit OOM — though 256 experts at top-8 with routed 1536 at the same total parameters is viable for a 7% throughput hit and may be worth it. On sparsity his position is that the current rate is not that high and he would rather go sparser than less sparse, while raising active experts without raising total experts likely buys little. Sliding-window effects came out noisy, sometimes with 2048 faster than 1024. One incidental finding worth fixing: the telemetry watch step runs at roughly 2% MFU and is dragging mean MFU down at watch-every-20. Meanwhile EP kept closing: an agent-run EP64 screen on a d5120 4-of-256 config reached 24.153% MFU at 462K tok/s with 2.765% assignment drop, by zero-padding the five non-expert Muon parameter stacks to 64 rows so the Newton–Schulz batch can shard across the expert mesh, and Larry Dial noted that EP may soon be the stronger option if it picks up the FSDP stack's tunings, none of which are MoE-specific.

What has still not happened is the preregistration. #6702 remains an empty issue untouched since late June, and the hero-run tracker #6689 still carries blanks for FLOP budget, token count, data mix and pre-registered loss — the shape is converging faster than the predicted loss that would let the run be scored honestly against a filed forecast. Work continues toward what was framed as an Aug 1 run; the schedule analysis in #7673 now plans around an August 15 pretraining start, and Percy Liang's community update put it as still pushing to land the final architecture with adequate MFUs and data mix. The token side firmed up: Will Held reported the raw pool at 25T pre-dedup, with Stack v3 replacing Dolma 3.5 and CommonPile's Stack v2, plus DocX extractions from Common Crawl that most pipelines skip, and observed that with the end-of-year budget at most 20T tokens there is now room for outright quality filtering rather than only upweighting and downweighting. Rafal Wojdyla noted the mixing work is gated the same way, and that once the new data mix and an architecture are settled, as many mixing experiments as fit will be launched. Launch plumbing moved in parallel: a rack-scale FSDP hero launcher landed in #7864 and #7865, validated at two data-parallel racks and global batch 2048 for 487,019 tok/s at 20.31% MFU with 32 of 32 tasks clean, and #7876 proposes disabling XLA GPU command buffers by default after three sequential 200-step two-rack runs completed 96 of 96 worker tasks with no failures or preemptions. Romain Yon worked the serving side in #7706, finding that the ~360B candidate's ~670 GiB of BF16 weights do not fit one 8×H100 node and leave a 4-GPU GB200 node with essentially no KV-cache headroom, and recommending two H100 nodes at pipeline-parallel 2 / expert-parallel 8 with each expert group kept node-local, or four GB200 nodes at expert-parallel 16 inside one NVLink domain. And the hyperparameters that the shape implies are now specified but not started: #7856 defines a 150-run learning-rate sweep — six widths from d512 to d2048, five token budgets from 30× to 600× active parameters, five multipliers around the MuonH heuristic — to check whether the recommended LR stays compute-optimal as size and tokens scale, with no runs launched yet.

The single-rack recipe became a self-contained, reviewable artifact this week. Larry Dial landed #7820, a minimal experiments/grug/moe_hero_fsdp/ variant — the fixed 64-GPU GB200 configuration inlined rather than routed through env-var knobs — plus the Levanter substrate it needs: the QuACK SM100 grouped-GEMM MoE backend, chunked large-vocabulary cross-entropy, and FlashAttention-4 CuTe segment bounds hoisted out of the layer scan. It reimplements the known-good subset of the larger shared-substrate PR #7779, which Matt Wittmann continued to work through review. The most consequential review finding was on the measurement itself: the variant's lm_flops_per_token charged full 4,096-token attention to all 48 layers when 40 are windowed to 512 and only 8 are global, making MFU 8.30% too high. The corrected window-aware count is 44.491B FLOPs/token instead of 48.186B, which turns a displayed 24.6% into 22.71%. Every hero number below is on the corrected denominator and is therefore not directly comparable to previously reported MFU. Three other findings landed with it: a non-finite loss now raises rather than breaking the loop and exiting success, the step returns a metrics-only result instead of declaring a checkpoint artifact it never writes, and the half-built HF export was flagged as still not reconstructable by any reader. Larry Dial ran the merged variant smooth for 400+ steps on one rack and recommended merging to unblock multi-rack debugging.

Larry Dial's handoff note on that PR is what made the multi-rack work tractable. It listed three items from #7779 that the minimal variant deliberately dropped, all exact no-ops on one rack and all load-bearing across racks: a replicated embedding table, so the token lookup is a replica-local gather instead of an all-to-all whose NCCL first-call rendezvous had been observed to wedge at eight racks; pinning the FA4 per-token metadata replicated, since otherwise XLA leaves the arange constant on {maximal device=0} and the downstream reshard becomes an involuntary full rematerialization serialized through device 0; and excluding the cross-rack replica_dcn axis from Muon's Newton–Schulz reshard. Rafal Wojdyla took that list into #7864, which makes rack count a first-class parameter: a required --dp-racks sets global batch to dp_racks × 1024 and allocates 16 GB200x4 nodes per rack, model state replicates across replica_dcn while Fully Sharded Data Parallel (FSDP) parameter shards and Muon Newton–Schulz stay on each rack's 64-GPU data axis, and the local embedding gather, replicated FA4 metadata and a single batch-sharded self-attention segment array remove the cross-rack device-0 paths. A companion change #7865 added --num-steps and a WANDB_PROJECT override, which is what turned the 25-step throughput probe into the 200-step validations that follow.

The scaling ladder that resulted, tracked as an explicit checklist on the 512-GPU wedge issue #7344, is the substantive answer to last week's open question. Two racks (128 GPUs, global batch 2,048) ran 200 steps clean at 466,703 tokens/s and 19.47% MFU, 32/32 tasks with zero failures or preemptions. Four racks (256 GPUs, global batch 4,096) ran 200 steps at 925,857 tokens/s, 19.309% MFU and 18.121 s/step, 64/64 tasks, final loss 3.009 — throughput scaled 1.98× from two racks for 0.24 points of MFU, which is a far gentler cross-rack penalty than the ~19% first-hop cost Larry Dial had warned about. Eight racks did not complete. The 512-GPU run reached step 80 and then failed in jit_train_step with cuMemAllocAsync failed to allocate 124108462544 bytes: 88 of 128 tasks across all eight racks, an allocator reporting 24.65 GiB in use against a 115.58 GiB request and a 138.22 GiB limit — over by about 2.02 GiB. The subsequent clique stalls and JAX shutdown-barrier failure were downstream of the memory errors, not an independent wedge. The successful four-rack run used the same code and reported 167.07 GiB after rematerialization at the step-80 compile against the eight-rack run's 167.82 GiB, and the logs do not identify the 115.58 GiB buffer, so the next isolation is an A/B of the watch-enabled executable against builds with profile-guided latency estimation (PGLE) or watch metrics disabled with the HLO buffer assignment captured.

Two stability changes made those numbers trustworthy. Rafal Wojdyla's #7876 disables XLA GPU command buffers by default for FSDP hero runs on B200 racks — not because they are unwanted, but because, as he put it, they are flaky on the cluster until someone makes them stable — and validated the default with three sequential two-rack 200-step runs: 96 of 96 worker tasks with zero failures, retries or preemptions, mean 468,678 tokens/s, 19.549% MFU and 17.898 s/step, and no CUDA graph destroy error of the kind that opened #5675. Separately, Matt Wittmann closed #7753 via #7755: Fray hardcoded leafgroup coscheduling for every multi-replica GPU gang, so a two-rack GB200 job was being bin-packed 14+18 across NVLink domains rather than landing as balanced 16+16 rack slices, silently and with normal-looking metrics. Fray now derives the level from the same Iris topology helper the CLI uses. Not everything is clean: the non-deterministic collective wedge in #7344 reproduced at two racks under expert parallelism with the NCCL termination watchdog armed and silent, and Iris marked the successful four-rack job's coordinator failed after a controller GC block even though all 64 tasks exited zero, now tracked as #7881.

Where that leaves the 20% bar: on one rack it is comfortably cleared, and the multi-rack medians now sit just below it — 19.5% at two racks and 19.3% at four — measured against the stricter window-aware FLOPs denominator this week's review installed. Work on the run this commitment was written for, originally slated for Aug 1 as an ambitious internal date, continues; the eight-rack out-of-memory is the concrete blocker, and the isolation plan for it is written down. Percy Liang's weekly update framed the state of things the same way: still pushing to land the final architecture in #7201 with adequate MFU and a settled data mix, alongside model sizing and timeline. The epic's own thread stayed quiet all week.

The 103.7M-document A/B on the immutable 100B testbed returned its verdict on Tuesday, and it was decisive but not sufficient. Across all 115 sources the word-shingle treatment avoided 407,207 complete, nontruncated low-overlap drops the baseline made; all 1,018 evaluator disagreements were read by hand and hash-verified (704 false positives, 314 true duplicates), putting reconciled semantic false-positive rates at 63.617% for the baseline against 52.149% for the treatment, an 11.468-point advantage that held on the gold-corrected bottom-hash sample (63.182% versus 51.222%, non-overlapping Wilson intervals) and survived standardization by source, length, truncation, and Jaccard. MinHash processed identical items and bytes at 84,007.55 versus 24,982.35 worker CPU-seconds, a 70.26% reduction, with connected components converging in 24 rounds instead of 53. The audit closed with "a run beyond 100B is not needed for this comparison" — and with the observation that a 52% false-positive rate is still a coin flip. That reframed the fix. #7591 grew an exact verifier on top of the word shingles: remove a candidate only when it is no longer than the retained canonical and every case-folded token 3-gram in it occurs there, with a character-5 Jaccard floor of 0.90 for under-tokenized text. On a frozen stratified sample of 5,410 resolved treatment pairs that rule found 689 true duplicates and a single semantic-label false positive — 99.855% precision at a 99.184% Wilson lower bound, but 27.527% recall — and scoring all 155,212 materialized candidates accepted only 23,362 of them (15.052%), including 0 of 1,251 massive_function_calling, 0 of 116 starcoder2/ir_python, and 0 of 2 Project Gutenberg candidates. The whole-source wipeouts of #6854 simply stop happening once removals are checked. On Friday #7591 was closed unmerged in favor of #7831, which keeps character-shingle MinHash and connected components as deliberately broad candidate discovery and puts all of the precision in a second Zephyr job that joins candidates to normalized text and writes sparse dup_doc removals only after direct comparison. When a candidate fails against the cluster canonical, saved locality-sensitive hashing (LSH) buckets nominate at most one retained local representative, requiring token-3-gram and character-13-gram Jaccard of 0.98, bounded at two comparisons per document, 64 representatives per cluster, and 2,000,000 characters of retained local text; repeated content IDs are handed to global exact dedup instead of carrying a fuzzy marker. It passes an independent 0.1B persisted-output replay and 329 local tests, and claims both #6854 and the templated-synthetic over-merge of #6851.

Verifying removals means the verifier needs the text, and pushing full documents through a shuffle makes a 104,490-document cluster the reducer's memory tail. #7866 answered that with read-only Zephyr memory stores — immutable Iris actors holding candidate text keyed by (file_idx, id), with ordered batched lookups and reconstruction from the same source shards after preemption — and was then split for reviewability into #7873 (the generic load_memory_store() primitive) and #7874 (the Datakit integration), dropping the worker-pool-sharing dependency in the process. The controlled 100B comparison emitted the identical 27,203 markers from 1,513,510 candidate members in both arms while cutting peak cluster-reducer resident memory from 16.25 GB to 2.56 GB (-84.2%) and comparable stage CPU from 6,131.44 to 4,268.61 seconds (-30.4%); loading and encoding 31.96 GB into the actors costs about 2,400 CPU-seconds, so a single consumer pays 9.0% more total CPU and only reuse amortizes it. A native-object follow-up took total stage CPU from 97,749.66 to 93,284.42 seconds (-4.6%) and left wall time at 22m03s, since the remaining tail is one shard spending ten minutes intersecting a giant token n-gram set. Two infrastructure faults surfaced along the way and merged: #7872 preserves ConnectRPC error codes so a stale actor endpoint re-resolves rather than retrying blindly, moves periodic CPU/memory sampling onto the shard's explicit worker context, and qualifies CoreWeave object-storage endpoints with the bucket so Polars stops issuing path-style requests that returned HTTP 400 before reducers read data; #7880 keeps actor listener sockets bound through Uvicorn startup after two host-networked actors on the 32-actor benchmark picked the same released ephemeral port. Around the verifier, the store plumbing was tidied: #7700 makes the shuffle implementation the canonical build_clustered_store, retiring the map-side per-input-shard caches at a production default of 2,048 reducers and 32 uniform subshards with no prior token hint required — the chicken-and-egg #7434 raised; #7816 adds a global exact-dedup pass emitting sparse dup_doc attributes for records MinHash skipped; #7827 flattens attributes into top-level Parquet columns; and #7829 stores result paths relative to MARIN_PREFIX so artifacts survive a move between clusters.

Decontamination's precision fix merged Monday: #7635 aggregates eval n-gram document frequency across source samples and drops a hash only at document frequency 50 or more in at least three sources, so the Gettysburg address and shared instruction templates stop clearing the 0.5 paragraph-overlap threshold while a high-frequency match confined to one source stays eligible for marking; it closes #7126, with #7678 removing a stale README paragraph after review. The recall side is now measurable against it. #7679 adds an opt-in min_abs_hits path that flags a paragraph holding enough distinct matching 13-grams even when they are a minority of it; at 8 hits, embedded recall on the labeled benchmark goes from 0.37 to 0.91 (overall 0.83 to 0.97) at precision 1.0, but a 100B before/after took the flag rate from 0.0041% to 0.0095% on real web data, and the new flags were dominated by enacting clauses, bible verses, multiple-choice instruction stems, and standard LaTeX identities. It ships disabled by default and rebased onto the global drop-set, so the open question in #6852 — where to put the operating point — is now a measurement someone can run rather than a guess. The domain axis got the week's other long investigation. Chasing the lexical embedding that collapses all code into one cluster #6850 and all non-English into another #6855, a corrected Arctic-to-Luxical distillation ladder over 146 sources and 74,752 held-out documents showed the student tracking its teacher closely (0.9259 pairwise fidelity) while still failing the collapse gates — the ceiling is the teacher, not the distillation. Direct audits then put Arctic Medium at 0.66915 overall macro-F1 against Luxical-One's 0.61727, with code improving most (0.79995 versus 0.68089) and multilingual regressing; an alternative-teacher sweep found Qwen3-Embedding-0.6B better still at 0.67664 with 46 rather than 60 failing sources and no multilingual regression, making it the next teacher candidate. Two follow-ups came back negative and were reported as such: compressing Qwen to 256 dimensions loses the multilingual gate, and a 750K-row student trained through a cross-dimension alignment head inherited all 46 teacher failures and added 94 of its own. A same-source geometry regularizer was also stopped after every weight traded probe quality for fewer failures. A parallel GLM-5.2 pilot that labels documents into a semantic taxonomy avoided assignment collapse (38 buckets, 24.53 effective) but exposed overlapping label scope, so it is not training a student yet either.

The corpus side was the week's volume. #7788 found that a Stack v3 row lists each file many times over — commonly ten identical entries with the same content ID — and that whole-document MinHash cannot see repetition inside a document; keying on (file_path, content_id) removed 1.205 trillion tokens, taking the measured total from 4,568,429,666,429 to 3,363,007,313,642 over the same 172,898,790 repositories, which means mixtures computed against last week's count over-weighted Stack v3 by about 35%. Repository-scale documents also needed 64 GB workers and a batch size of 512 to tokenize at all. GHALogs went from unusable to a real source in three steps: #7772 fixes #7770 by reading each archive member as a gzipped tar and emitting job logs rather than decoding gzip bytes as UTF-8, splitting logs above 8 MB into line-aligned chunks (the tail is 0.6% of job logs and 60% of the bytes, so a cap would have discarded most of the source) and bounding the quadratic rescan that pseudonymization does per identity; #7823 validates each nested archive up front after 58 published members with corrupt gzip streams exhausted their shards' retries; and #7828 replaces the legacy 150B estimate with a measured 253,343,866,746 tokens over 2,613,421 documents. Seven more sources registered with exact tokenizer counts rather than estimates: the AgentTrove terminal-agent transcripts #7766, filtered from 1,696,847 rows to 781,076 by dropping proprietary teachers and exact duplicates and tagging only the ~20k rollouts that carry a real verdict, separately from the 177k that merely timed out; 142 pinned Benjamin Feuer rollout repositories #7824, alongside an org-wide scan that turned up 11,082 agentic-trace datasets not in any Marin index; English DocxCorpus #7733; Tim O'Donnell's biocorpus #7815, pinned by a fingerprint over file paths, sizes, and hashes so a bucket mutation fails before download; a balanced functional-DNA source #7819 whose region tags #7841 then rewrote as natural language ([Region: promoter]) so the words reach the model, after Gonzalo Benegas asked whether a model would generalize to unannotated genomes; the synthetic identity dataset generated on ten GB200 jobs and recorded in #7817; and reconstructed Nemotron-Code-v1 contents #7811, rebuilt byte-for-byte by resolving 513.1M (repo, commit, path) triples through the Software Heritage graph. #7822 brought English Common Corpus and StarCoder2 low-resource IR back, and #7852 proposes 71 independently mixable Nemotron v3 post-training sources #7826 at 169,721,332,606 tokens. Will Held published a token-count viewer putting the raw pre-dedup pool at 25T, the largest change being Stack v3 displacing Dolma 3.5 and Common Pile's Stack v2 — and noted that with an end-of-year budget of at most 20T tokens, there is finally room to filter on quality outright rather than only reweight. Two forward-looking threads opened: #7813 proposes instruction backtranslation over the 146,661 Stack v3 repositories (9,089 with at least ten stars) that use scientific software, detected through imports, dependency manifests, and signature filenames rather than name matching, with GLM-5.2 writing the brief a scientist would have handed a model — Will Held circulated a sample and immediately flagged it as "probably a bit too well specced"; and on the PDF pipeline #7616, Mark Muchane argued for optical character recognition (OCR) overlapped with downloading, and Will Held countered that on a healthy shared cluster staged execution is usually efficient enough that the GPUs are busy with other workstreams anyway — a point Mark Muchane accepted, since the full pipeline will not run often.

Expert parallelism (EP) got a clean, one-mechanism-at-a-time rebuild. #7279's MoE-hero ladder ran eight isolated 25-step gates on one GB200 NVL72 rack at EP64 — d5120, 256 routed experts, top-8, batch 1024, quantile-balanced routing with no auxiliary balancing loss — changing exactly one thing per gate. The ragged dispatch baseline started at 14.9614% median MFU and 260,426 tokens/s; a fixed-capacity all-to-all at capacity factor 1.0 took it to 19.3689%, rebuilding the send buffer as an int32 source scatter plus an activation gather added 2.51 points for twelve net lines of backend code, and two structured custom vector-Jacobian products for the dispatch and combine gathers added 2.25 more, reaching 24.1231% MFU and 388,503 tokens/s. Three fidelity mechanisms were then measured against that point and all three declined: capacity factor 1.0625 cost about a full point to move drops from 9.68% to 7.25%, receiver-ECHO cost 5.91 points and 561 net lines to reach 4.41%, and same-step spill to the next-ranked selected expert cost 0.04 points and 106 lines to reach 5.88%. Under the ladder's declared highest-MFU, least-code rule the plain stack was selected and passed a 200-step gate at 23.6969% MFU, 382,902 tokens/s and 7.41% final drops, landing as #7890. The drop column is the thing to read alongside the MFU column: last week's counter established that dropped assignments gather a zero pad row and do less real work, so a run that drops more reads higher. Matt Wittmann's expert-parallel core and moe_hero_ep template #7780 measures the same architecture at 22.398% MFU and 346,950 tokens/s at 1.444% drops across three placement draws — 0.102pp under its preregistered 22.5% gate and 0.898pp above its falsification threshold — which is consistent with, rather than contradicting, the 23.7% figure at 7.41% drops. That PR is unusually candid about provenance: the quoted numbers came from research build c24ccfcc2 carrying a different MuonH, and all three draws ran a manual profile-guided latency estimation (PGLE) profile worth +0.427pp that the template deliberately does not ship, so as shipped it should land about that much lower. Its review also turned up that both hero launchers declared weight_decay=0.1 that no optimizer builder ever read — and so does launch_cw_scale.py on main, meaning every recorded Grug MoE measurement to date trained at zero weight decay #7783. The three older draft extraction markers #7488, #7490 and #7493 were closed as superseded by that stack.

The week's most useful outside number came from #7668, a Megatron calibration run on 64 GB200s, closed with a ceiling-first ladder and a sharp caveat. The promoted BF16 configuration — 4-of-128 experts, ~250B total / 15.95B active, HybridEP with 32 communication SMs, EP64, microbatch size 1, no recomputation — reached 8.3400 s p50, 502,916 tokens/s and 32.407% MFU on the same Levanter-compatible FLOP formula and 2.5 PFLOP/s GB200 denominator Marin uses, stationary across the measured window. The direct 8-of-256 shape reproduces as a 31.322% systems ceiling. But that envelope uses Megatron's force-balanced benchmark router. With the stock auxiliary-loss-free router at its documented 1e-3 bias rate, the same stack falls to 14.720% MFU over 100 measured steps, degrading from 24.235% in the first twenty to 14.034% in the last twenty as the mock-data loss falls and routing specializes. Post-dispatch telemetry names the cause directly: force-balanced traffic holds average EP-rank overload at 1.013 and worst-rank at 1.023, while the learned router starts at 1.338 average / 1.971 worst and is still at 1.212 / 1.724 by step 18; faster 4e-3 and 1e-2 controller settings failed the preregistered balance gate of 1.05 average and 1.20 worst-rank overload. The comparison that actually matters ran the PyTorch lane with real quantile balancing, capacity 1.0625 and three spill attempts over 350 steps: 17.041% MFU and 263,971 tokens/s, 3.667 points below the corrected 20.708% JAX point and 17.7% fewer tokens per second. Along the way the ladder rejected a long list for Marin's stack — GatedNorm at +12.4% step time, exact dropless quantile balancing at 22.782%, compiled capacity-plus-spill at 18.975%, the current MuonH/HyperBall path at 5.167%, generic NCCL all-to-all at 4.423%, and microbatch size 2 which OOMs in the first backward — while exact Grug head gating and Q/K normalization cost only 1.496% and were retained, and exclusive sparse attention at +4.3% was kept quality-conditional. Read together: PyTorch has a genuinely faster systems ceiling, and nobody has a trainable router that reaches it.

On the Fully Sharded Data Parallel (FSDP) line, Larry Dial filled in the architecture tracker #7201 across three sweeps. The full d6144 4-of-128 candidate with a dense multi-token-prediction (MTP) head, heterogeneous 12/6 key-value grouped-query attention and two MoE expert chunks measures 22.24% MFU / 246.0K tokens/s at one rack, ~84 days for 20T tokens at twelve racks; the two real throughput costs are the short convolution at −2.0 points and the MTP head at −1.8, while partial rotary position embeddings (RoPE), wider MTP and heterogeneous key-value heads are all roughly free. At two racks and 256 experts (~716B total) the picture changes: four MoE expert chunks is the knee (16.41% / 18.10% / 16.36% at chunk 2 / 4 / 8), MTP costs only 0.56 points because the cross-rack all-to-all dominates the step, and the ~18.7% two-rack ceiling sits well below the ~22% one-rack figure — the gap is precisely the expert all-to-all leaving NVLink, which is the case for the EP work above. A separate FSDP variation sweep found the current best at 25.5% p50 / 284K tokens/s with the convolution off and four chunks, and Larry Dial argued that going from 48 to 64 query heads costs only 7% of throughput while adding almost no parameters or activation memory, likely a favorable quality trade; he also flagged that the periodic watch step runs at roughly 2% MFU and is what drags mean MFU below p50. The long-context sweep produced the week's most quietly important methodological correction: at constant tokens per step, 8k context is nearly free (−1.8% tokens/s) and 65k costs 35%, but reported MFU is window-blind — lm_flops_per_token charges full O(seq²) attention on all 48 layers when 40 of them are windowed to 512, overcounting by ×1.08 at 4k, ×1.17 at 8k and ×2.14 at 65k, so reported MFU climbs 22 → 24 → 36% while true efficiency falls 20.6 → 20.3 → 17.0%. Trust tokens/s until the attention term is made window-aware. Two negative results closed off directions elsewhere in #7279. Latent MoE, which projects down before dispatch to halve the all-to-all payload, does exactly what it promised — expert all-to-all exposed time fell 54%, against a −50% prediction registered before the runs — and still lost, 24.842% for the dense control against 24.610% and 23.425% for the two latent arms, because exposed collective is only 9.8% of the span and the expert all-to-all is half of that, so the mechanism addresses ~2.7% of step time while the projections add 7.52% of analytic FLOPs that landed as +10.8% compute. The same profile surfaced something worth more than the direction it killed: three of the twelve all-to-all operations are scheduled on the compute stream at 0% overlap, 422 ms of every 15.3 s step and 31% of all exposed collective time, unexplained and needing no architecture change. Meanwhile the FP8 forward-dispatch wire #7665 was measured across a 16× range of expert-parallel degree at fixed per-device tokens, and the effect is monotone in the collective's share of the layer rather than in model size: 1.019× forward-plus-backward at EP4, 1.101× at EP16, 1.144× at EP64, with weight gradients bit-exact against the control. Two methodology traps were pinned in the process — an f32 rebuild of the weight-gradient operand materializing a 1.7 GB tensor the control never creates (casting through bf16 is exact and recovered the backward), and a multi-process timing harness where jax.block_until_ready waits only on process-local shards, producing a confident 2.774× that was pure drift. The in-step measurement with rematerialization on remains the one that decides it.

Reliability is now the binding constraint on multi-rack claims. Three sequential two-rack 200-step FSDP hero validations with XLA GPU command buffers disabled ran clean at 468,678 tokens/s and 19.549% MFU with zero failures across 96 worker tasks and no CUDA-graph destroy error, and four racks held at 925,857 tokens/s / 19.309% #5675; the eight-rack 512-GPU run then reached step 80 and failed with a 115.58 GiB cuMemAllocAsync request against a 138.22 GiB limit on 88 of 128 tasks across all eight racks — a shape problem, not a bad node, with the four- and eight-rack compiles reporting nearly identical post-rematerialization footprints. The older silent 512-GPU wedge #7344 narrowed on two fronts: Matt Wittmann reproduced it at two racks under expert-axis 64 where the original was eight racks at expert-axis 1, so neither the eight-way replica axis nor the FSDP all-gather is required, and the collective-timeout watchdog was confirmed armed at 600 s yet did not fire across an 879 s wedge, with no NCCL output at all in 33,830 log lines. Rafal Wojdyla posted a rack-by-rack checklist with one, two and four racks confirmed clean and eight outstanding, and #7796 was filed to persist NCCL reliability-diagnostic snapshots before any recovery action, since the entire investigation ran without one. Russell Power took the deterministic sequence-8192 stall #7694 apart by elimination: a pure EP=16 reproduction disproved the original FSDP diagnosis, and the backend matrix now reads THD compiling then sitting stationary with perfectly aligned collective counters, CuTe returning a finite forward loss of 11.7966 then non-finite gradients from segmented backward, cuDNN with no plan for the shape, and the reference backend sustaining two 16-GPU burns past 5,000 updates — same pinned CUDA 13.0 and NCCL 2.28.9 in the healthy and failing arms, which localizes it to fused-attention lowering at sequence 8192 rather than the fabric. Two attention correctness items also landed: #7630 merged, resolving FA4 THD window sizes before CuTe tracing so the DSL stops evaluating both arms of a plain if, and Romain Yon opened #7894 for a real packed-segment bug — FA4 prunes key tiles using the first query in an M tile, so the old sentinel for leading padding could discard keys a later valid query still needs. The GPU canary caught a stack break the same week: quack 0.5.0 evaluates a cute.core.ThrMma annotation at import against the pinned CUTLASS DSL 4.6.0, which no longer exposes it, so every GPU run touching the FA4 backward path on main was broken rather than just the canary #7777; pinning quack 0.6.1 fixed it and the H100 canary completed 49 steps #7803. Looking forward, the transport A/B #7670 closed with NCCL static all-to-all and ECHO-ragged throughput-equivalent at ~279K tokens/s — ECHO-ragged buys near-zero drop (0.02% against 1.32%) for about a point of MFU — while HybridEP trained correctly at top-8 for the first time but ran roughly 5× slower at ~85% exposed communication. And a new experiment #7891 takes the zero-drop question head-on, implementing MoonEP plus the Kimi K3 global-histogram quantile-balancing update in Levanter — Larry Dial had flagged the K3 report in #news days earlier — with a declared bar of zero assignment drops at EP64 while retaining at least 21.7% median MFU. Its correctness gate passed at four GPUs with zero drops, an audit caught the estimator operating on raw logits where the report specifies sigmoid scores, and the rack gate is currently blocked on an illegal-address fault in the ragged device kernel. Separately, #7681 notes that Levanter still has no in-framework rank-level straggler attribution despite two prior full implementations, and Will Held's schedule modeling #7673 — which Percy Liang pointed at in the weekly update — consumes these MFU numbers directly and asks for the next thing to measure: d7168 and d8192 throughput on the same accounting, to find where the curve falls off.

Last week ended with a diagnosis and a proposed contract change: a ~1 unit in the last place (ulp) rounding difference in the ring reduce-scatter's expert combine gets amplified by MoE routing into a 0.03–0.29 probability swing, so the GPU parity gate's cross-rank sentinel was asserting a quantity the stack does not hold stable, and the recommended landing was to keep the 64-case golden assertion, swap the cross-rank assertion for a same-rank determinism check, and log rank spread rather than assert it. This week Romain Yon shipped exactly that contract inside a much larger piece of work: #7758 runs GrugMoE pipeline parallelism (PP) through native vLLM across every Iris task, and its Snowball parity suite scores all 64 representative goldens, requires two fresh computations on every data-parallel (DP) rank to match exactly, and records cross-rank variation as a diagnostic instead of a failure. The vLLM-only representative probability bound is relaxed to 0.125 while Levanter keeps 0.075 — an explicit acknowledgement that the collective's destination-dependent rounding is real and not the serving path's fault. The measurements bear the diagnosis out: on both arms the 16 repeat observations formed eight exact same-rank pairs, greedy token 423 on every rank, 49 shared top tokens at one-node PP1 and 48 at two-node PP2, and maximum cross-rank probability spread 0.108477 and 0.046884 respectively. Determinism within a rank holds; agreement across ranks is now measured, not demanded.

The serving architecture underneath is the more consequential change. The leader Iris task starts vLLM's normal API, scheduler, and local DP and expert-parallel (EP) workers; each follower task starts a bare native MultiprocExecutor for its pipeline stage. Iris supplies only a Torch rendezvous and small ready/shutdown registry records, and vLLM owns process creation, rank mapping, message queues, GPU selection, and stage execution — which let the earlier draft's custom Fray named-port path and custom pipeline forwarding be deleted outright. The Marin diff fell from 21 files and 1,609 additions to 15 files and 1,073 additions, and across Marin and the vLLM fork together additions fell from 2,231 to 1,433, a 36% reduction. The launch validates the exact process, node, DP, EP, PP, tensor-parallel (TP), and GPU placement plus each worker's model layer range before it sends a single prompt, and a follower exit tears down the gang. Qualification ran twice at production priority in cw-us-east-02a: PP1 on one 8×H100 task verified eight placements over layers [0,26), with worst greedy gap 0.054937, probability error 0.055502, and top-probability L1 error 0.137775; PP2 on two gang-scheduled 8×H100 tasks verified 16 placements, task 0 owning even process ranks and layers [0,13) and task 1 owning odd ranks and layers [13,26), with worst greedy gap 0.012890, probability error 0.083776, and L1 error 0.175125. All 64 goldens passed on both. The runtime is pinned to a single vLLM commit from the fork's vllm#25. The PR is still draft and unreviewed.

The topology it implements comes from #7706, which Romain Yon posted for comment in #inference. Sizing the ~360B-total / ~21.5B-active candidate against a 359.6B split of 347.9B routed plus 11.7B non-routed parameters gives about 670 GiB of unique BF16 weights; because the current TP=1 implementation shards routed weights under EP but replicates the non-routed weights on every GPU, EP8 costs 102.8 GiB/GPU, or 822 GiB across one 8×H100 node against 637 GiB allocatable — one H100 node simply cannot hold the model. A four-GPU GB200 node at EP4 needs 183.8 GiB/GPU, 735 GiB against 740 GiB allocatable, leaving 1.2 GiB/GPU before KV cache and runtime, which the issue calls a capacity edge rather than a serving shape. The recommended starting points are two H100 nodes at PP2 TP1 DP8 EP8, keeping each eight-GPU expert group node-local so only one pipeline boundary crosses InfiniBand, and four GB200 nodes at PP1 TP1 DP16 EP16 inside one NVL72 domain. #7758 then built and qualified precisely the H100 shape. Making that qualification runnable interactively took its own change: #7787 gives dev_gpu.py --nodes N a gang-scheduled hold on N whole GPU nodes under one Iris job, renames --gpu-count to --gpus-per-node and rejects fractional pods before submit — a two-GPU pod would let both halves of a nominal two-node session land on the same machine with nothing downstream catching it — and binds GB200 gangs hard to a single nvlink.domain. Adjacent to serving but bounding it, #7695 measured the data plane that will refresh these workers during RL: one 719 GB BF16 policy moves GB200 rack-to-rack in 7.31 s under the current-like 1→8 fan-out, 3.97 s under an ideal pre-sharded 8→8, and 26.96 s at the 1→1 bulk floor, while all 12 direct GB200-to-H100 TCP attempts failed with EHOSTUNREACH or timeout and the rejecting network component remains unidentified.

The other numerical-parity thread this week was on the export side. Tim O'Donnell found in #7801 that transformers 5 serialises rope as a single rope_parameters block, while transformers 4.x reads rope_theta and rope_scaling — and, critically, does not error on the 5.x shape: it ignores the block and falls back to the architecture default. A checkpoint exported from a transformers-5 training environment therefore loads under 4.x with the wrong rope base and no scaling, silently. On a Qwen3 1.5B trained with Llama3 rope at theta 500000, transformers 4.57 loads theta 10000, and mean negative log-likelihood over three held-out documents goes from 2.414 to 3.179 nats/token, with the error growing from 0.25 nats at 361 tokens to 1.34 at 683 — short smoke tests never see it. Anything pinned to transformers 4.x is affected, including vLLM 0.9.2 eval workers. The fix emits both shapes, is additive and a no-op where both already exist, runs after the config_overrides merge, and is verified against 4.57 and 5.14; already-published checkpoints still carry only the 5.x shape and need a config rewrite or loader-side repair. It belongs to the same family as #7743, where a Grug 67B agentic SFT checkpoint's export was repaired to carry both EOS IDs, a generation config, and the right tokenizer class — and the repaired cohort still returned 0.00577 mean reward across 506 scored attempts with every attempt ending by timeout or signal, which moved the diagnosis from export metadata to agent control. Elsewhere, #7863 added a bounded Prometheus bridge that filters vLLM metric families, preserves replica labels through reset-aware deltas, derives KV peaks from raw samples, and reports source availability when a scrape fails, giving served endpoints post-mortem evidence they did not have. And in #code-talk, Jeff Hammerbacher answered Rohith Kuditipudi's question from last week about a canonical logprob-scoring codepath with a mumwelt-generated writeup of what exists today.

Of the three sub-issues, correct GPU inference #6869 is where all the motion was: it now has a multi-node gate whose numerics contract matches what the hardware actually guarantees. Correct TPU inference #6868 was quiet — the narrowed TPU parity PR #7556 has not moved since 26 July, and the long-context divergence it defers to #7555 is untouched, so the 16K/32K bounds remain compatibility smoke rather than a parity claim. The GPU performance target #6870 is still waiting on a defined "fast enough": #7758 reports correctness and placement, not tokens/s/GPU, and #7706 is explicit that its topologies are reasoned pre-benchmark choices whose decisive next step is throughput and utilisation measurement on a real serving export.

The consolidated launcher from last week's #7570 spent this week getting its dependencies out of Marin entirely. #7683 moved endpoint-backed evalchemy off the bespoke :evalchemy-tpu container and onto a commit-pinned Python 3.12 uvx environment running on the standard Iris task image, closing #7270; each benchmark installs its own extra plus CPU-only PyTorch wheels, and a Qwen3-0.6B H100 smoke ran MMLU and GSM8K through the new path with the first eval child installing 168 packages from the shared uv cache in 660 ms. Upstream, evalchemy#55 removed the accidental torch edges from six benchmark extras — BigCodeBench, CruxEval, LiveBench, MTBench, MixEval and alpaca_eval imported it only for cleanup, no-grad wrappers, RNG helpers, or eager local-model adapters — and CI now checks all 32 benchmark install and import paths. The same PR introduced config/external/{evalchemy,harbor,MarinSkyRL}: independent uv projects whose lockfiles are the revision source of truth, advanced by one explicit config/update-external.py command, so an external head can move without entering Marin's root solve. Russell Power asked for that to run on its own, and #7721 now bumps all three heads at 09:00 UTC daily into a single bot-owned PR, closing #7682 — which was opened after a Grug OpenCode run lost all three agent trajectories to a Harbor bug already fixed on main that the independently pinned dev wheel could not consume. #7750 finished the job: Harbor, harbor_config, Daytona and universal-pathlib are gone from Marin's root environment and lock, and the pinned external callout owns both policy preflight and execution. Policies moved out of Python at the same time. Benjamin Feuer pushed for this in #7688: in-distribution eval suites need to ablate context length, summarization strategy and timeouts without patching a worktree, and every such change meant editing Marin's central registry. #7723 added --harbor-config, which validates a native Harbor JobConfig YAML against the pinned schema before Iris opens, injects the served endpoint, model, task cap and resolved secrets, and writes a SHA-256 digest of the policy into the durable record; a live AIME smoke on v5litepod-4 finished in 13m31s with the policy digest recorded. #7750 then converted the remaining ten catalog policies — tb2, tb2-lite, the three SWE-bench variants, GAIA, BFCL, Aider, MedAgentBench and FinanceAgent — to checked-in YAML preserving their prior stable JSON and digests, and deleted HarborRunConfig and its lowering code, closing #7746. Python keeps what Marin actually owns: evalchemy definitions, suite membership, hardware and model selection, and secret-source declarations.

Alongside, #7684 pinned the nine agentic Hugging Face task repositories to full commits and represented each as an ordinary artifact resolving to GCS on GCP and S3 on CoreWeave, so a sweep no longer redownloads from Hugging Face per job: a one-task Terminal-Bench run staged 7 files and 34,255 bytes locally instead of pulling all 861 repository files, and a new rigging.filesystem.copy_tree replaced the GCS-specific helpers that had been constructing a GCS filesystem even when handed a CoreWeave s3:// prefix. Harbor still needs local task trees for checksums, container build contexts and sandbox uploads, so the staging boundary stays — 90.4% of the 17,406-file, 1,614-task catalog is under 16 KiB, and native remote reads would trade one large download for a great many small ones. #7674 documented the whole launcher with copyable commands and, in the process of exercising them, turned on distributed streaming for the Grug data-parallel serve: each of eight ranks now reads a 15.6 GiB shard rather than the full 124.9 GiB export, finishing in 14.63–14.75 seconds against 57–229 before. Russell Power noted in infra that the same configuration brought the 60B MoE's load time down to about ten seconds. The week's harder theme was that an evaluation still does not reliably survive its own infrastructure. #7835 is the sharpest case: Marin keeps completed Harbor trials in the eval pod's /tmp/harbor_workdir until the entire driver returns, so preemption before that point discards them — one multi-hour selective resume lost five completed trials. The regression is traceable, and the issue traces it: the pre-consolidation evaluator registered an on_trial_ended upload hook specifically for preemption safety, #7570's move into an isolated subprocess replaced it with a post-run sweep, and #7625 and #7750 each preserved that ordering while changing other things around it. #7789 is the companion: Marin restores trial directories but never seeds Harbor's job-level config.json, so a resubmission is planned as a fresh job. The concrete run behind it has 300 durable results, 297 with a verifier result and three left unscored after 504 upstream timeout; a safe resume should run exactly those three. #7848 seeds that config and uploads the trial directory at the right prefix, and drops Marin's identity monkeypatch now that harbor#53 excludes the snapshot-mode kwarg from resume identity natively. Russell Power's review question there was whether a user could accidentally resume some unrelated earlier eval; the answer is no — output_dir is minted per launch from timestamp, model, eval key and a random suffix, and restore only globs that run's own prefix. Benjamin Feuer also opened #7833 against the recurring bug class underneath several of these: Terminus-2 implements context summarization privately, so each failure gets patched at one agent-private seam and the class returns at the next. The latest instance swallowed a failed proactive summarization and dispatched a near-full conversation with no explicit token budget, which vLLM rejected outright; the prior instance gave the summary subagent no output budget at all, and 59% of summarizing trials fell into a degenerate no-LLM fallback scoring 0.19 against 0.82 for trials that never summarized. The proposal is to make summarization a first-class Harbor abstraction with a testable contract. Draft #7837 works the hardware end of the same problem for GB200 agentic runs: an Iris priority band and explicit serving cluster replace a worktree monkeypatch, and batch serving now separates queue admission from model startup after a healthy evaluation failed on cw-us-east-08a because all eight singleton GB200 slots were occupied and its endpoint stayed admission-queued inside vLLM's 40-minute startup limit. Benjamin Feuer rewarmed the Daytona snapshots and is adding a Harbor regression test so they cannot silently drift again.

On the reading end, the dashboard got two substantial passes. #7752 gave run detail a results header — the rolled-up primary metric with stderr, computed server-side by the same function the leaderboard uses — plus per-question outcomes that treat an unscored sample as its own ungraded bucket rather than a wrong answer, a faceted filter bar on Runs, a Debug tab that surfaces records which failed to parse during a scan, and a Postgres-free local mode so the SPA can be run without a database or cluster. #7802 then rebuilt the fleet pages around a single instrument: every benchmark is a gauge in fixed column order with the stderr as a whisker and a benchmark the model never ran rendered as an explicit dashed slot, so coverage gaps stop flattering a leader — the leaderboard mean is scored only over benchmarks a model actually ran, low-coverage rows are flagged, and compare scores models on their shared benchmarks. A new model page resolves how a model's many runs merge, showing the latest succeeded run per benchmark within the newest version cohort with a sparkline underneath so a regression is visible. #7759 cleaned up after the externalization: the migration renamed the runtime provenance key, and accepting evalchemy_image and eval_image as historical aliases restored 381 GCS and 199 CoreWeave records that had become unparseable without rewriting any object-store artifact. #7895 consolidated CLI and pipeline outputs under shared evals roots and switched ingestion to delimiter-based listings at the two supported record depths, so scanning stops traversing result, sample and trajectory payloads — across 632 records, 194,817 objects and 66.9 GB, cold ingestion fell from about 50 seconds to 15.1, and a cached pass takes 3.1. #7800 names what is still missing: an operator cannot yet tell from one run page whether an evaluation was incomplete, failed in infrastructure, timed out, produced malformed output, or finished with questionable grading. That is one line in the fifteen-point definition of done Benjamin Feuer wrote onto #6503 this week, whose more demanding entries are end-to-end parity tests placing every approved baseline within one to two standard deviations of a published result from a trusted venue, nightly spot-check parity to catch regressions, and an immutability guarantee that captured evals cannot be clobbered or mutated without a logged change.

Grader correctness stayed under scrutiny. #7657's self-contained MATH graders reproduce lm-eval-harness verdicts while skipping sympy's LaTeX parser wherever the answer shape already settles the question, and they agreed with the reference on all 34,624 differential pairs — but Rohith Kuditipudi's review found two real divergences anyway, both reproduced against the pinned reference environment. The port zeroes any bracketed answer containing a comma on the premise that no grammar production covers it, except function-call arguments, so (f(x,y)) parses fine upstream and scores 1.0 while the port short-circuits to 0; the differential run cannot see this because MATH reference answers never take that shape, which makes the divergence silent and asymmetric — it only ever turns a correct verdict wrong. The second is that dropping the reference's duplicated \% replacement is not a no-op, because str.replace is single-pass. It is a good illustration of why the eval-trust work of the past fortnight keeps paying: a differential harness over real data still misses bugs that only fire off-distribution. Rohith Kuditipudi asked in the evals channel whether marin.evaluation.graders is the right home for self-contained grading code, and it is where the module landed. New eval designs also arrived off the wish-list in #7090. Munot Ayush Sunil's #7658 proposes a procedurally generated, difficulty-parameterized proxy for SciCode with three response modes — multiple-choice for in-loop logprob scoring at 150M–1B, line-infilling at 1B–7B, and native generation at 7B and up — and difficulty as a vector of independent axes rather than a scalar, with D50 as the headline metric instead of a raw pass rate. Yiyuan Li asked how the published leaderboard numbers relate to the paper's 4.6%, and the answer sharpened the case for the proxy: the ~60% figures are subproblem accuracy over 288 items against the paper's all-or-nothing main-problem metric, and with subproblems nested in 80 main problems the minimum detectable difference on unpaired aggregates is around 11 points, so the top of that leaderboard is a single statistical tie. rfgordan's #7776 picks up robustness: nine seeded conditions over GSM8K's 1,319 items, six of them HELM invariance perturbations applied to the question text and three prepending 2, 8 or 32 unrelated prior exchanges, with zeroeval's JSON-answer scaffold left verbatim in every condition — a deliberate choice, since #7321 measured 76.5% against 38.5% on identical GSM8K outputs from extraction differences alone. The epic itself closed with both sub-issues done; the standing eval work continues under #6503.

Telemetry was rebuilt end to end and then broke the thing it feeds. #7840 gave Finelog an authenticated POST /v1/telemetry route that normalizes bounded JSON batches into an ordinary telemetry_v1 namespace, forwarded to the parent hub with the same cursor and at-least-once behavior as every other namespace; #7839 then cut Levanter, Iris, Zephyr and Marin's vLLM off Telltale and onto it in one move, with no dual-write and no compatibility shim, so the server wheel had to reach every cluster before the producers landed. Russell Power described the point of the REST boundary as avoiding a circular dependency: Rigging can post telemetry without taking Finelog as a hard dependency. The first release build panicked on the marin-dev canary while registering the namespace — eager registration called Handle::block_on from a Tokio runtime thread on disk-backed deployments, and a /health check that passed after Docker restarted the process hid it — so #7851 made registration lazy through spawn_blocking. Then the volume problem. Levanter's telemetry tracker published one row per histogram bin, per metric, per step; a six-layer MoE router produced 774 rows a step, and two training jobs on cw-us-east-08a drove telemetry_v1 from roughly zero to 24.5M rows an hour, 90% of it bucket rows that no dashboard or alert reads. Separately the TrainingProgressStalled alert's phase-enrollment query had no lower time bound and scanned the whole namespace once a minute, which at 68M rows saturated the Finelog hub's four vCPUs: load average pinned at 4.2, compaction time per hour rose from 165s to 5,145s, and queries stopped completing. Pausing the rule #7867 returned the hub to a load average of 0.46 in five minutes. #7868 is the real fix and reads like a measurement log — export reduced moments plus count and sum instead of buckets, republish phase every 60 seconds so a job that hangs before its first step still has a recent row, and bound enrollment to a 15-minute window, costing 339ms and 337 of 26.9K row groups at 302M rows, a cost that tracks the window rather than the namespace. The alternatives were measured rather than assumed: a 7-day lower bound cost 16.83s and touched 22.3K of 26.9K row groups, worse than the 5.43s unbounded scan that caused the incident, and the existing bloom filter on name pruned 11% at a cost of 7.66s because phase is written alongside every other metric and is present in nearly every row group. The same bounding logic recovered Iris controller RPC metric queries #7672, where a five-minute timestamp predicate cut a 16.954-second scan of the 988,423,984-row telltale namespace to 7 row groups and 254 KiB. #7863 is the larger follow-on: canonical job, task, attempt, worker, process and execution identity on application telemetry, a node agent running once per Kubernetes node for host pressure and DCGM GPU readings, and node_name on task pods so per-task rows can finally be joined to per-node hardware — the join #7804 exists to establish. Four Levanter trackers still export four different moment sets from the same SummaryStats, which is now written down as #7887.

Iris's priority model was rebuilt after it stranded real work for hours. Benjamin Feuer reported five federated jobs sitting PENDING at the parent for 6–12 hours against cw-rno2a #7703, a cluster whose 360 running GPUs were all preemptible iris-batch work: the federation availability gate compared a job's need against a peer's free chips alone, a number that is priority-blind and also subtracts unbound Pending pods, so queued work suppressed the very signal that would let other work in. #7718 has peers report held_by_band beside their free amounts, making a candidate's effective capacity its free amount plus everything held below its own band — an interactive job now reaches a peer saturated by batch work and Kueue does the eviction, while a batch job, outranking nothing, keeps waiting. Underneath that, the stored zero priority band turned out to be a bug factory: PRIORITY_BAND_UNSPECIFIED = 0 named a state no row should hold, five call sites each re-derived a band from it and disagreed #7734, so a child of a batch job counted against its user's budget, sorted ahead of production work under ORDER BY priority_band ASC, lost its inherited band across a federated handoff, and a batch-capped user could not spawn children at all. Russell Power's suggestion in review became the design: rename the zero value INHERIT, resolve it exactly once at ingestion, and assert it never reaches storage #7737. Kubernetes child jobs then inherited parent bands as they already did on GCP #7725, and direct dispatch finally applied budget demotion and ranked candidates by effective band before the promotion cap rather than after it #7727. #7792 found the budgets themselves misconfigured: none of the three CoreWeave clusters declared a user-budget tier, so every submitter fell through to a default limit of 1,000 while a single 8-GPU node is worth 10,688 points, demoting essentially all work to batch — on cw-rno2a, 432 of 512 GPUs sat behind priority-0 work and five six-node gangs waited as long as 4h51m on PreemptionNoCandidates. Preemption also stopped lying about itself: a Kueue eviction had been recorded as WORKER_FAILED or plain FAILED with an empty reason, charging a single-task job against a zero failure budget #7806, and now records PREEMPTED with the eviction condition cached across the terminating window #7808. Finally the dispatch token bucket, not Kueue, turned out to be gating large jobs — 128 tasks a minute gave a minhash job with 3,042 pending tasks about 135 pods a minute and cost a four-rack MoE run a 41-minute delay before Kueue ever saw it — so #7877 raised it to 512. That immediately exposed 16,351 pods and 10,466 workloads to Kueue on cw-us-east-08a, which OOM-killed at the 2 GiB fleet default and then flapped seven times on leader election lost when its stock 100 QPS client limiter queued requests up to 18.8s past the 10-second renewal deadline — the same failure mode that hit cw-rno2a the week before, fixed here with an 8 GiB manager and a 1000/2000 client limit applied through a Pulumi update targeted at the Kueue Helm release alone.

Several long-standing failures got root causes this week rather than restarts. The Kubernetes control loop could stall for hours inside terminal-pod garbage collection #7881: the list was unpaginated and the client reads the whole chunked body under a per-recv socket timeout that every arriving chunk resets, so the read had no upper bound at all — in a namespace with more than 1,300 terminal pods a thread dump showed control-loop parked in the response body for over 90 minutes while 64 finished GPU pods still reported as running. #7882 makes every list chunked, moves the sweep off the control loop onto its own periodic thread, and defers configmap and pod-disruption-budget cleanup behind a freshly read active-pod set so a stale read cannot delete shared resources out from under a retry. A bad iris-task:latest push replaced the multi-architecture index with an amd64-only manifest; four arm64 GB200 nodes cached it and, under imagePullPolicy: IfNotPresent, never re-pulled, so every task scheduled there died in stage-workdir with exec format error for hours while the attempt recorded only Init:Error — because that init container had never set terminationMessagePolicy, so the kubelet wrote no message at all. #7741 fixes the reporting and adds an ArchMismatchImageExecuted rule whose fallback signature matched 0 of 11,182 container statuses across all 208 nodes, and #7735 pins task pods to the deploy-tree image and refuses remote latest publication outright. Benjamin Feuer's Terminus-2 evaluations were failing against vLLM on cw-us-east-08a with 504 upstream timeout after 120s on 95 of 234 completed Qwen dev-set trials #7773: a cluster-tagged capability URL bypasses the parent's endpoint registry, so the relay sent no timeout and the decision handler substituted its 120-second default, overriding the child's registered 2,100-second inference deadline. Russell Power's call was that the parent cannot know what the child registered and should simply use a maximal deadline, so #7782 set it to an hour and reconciled the load-balancer backends to match. #7807 closed a hang that had survived one previous fix: botocore sets Expect: 100-continue on every UploadPart, aiohttp waits for the interim response before writing the body, and none of aiobotocore's bounds arms during that wait — 33 shards of a datakit normalize run sat in UploadPart to CoreWeave object storage for 13 hours with a correctly applied 120-second read timeout #6719, which a whole-request 600-second deadline now catches. Hardware got the same treatment: g5b91dc on cw-rno2a was blamed by name in two consecutive 10-node RL gang failures with IBV_WC_RETRY_EXC_ERR, 20 node-hours for zero banked steps, while staying Ready and schedulable #7869 — and because those gangs use leafgroup coscheduling, resubmitting was not a fresh draw, which is why #7871 asks both for per-job node exclusion and for a written answer on who may cordon a shared node. Fray was separately misplacing multi-rack GB200 gangs by hardcoding leafgroup coscheduling, splitting a 32-replica job 14/18 across NVLink domains instead of 16/16 and silently pushing expert-parallel all-to-all traffic onto InfiniBand #7753; #7755 derives the level from Iris's existing variant-aware rules. Monitoring caught up in parallel: a restored high-density infrastructure status page inside Grafana #7749 with regional provisioning success on a logit axis #7771, a federation reachability alert that pages after five minutes and records the July 28 cw-us-east-08a incident where the controller was healthy but its public load-balancer address timed out externally #7707, a 30-minute end-to-end canary submitted through iris.oa.dev to every peer #7708, kernel-deadlock node conditions #7738 after a CPU soft lockup cordoned the only CI controller node for three days #7732, and a training-stall detector gated on Levanter phase enrollment so a job too old to emit the new producers stops warning while training normally #7701.

On the infrastructure-as-code side, Will Moss imported all of GCP identity and access management into Pulumi's marin stack #7810 — 716 resources imported with no change to live GCP, human principals encrypted under the same KMS key as the state so no email lands in this public repo, and the CI preview redacted to match. Every binding is non-authoritative by design, so drift against grants nobody declared is still invisible and is tracked as #7713; the audit itself produced a list worth reading in #7715, including roles/editor and roles/iam.securityAdmin on the Google APIs service agent, which lets that identity rewrite the project's IAM policy, and ten Artifact Registry repositories granting read to allUsers. With IAM in code, granting access becomes a reviewable pull request: #7898 adds add-grant and review-grant skills that encrypt a principal into a PR and decrypt a diff for a human to confirm before merging and applying, closing #7262. Smaller Pulumi work made the tool usable: the state backend now lives in project config rather than a separate login step #7805, kubeconfig is finally out of state #7745, previews post one aggregated comment per PR with real red/green diff rendering #7740, the four genuinely different patterns infra/ has accumulated are documented #7568, a soft guard warns before a pulumi up against a production stack from a non-main branch #7897, and the Finelog Kubernetes deploy is moving off hand-applied manifests #7690. Echo became the place incidents live: #7739 imported all 34 checked-in .agents/ops records into tagged wiki entries and deleted them from git after read-back verification, and this week's incident writeups are cited from PRs by their Echo URL rather than reconstructed from Slack. #7747 made Echo search federate across the wiki, repository files at main, pull requests, issues and Discord in one ranked result set, indexing 3,053 files and 21,815 chunks, after #7680 found that hybrid search had never used the pgvector index at all and cut an unfiltered query from 322ms to 6ms. Romain Yon asked the obvious question — whether any of this actually helps agent sessions — and #7825 proposes answering it by classifying Echo calls in local transcripts as helpful, neutral or harmful; Russell Power noted this beats his previous plan of assuming it works. Elsewhere: a human-gated controller rollout skill replaced the single-cluster restart #7754 and Rafal Wojdyla reported all Iris controllers redeployed on current main; fsutil gave every Marin bucket across GCS, CoreWeave and R2 one browser and CLI #7689; multi-node dev GPU sessions now hold whole nodes gang-scheduled inside one NVLink domain #7787; and Zephyr's Arrow-native path measured 1.76x on a pinned 1M-row map/filter/write benchmark and 25.7% less stage CPU on a 6.4M-row federated A/B, against a 10x end-to-end target that is being treated as an evidence gate rather than a forecast #7888.

Last week's tool-calling SFT work ended with Levanter's tokenization byte-exact against Axolotl on real agentic rows; this week the question moved one stage upstream, to whether the trace corpus feeding those templates is even readable as SFT. Will Held sampled 175 Qwen3.5-122B-A10B-FP8 131k OpenCode traces from the agentic-trace index #6191 and found 170 of them with an empty first user message across 34 of 35 datasets — empty in the raw Hugging Face rows too, so ingestion had not lost anything, and his read was that these suit pretraining rather than SFT. Benjamin Feuer rechecked 220 rows across 44 queryable datasets, confirmed 185 empty first turns, and located the missing content elsewhere: 162 of those rows still carry a non-empty first-step prompt_token_ids with matching literal completion data. The distinction is the harness. Terminus-2 runs inside Harbor's own inference-side harness, so Harbor observes the structured trajectory and exports task-grounded messages, tool calls, and tool results directly; installed harnesses such as OpenCode run in a remote Daytona sandbox and call back to the inference service, so the displayed conversations field is a lossy projection and only a recording proxy captures the teacher's literal prompts and completions. Training on conversations for those rows would be wrong; the literal tokens have to be decoded with the teacher tokenizer, parsed into structured messages and tools, and re-rendered with the trainee's tools-aware chat template.

The gap that made the datasets easy to misread — no standalone trace-to-SFT entrypoint in Marin, the reconstruction tooling still living in OpenThoughts-Agent — is what #7830 now proposes to close, in draft. It adds a Datakit transform and reproduction recipe that scans an input dataset's agent column, requires one uniform recognized harness, preserves Terminus-2 conversations as literal ground truth, and rebuilds installed-OpenCode exports from prompt and completion token IDs. Unknown, mixed, and mismatched harnesses fail before conversion, as do missing or misaligned records, malformed tool schemas or calls, and unpinned teacher tokenizers — the design point is that it refuses to emit promptless examples rather than emitting them quietly. Validation is the substantive part: the checked-in recipe pins all 29 Hugging Face sources and the teacher tokenizer behind grug-67b-a2b-agentic-sft, with an expected 77,012 accepted records, and a full campaign on the CoreWeave rno2a cluster passed the row-count gate for 28 of them while catching one real provenance error. curriculum-medium had been pinned to a later literal-rescue rewrite and produced 455 records instead of the historical 451; pinning the revision the archived training run actually used reproduced all 451 at the canonical-record level, and the corrected job then completed on rno2a with 451 accepted, zero failures, zero preemptions. A 22-row compact golden matches the archived training parquet exactly. That is a replay of a shipped SFT mixture from durable links, and it is the sort of check that only finds anything when the pins are wrong.

The trace inventory those pins draw on also got much larger and much better bounded. An org-wide schema scan of laion, DCAgent, DCAgent2, mlfoundations-dev, and penfever, keyed on the conversations-plus-agent column signature every Terminus-2 / Harbor trace dataset shares, turned up 11,337 agentic-trace datasets; deduplicating against the AgentTrove source table and the #6191 cohort index (337 known, 298 re-confirmed by the scan) leaves 11,082 unregistered. Benjamin Feuer also reconciled the index itself against the live campaign trackers, adding the successful GLM-5.2 / Terminus-2 rno2a cohort of thirteen task-sets and two newly published Qwen3.5-122B 131k OpenCode datasets, with failed, incomplete, and verifier-invalid runs deliberately omitted. The sibling registration PR #7824 merged 142 of those pinned Penfever repositories as pretraining text; #7830 is written to reuse that same source catalog as a transform rather than maintaining a second rollout inventory, so a registered dataset can feed either the text path or the harness-appropriate SFT path.

On the rollout-source side, #7711 merged, ingesting the 3,200 GLM-5.2-FP8 KernelGym trajectories in marin-community/glm-5.2-kernelgym-rollouts — KernelBench-style kernel-optimization tasks split evenly between inline CUDA and Triton, each row a repair loop that writes a kernel, gets the evaluator's verdict back, and reworks until it passes. The scoping issue #7710 and the implementation both turned on things the schema does not advertise. The messages field carries sampler bookkeeping that turns[] does not: every row ends with a bare KERNELGYM_FINAL sentinel, and 122 further assistant messages across 85 rows are rejected answers to an interleaved "is another attempt worthwhile?" prompt, so each is dropped together with the prompt that drew it and no unanswered control prompt survives mid-transcript. Truncation is common and stop_reason does not flag it — 2,459 of 6,183 generation turns spent the full 16,384-token budget, detectable only by completion tokens hitting the cap — so TruncationFilter is a required argument rather than a default and enters hash_attrs, meaning the strict ANY_TURN variant (1,946 rows dropped) is a rerun at its own output path and cannot be served from the lenient cache. The registry ships FINAL_TURN, dropping 365 rows; the chain ran end to end on cw-us-east-02a for 2,835 kept rows and 64,054,289 tokens after a rework and re-measure. Reasoning stays inline, with the dangling </think> becoming a paragraph break rather than being deleted, since GLM emits it with no surrounding whitespace and stripping it would glue the last reasoning sentence to the first word of the answer. Will Held folded the corpus into the public token-count viewer in #data-mixing alongside the Stack-V3 swap, putting the raw pre-dedupe pool at 25T tokens. Feeding the next cohort, the GLM-5.2 generation run over the 959,216 instruction_seed prompts of hero_run_4_code #7697 launched its full 128-shard production queue at batch priority after cache-resume support in #7698 preserved partial regional model-cache files through a retry, with 88 complete servers at tensor parallelism (TP) 8 and expert parallelism (EP) 8 running on 704 B200s in the first snapshot and deterministic shards so preemption costs a relaunch rather than regeneration. And Russell Power tied the data-manager epic #6496 to two explicit contracts: ingress #7799, an immutable normalized pool version that SFT and RL runs fingerprint instead of reading a mutable Hugging Face head, and egress #7584, an appendable columnar archive of evaluation records and trajectories.

The raw pool reached 25.2T tokens pre-dedupe across 144 sources, 25.5% synthetic, up from 23.2T over 139 sources a week earlier. The largest move was the code swap finishing: Stack v3 entered the registry at 4.57T while Common Pile's Stack v2 and the Dolma 3.5 code derived from it left, a net code swing of roughly +3T. Will Held posted the refreshed token-count viewer and drew the operative conclusion from it: with an end-of-year budget of at most 20T tokens, the mix no longer has to be built purely by up- and down-weighting sources, and can afford outright quality filtering — dropping the lowest-quality tokens until the deduped count fits the budget, on top of the store infrastructure Rafal Wojdyla built. Then the Stack v3 number itself moved: the duplicate-file keying fix #7788 removed 1.205 trillion tokens from the source, and the consequence for this epic is that every mixture computed against the old registry count had been over-weighting stack-v3 by roughly 35% relative to its real content. A companion correction #7828 replaced the GitHub Actions log corpus's legacy 150B placeholder with a measured 253.34B tokens over 2,613,421 documents, so that the source registry is the single token count mixture planning reads — possible only after #7823 taught materialization to skip 58 malformed nested archives in the published Zenodo release rather than burn every shard retry on them.

The open question is how much of the existing swarm evidence survives all of that churn. The surrogate-reuse debate between Percy Liang, Rafal Wojdyla and Larry Dial — whether a frozen per-source embedding lets the existing 840 swarm points carry across new buckets, a new token budget and a new architecture — belongs to the mixture-surrogate work; what it settles for the data mix is the sequencing: as many mixing experiments as fit get run once the new data and architecture are pinned, with old and new runs fit as one model and checked on held-out rows. Separately, Larry Dial flagged that some sources in the current mix epoch up to seven times and asked for token-level train-loss-versus-epoching analysis on exactly those buckets, and Will Held circulated prescriptive scaling laws for data-constrained training to the people fitting the penalty.

Michael Ryan's weekly sync on the LLM extraction recipe #2351 reported that the 3,000 raw crawl archive (WARC) extraction of llm_pipeline_v1 finished, which is enough data to resume distilling the recipe back into the fast ModernBERT and FastText pipeline. He also rewrote the extraction spec after finding the old one dropped parts of code blocks — execution output lines, for instance — in ways Levenshtein distance forgave but downstream training would not, and switched the output back to markdown on the rule that discarding information is a bad trade for lower loss or benchmark scores. To score that he defined a fidelity metric over normalized 5-grams: KEPT (the fraction of gold sentences that survive), CLEAN (the fraction of emitted text that exists in gold), and their F-score with recall weighted twice as heavily, plus the same construction restricted to code lines, math expressions, table cells, and speaker attributions. Models trained on the new spec come in slightly behind llm_pipeline_v1 on loss and benchmarks at 300 WARCs, which he attributes to no validation set or benchmark targeting markdown formatting, and his honest read is that hill-climbing the spec was harder than hoped. Coverage against other corpora is uneven — 88.3% of DCLM and 89.6% of the internal high-quality v2 set, but 62.4% of Nemotron-CC and 43.6% of the medium-quality tier, the misses being mostly short travel and informational text. On mixing, he is running the Olmix method but bucketing by quality × domain rather than topic alone, optimizing a flat unweighted mean of 42 bits-per-byte (bpb) tasks (19 code, 15 QA, 8 math) with DCLM Core v2's 22 tasks strictly held out of the objective; the swarm needs roughly 72 hours and preliminary points already favor the high-quality extractions, with transfer to the held-out DCLM Core suite still the open question.

Two new content types moved from wish to spec. The DOCX import landed as a stopgap in #7733, taking 245,018 entries from the public DocxCorpus manifest down to 242,086 deduplicated documents worth 1.497B tokens under ODC-BY metadata terms, with original document copyright and no dataset-specific PII filtering. #7736 then specs the real version: a pipeline that queries the Common Crawl indexes by MIME type and URL directly rather than reading the DocxCorpus manifest, because DocxCorpus has extracted only a fraction of the DOCX files in the crawl, reusing the WARC range-fetch and provenance fields from the focus-crawl step. Bilibird raised the sharper question of whether the resulting tokens are worth having, noting that many of the imported documents are unfilled form templates. On the PDF side, #7617 pins the input to the CC-SUPPLEMENTAL-2026-22 focus crawl on the public Common Crawl bucket and the output to Marin's normalized-document shape with WARC-record provenance, so the four build steps under #7616 get a stable target, and flags that the personal-data and copyright risk of PDFs needs review before any public release. tinuademargaret volunteered for it; Will Held redirected, since the PDF work is an onboarding task for an incoming Open Athena hire, and offered to spec the other unextracted Common Crawl MIME types instead — #7736 is the first of those. Meanwhile the long-dormant proposal to ingest GH Archive event JSON #5099 was closed as stale, and Will Held expects the last large pretraining token bump of the year to come from his Nemotron code reproduction, whose reconstructed contents registered in #7811 at a provisional ~465B tokens — though he suspects it overlaps heavily with Stack v3.

Last week's surrogate work ended with a plan to keep sampling widely and interpolate inside the sampled regime, on top of the existing 840 swarm runs. This week the question became what exactly transfers. Percy Liang opened the thread with three points. First, rather than asking how far the kernel ridge regression (KRR) surrogate and Calvin Xu's effective-exposure DSP extrapolate in the abstract, solve for mixtures w(λ) = argmin_w Loss(w) + λ·Dist(w, swarm points) and see how low λ can be pushed — the aim is not arbitrary extrapolation but the best mix subject to the radius of generalization. Second, the point of parametrizing the predicted loss by embeddings of individual documents is to reuse old swarm data as buckets change and new sources such as Stack v3 arrive, but the target token count — which sets simulated epoching — and the architecture are changing too, and it is unknown how much those matter. Third, a new swarm on the updated data and architecture will run regardless, so the useful move is to fix the sampling distribution from what has already been learned: it makes no sense to evaluate a mixture that downweights high-quality data, and a smarter design may allow a smaller swarm. The risk is bounded because, unlike scaling laws, the predicted mixture can simply be trained; if it comes out badly it becomes another training point, so the whole procedure can be incremental.

The substantive disagreement was about what belongs in the config. Rafal Wojdyla's working hypothesis was that a relative-frequency vector over a frozen embedding space describes the probability of sampling a given flavor of data and so does not depend on raw volume or exact bucketing, which is what lets evidence carry across; source volumes and the target budget then enter through simulated epoching and the repetition penalty. Percy Liang pushed back: what is being modeled is a function from config to loss, every (config, loss) point has to come from the same function, and anything held fixed in the function — architecture, total token count — has to move into the config, which the embedding alone cannot do because it does not know about token counts. Rafal Wojdyla answered that the epoch-penalty term does carry both, since epochs are a function of budget and source count. Will Held asked whether KRR gives Bayesian search natively, and Percy Liang's position was blunter — just do Bayesian optimization — which Rafal Wojdyla was happy to try. Larry Dial asked whether the original design was a single sweep with no priors on what data is good and whether extrapolation is only needed because the team is one-shotting the optimum from that sweep; the answer was yes on both counts, with an iterative component wanted as far as time permits. The plan of record is to fire as many mixing experiments as fit once the new mix and architecture settle, fit one model over old and new runs together, and check the transfer with held-out evaluation.

What makes the sampling-design question urgent is that the pool has outgrown the budget: the raw pre-dedupe pool now stands at 25T tokens against an end-of-year budget of at most 20T, so selection can be expressed as outright quality filtering rather than only as an up- or down-weight — which changes what the mixture surrogate is being asked to optimize. A further bump is expected from the Nemotron code reproduction, though likely heavily duplicated against Stack v3. The first diagnostic question on the new DOCX source came from Bilibird, who noticed a lot of unfilled form templates in the sample and asked whether those are tokens worth keeping; the source landed as docx-corpus/en at 1.497B tokens #7733, with a proper Common Crawl DOCX-to-text pipeline still open as #7736. Will Held also flagged Liquid's 230M encoder as a candidate for quality tagging.

Two accounting fixes moved mixture weights directly. The Stack v3 duplicate-file keying fix #7788 took the source from 4.568T to 3.363T tokens over the same 172,898,790 documents, changing the denominator every mixture weight is computed against. GHALogs was re-measured exactly at 253.34B tokens over 2,613,421 documents, replacing a legacy 150B estimate and making sources.py the sole token count used for mixture planning #7828; the reconstructed Nemotron-Code-v1 contents are registered with an explicitly provisional ~465B count scaled from v2's measured tokens-per-document, pending real tokenization #7811. On the diagnostics side the week was mostly housekeeping: the served full-document perplexity-gap heatmap viewer #5205 was swept closed as stale, as were the remaining held-out perplexity slice proposals for chat and agent traces #4963, game and music notation #5062, and GH Archive event JSON #5099, continuing last week's sweep of the #5005 backlog; the perplexity-gap tooling itself was kept green by moving its tests onto checked-in tokenizer fixtures instead of Hub downloads #7862. Following last week's finding that repetition harm collapses to −0.003 bits per byte at a 100B-token budget #7067, Larry Dial proposed a token-level look at train loss against epoching for the buckets that repeat most, noting that some data in the current mix is epoching up to seven times.

The August hero-run recipe lost a component and gained a schedule study. Larry Dial closed multi-token prediction (MTP) out of the upcoming run #5772 for three reasons: memory pressure on the target d6120 model, where fitting even an MTP head with a hidden-dim-wide MLP and no MoE on it costs 7% throughput — and that head is already much narrower than any lab has shipped; complete loss blow-up on roughly 10% of small-scale runs with MTP on, root cause unresolved between the fused cross-entropy over both heads, a precision issue in the H100 kernels, and an interaction with hyperball on the MTP component; and the precedent, since DeepSeek trained V4 with MTP but dropped it at inference in favour of DSpark, while Poolside and Arcee Trinity Large never trained with it at all. If inference speed matters, DSpark gets trained after the main model. With MTP out, the schedule sweep #7717 tuned what remains: 25 runs on the current-best d768 grug MoE sw2k-g4 recipe over the datakit two-stage mix, 8,420 steps and 4.42B tokens each, comparing linear decay, three warmup–stable–decay (WSD) stable-phase lengths, and a 1−√ cooldown across peak-LR multipliers. 1−√ at 1.4× the MuonH heuristic peak was the best point at 3.2642 Paloma macro loss, ahead of best-linear 3.2670 and best WSD-10% 3.2671; the fast early decay tolerates a much higher peak (still fine at 1.8×) where linear and WSD top out at 0.9–1.1× and degrade above, and longer stable phases were monotonically worse. Larry Dial declined to promote it anyway — ~0.003 nats is not enough to change the recipe on, and the setting will be retested at larger scale first. The larger question is now specified as a 150-run sweep #7856: six widths from d512 to d2048 × five token budgets from 30× to 600× active parameters × five LR multipliers, testing whether the May-Recipe MuonH heuristic (which falls as tokens−0.395) stays compute-optimal as both width and token budget scale, or whether the hero recipe needs a systematic correction. Per-cell base LRs and step counts are written down; no runs have launched. Architecture reading continued alongside: Larry Dial flagged in the news channel that Kimi K3 drops rotary position embeddings from multi-head latent attention (MLA) entirely and recovers local position awareness through KDA layers — a direct echo of last week’s transition staircase, where going to no positional embedding cost +0.0653 Paloma — and read K3’s per-head gate as an MLA-specific artifact, since MLA’s key–value matrices are not square and per-head is closer to square; Kaiyue Wen noted that GLM-5 does the same.

Russell Power ran the nested expert-bank question to a conclusion in #7667: can one E256 MoE be trained so that experts 0–127 and 0–15 are also usable as standalone banks, inside a single forward pass, backward pass and optimizer update? The week opened by invalidating the earlier burn rather than defending it — its control forced one document per 8,192-token example, bounding average useful-target occupancy at 28.05%, and its nested-evaluation callback perturbed training state, so nominal tokens overstated real targets by at least 3.57×. The corrected experiment reproduces an immutable d768 source through update 1,000 to a median absolute pointwise train-loss error of 0.002285 nat before comparing anything. The follow-up that mattered was preregistered: a five-arm 10B-token plan pinned at a commit, isolating E128 and E16 prefixes against a rotating two-of-eight-layer schedule, with a +0.10-nat Paloma stop threshold and a two-consecutive-gate stop rule — a rule that was genuinely exercised, since the treatment crossed +0.10 once at update 7,000 and then recovered rather than repeating. The endpoints are clean: matched standalone E128 finishes at 3.181439 Paloma, nested-naive E128 is +0.033787 behind, nested-layerwise +0.150324, and the best post-hoc chop of an uncompromised E256 checkpoint +0.357842. Direct training wins at equal tokens, so nesting is not a quality trick — it is an economic one. Joint training plus direct cooldowns reaches the E256 target after 3,500 updates and standalone E128 after 6,000, for 48.86 H100-hours (1.240× one E256 run) against 74.95 hours and 1.901× for training the two separately: a 34.8% total-compute saving, or 1.148× critical-path time if the cooldowns run concurrently, with the eligibility mask itself costing 0.41–0.49% per step. A zero-E256-degradation strongman — train E256 normally, extract the best post-hoc E128 bank, cool it down — did not overturn that, and balanced complements produced near-identical banks but a 31.1% quality-adjusted full-model tax and is Pareto-dominated. The same burn also surfaced a real bug, filed as #7712: the periodic full/E128/E16 evaluation callback changes the next training update. Median treatment loss jumped from 4.692 to 7.532 across the windows after evaluation at update 10,000, and an earlier gang went non-finite three updates after the same callback. A same-topology counterfactual settled it — two arms copied from the same clean step-10,114 checkpoint onto identical eight-GB200 meshes matched within 0.00866 nat until the callback, after which the evaluating arm went non-finite at step 10,123 while the other stayed finite through 10,137. Production now runs with periodic evaluation disabled; terminal evaluation is still safe because no optimizer update follows it.

The other Russell Power line asked whether cheap shallow training can produce a ~5B-active checkpoint fit for SFT or RL more than 5× faster than clean full-depth training #7836, implemented in #7838. No arm cleared it. On 16 GB200s the uniform d3072/L48/E64 top-4 control finished in 10h22m at 1.9903 train loss and 2.8192 Paloma micro loss. Parameter- and FLOP-matched fat-first and fat-middle layouts — shared capacity moved toward the input — were a flat null, 0.98× speed and terminal losses within 0.002 of the control. The 95/5 narrow-and-shallow-to-full arm hit 4.77× but stalled at 4.0338 / 4.8286, and a 90/10 variant that kept 64 experts and top-4 routing managed 3.33× at 4.8521 Paloma, so a longer full-depth tail did not rescue capability. The most interesting result was a negative one about stochastic depth: random sample-four layer dropout trailed a physically four-layer stack by 0.8885 trailing-64 loss during the shallow phase, then passed it over the common full-depth tail and beat it on all 16 Paloma subsets — yet both still lost to simply stopping a clean run early, on every subset. What survives is a systems observation rather than a recipe: the fixed-64-expert d768/L1 source sustained 2.50M tok/s, 13.14× the control, at zero routing overflow, but never produced a full-size checkpoint.

Three configuration defects in the shared recipe came to light. Neha Hulkund found that GrugMoeLaunchConfig.init_from is a documented no-op #7742: it resolves into TrainerConfig.initialize_from, which neither grug train loop reads, so every chained phase silently trained from random init — her d512 mid-training run opened at train/loss 11.803 against a random-init 11.802, and every eval came out worse than the base it was supposed to continue from. Russell Power handed it to weaverbot, which fixed it in #7791 by loading prior weights (plus pending_qb_betas for MoE) with a fresh optimizer and step 0, while keeping own-run full-state restore ahead of it so preemption recovery is unaffected. Matt Wittmann reported that weight_decay=0.1 is set on the Grug MoE optimizer and never read #7783, which means every recorded Grug MoE measurement to date ran at zero decay — the numbers are internally consistent and it is the configuration that misleads, so implementing decay now would need its own matched A/B rather than a quiet fix. And Tao Xu proposed replacing the current quantile-balancing (QB) implementation — local-batch quantiles averaged across replicas, straight from the blog post — with the K3 paper’s bucketed all-reduce approximation of the global-batch quantile #7812, citing Qwen’s finding that global-batch auxiliary loss beats the local-batch form. On mid-training, Neha Hulkund ported the Delphi K=0.20 continued-pretraining recipe onto the grug MoE ladder at the d512 rung #7744: the mix ranking from #6279 reproduces under routing, but the axis is nearly flat here — across the whole replay/math range held-out finemath loss moves 0.025 while Paloma macro moves 0.092, so on validation loss alone the least-math mix is the better trade, the opposite of the original pick. A parallel dense-ladder replication at 3e18, 9e18 and 2e19 #7790 reproduced the original Paloma macro numbers to within 0.14% relative, reconstructed by transcribing each completed run’s own manifest rather than re-deriving the cells from scaling heuristics that have since moved. Finally, Russell Power closed out the Qwen3 32B→0.6B distillation study #7656 with its report and implementation in #7671: rank-512 factorized teacher initialization, the only treatment promoted from the 100M-token screen, reversed at 1.8B tokens and finished 0.39% worse than plain from-scratch forward Kullback–Leibler (KL) distillation on mean held-out negative log-likelihood (2.6644 vs 2.6541), while official Qwen3 0.6B Base initialization plus forward KL led both the loss (2.5579) and the four-task zero-shot mean.

Nothing landed directly against this epic: no pull requests or issues were assigned to it, its own comment thread stayed empty, and last week's serving work — the single marin-serve lifecycle and the persisted vLLM compilation cache that cut manager start-to-ready from 59 to 30 seconds on an H100 — drew no follow-up. What rollout-speed thinking happened this week happened upstream of the serving stack, in deciding what shape the ~360B candidate has to be served in at all.

Romain Yon wrote up serving-topology starting points for the July 27 ~360B-A22B GrugMoE candidate under high-concurrency rollout and batched-eval load #7706, explicitly optimizing tokens/s/GPU and sustained utilization rather than per-request latency. The sizing proxy rules out the obvious shapes: with tensor parallelism (TP) at 1, expert parallelism (EP) shards the 347.9B routed weights but replicates the 11.7B non-routed weights on every GPU, so EP8 needs 102.8 GiB/GPU — 822 GiB across one 8×H100 node against 637 GiB allocatable — and a four-GPU GB200 node at EP4 leaves 1.2 GiB/GPU before any KV cache. The recommended first shapes are two H100 nodes at pipeline parallelism (PP) 2 / TP1 / data parallelism (DP) 8 / EP8, keeping each eight-GPU expert group node-local so only one pipeline boundary crosses InfiniBand, and four GB200 nodes at PP1/TP1/DP16/EP16 inside one NVL72 NVLink domain. He flagged these as reasoned pre-benchmark choices — the exact final architecture is unsupported, the DP×PP×EP engine path unverified — and posted them to #inference for comment. The engine-side precondition is in draft: running GrugMoE pipeline parallelism through native vLLM across Iris tasks #7758 passed all 64 Snowball goldens at both PP1 and PP2 on cw-us-east-02a, with placements and layer ranges verified before a prompt is sent.

The other measurement worth having is what it costs to hand a fresh policy to the rollout workers. A synthetic screen of the 719 GB BF16 policy over NCCL #7695 put the current-shaped 1→8 fan-out at 7.31 seconds (98.4 GB/s), a 1→1 bulk floor at 26.96 seconds, and an ideal pre-sharded 8→8 transfer — which MarinSkyRL does not implement — at 3.97 seconds. These are warm memory-to-memory transfers over an already-initialized communicator, so 7.31 seconds is an optimistic floor rather than policy-refresh latency; it excludes parameter export, per-parameter synchronization, vLLM weight loading, and the generation pause. The H100 rows are still blank, and that is the finding that matters for an epic aimed at H100s: all 12 bidirectional direct-TCP probes between the sampled GB200 node and both H100 clusters ended in EHOSTUNREACH or timeout, and which network component rejects them is unknown. Two smaller pieces moved in the same direction: RL vLLM weight reloads dropped their Flax dependency for TPU-inference's flat_state/get_value protocol #7693, keeping the attention-head reshape and 128-element padding hot swaps need, and a bounded vLLM Prometheus bridge landed #7863 toward the ask that inference workers report throughput, time to first token, time per output token, queue depth, and KV-cache use on the same path as everything else #7798. In Discord, Rohith Kuditipudi proposed retiring Marin's save_logprobs path in favor of requesting prompt_logprobs from vLLM directly — he is its only client, and the one gap is that vLLM on TPU will not serve prompt logprobs with automatic prefix caching enabled — and Russell Power agreed to prune it.

The week's substantive move was Russell Power's #7883, which makes pretraining, SFT, SkyRL reinforcement learning, and Evalchemy/Harbor evaluation expressible as a single typed artifact graph while keeping SkyRL an isolated external runtime: Marin installs only an exact launch-only revision into a disposable environment, and the GPU task uses a digest-pinned image whose embedded trainer revision is validated before submission. The RL artifact owns immutable model and data locators, a semantic role plan, durable checkpoint and attempt roots, and a terminal manifest; a run resolves only after MarinSkyRL validates the exact Hugging Face policy export, which then feeds the shared evaluator with no model-catalog edit. The iceball-micro experiment exercises the whole graph on a random-init Qwen3-0.6B-size architecture — a pinned FineWeb-Edu slice for 16 pretraining steps, No Robots for 8 SFT steps, GSM8K for 8 group-relative policy optimization (GRPO) steps, then the terminal policy served once for Evalchemy GSM8K and Harbor AIME smoke tasks. Evalchemy returned 0 exact-match on 16 samples and Harbor 0 reward on both AIME trials, which is what a deliberately tiny random-init model should produce; the result being validated is that both evaluator paths recorded. The resilience evidence is the more interesting part: the RL attempt resumed from step 6 after a planned preemption, and when a replacement worker later stalled during the last policy mini-batch and was itself preempted, the next node resumed from the same durable checkpoint and finished. The change depends on MarinSkyRL #275 at a pinned commit and closes #7797, the versioned-ArtifactStep spec filed earlier in the week under Russell Power's standing goal of driving every post-training experiment from the Marin repo #7098. It was still open at the end of the week, at roughly 3,500 added lines. A self-review pass resolved 14 of 15 advisory findings by centralizing defaults, using the repository storage abstraction, and dropping literal-only tests; the two retained fields, target_cluster and parent_cluster_config, are kept deliberately as the federation escape hatch that preserves the existing launcher's --target-cluster semantics, with ordinary same-cluster artifact runs leaving both unset — the contract #7098 asked for, that the launcher boundary not encode Iris-only fields into the experiment contract.

Last week's offer from SkyRL's maintainers to rebase MarinSkyRL onto upstream HEAD produced no visible follow-up in Discord or the repo this week; the scoping doc from the SkyRL side is still the awaited next step, and the fork remains where the work is happening. Two concrete framework gaps surfaced meanwhile. Benjamin Feuer pointed out that the Delphi MoE is a custom architecture, so MarinSkyRL will need a PyTorch training interface for it before the midtrained MoE checkpoints can go through a vanilla GRPO comparison — the suggested interim path is light SFT and a handoff to evals rather than blocking on RL. And #7654 closed: MarinSkyRL's nightly single-H100 GRPO gate on GSM8K had been submitting an Iris job under a fixed canonical name, and Iris replayed the previous day's task output — July 27 timestamps, the old Ray session path, the old PID and its libcudart.so.13 traceback — then never reported a terminal state, burning the full 90-minute CI budget without the trainer's own first stdout ever appearing. MarinSkyRL #186 mitigates by including the Actions run ID and attempt in every Iris job name; the underlying Iris behavior of replaying a terminal task's logs on name reuse is noted as unaddressed. Separately #7798 specifies the operational telemetry the framework still lacks — training, rollout, and inference workers emitting one durable Finelog stream keyed by root run, Iris job, task, worker, role, rank, and restart attempt, so a bounded RL run can be diagnosed from Grafana without opening worker-local logs, with W&B kept for experiment tracking rather than made a dependency for diagnosing a live or failed worker. Housekeeping: #7693 removed the Flax NNX dependency from RL vLLM weight reloads, since Flax 0.12.4 imports the removed jax.core.Effect under JAX 0.11 and broke import of the inference context at test collection; source weights now use TPU-inference's flat_state/get_value protocol, retaining the attention-head reshape and 128-element padding that hot swaps require, and the safe RL suite collects and passes 194 tests under JAX 0.11.

Romain Yon turned the Levanter-versus-MarinSkyRL training gap on Grug 67B into a content-addressed fixed-replay benchmark #7878. On a frozen 4,096-sequence replay and a four-host, 32-H100 topology, Levanter's fastest valid update takes 255.859 s and MarinSkyRL's takes 6,991.670 s — 27.3×, a 6,735.810 s gap. A separately labeled present-day matched-cross-entropy run instruments all 26 routed expert modules and finds routed experts consuming 5,583.786 s, 81.5% of that run's 6,854.120 s critical-rank wall, with an exact nonexpert remainder of 1,270.334 s. The report is careful about what that does and does not establish: even crediting all of Levanter's 253.176 matched-cross-entropy seconds to expert work leaves a 5,330.611 s hybrid difference, but it labels this scale context rather than a lower bound on the frozen gap, because the full-replay stacks use different represented parameters and routes and the aggregate 111.351 s source bridge does not partition component drift. It also retires the older cross-stack matched-cross-entropy causal claim outright — Levanter starts from FP32 masters while MarinSkyRL starts from stored BF16 parameters, and the two full-replay losses differ by 0.95% — with a smaller common-representation oracle used to show the stacks agree semantically when both load the same represented values. The next discriminator is named concretely: run the parity-passed grouped experts against the eager expert loop on this same 32-H100 replay and boundary, with identical represented values and attribution enabled in both arms. Romain Yon then set the mixed-scope PR aside unmerged, moving the one focused correctness fix into #7894 — FA4 prunes key tiles using the first query in an M tile, so the old seq_len sentinel on leading padding could discard keys that later valid queries in the same packed segment still need — and keeping the benchmark, replay drivers, and report on a preserved branch.

The other measurement was the data-plane cost of policy refresh #7695, whose transfer numbers sit with the rollout-speed work; what it establishes for this framework is that the measured 7.31 s for one 719 GB BF16 policy is an optimistic floor rather than policy-refresh latency, because MarinSkyRL builds one update group containing trainer rank 0 and every rollout rank and broadcasts one parameter at a time while the benchmark sends the same bytes in bulk over an already-initialized communicator, and the ideal pre-sharded row is a ceiling for a shard-aware sender MarinSkyRL does not implement. The H100 rows have no number at all, so policy refresh to H100 rollout workers stays unmeasured. On the running fleet, Benjamin Feuer reported median step time down to roughly 1,000 s across datasets on 32×H100 at 32k context with router replay, Qwen3-Coder-30B-A3B, and the opencode harness doing real tool use with remote container ingress and egress managed by Iris — while noting that the hyperparameters were tuned for weaker models and are almost certainly too aggressive at this scale.

The two-week agentic-RL campaign closed as #7784, and its most valuable output was not the one it set out to produce. The stated goal was to make end-to-end agentic RL work at all on CoreWeave Iris — Qwen/Qwen3-Coder-30B-A3B-Instruct with OpenCode as the agent harness, 221 merged pull requests across MarinSkyRL and marin-community/harbor — and then to use the working stack as an instrument for measuring TaskTrove dataset quality. Both happened, and the second turned out to be worth more. The sweep drove TaskTrove from v3.7 to v3.17, repairing roughly 195k task instances and making a further ~155k RL-ready across about 40 datasets. Efficiency moved hard in the same window: per-arm footprint fell from 8 nodes / 64 GPUs to 4 nodes / 32 GPUs while the context budget rose from 32,768 request tokens / 4,096 new tokens per turn / 30 turns to 131,072 / 16,384 / 90 — a 4× context window and 3× turn budget on half the hardware — with context parallelism (CP) taking Megatron backward-pass out-of-memory failures from ten consecutive losses to zero, a result Benjamin Feuer was careful to frame as survival rather than headroom, since two single-shard arms polled at 75–77% memory and then died within ~2 GiB of the card. The scope caveat was stated up front and repeated in the wrap-up posted to #reinforcement-learning: every arm was measured on its own training-reward distribution, nothing was evaluated on a benchmark or held-out set, and across eight salvaged checkpoints no arm demonstrated a measurable gain over its base model.

The durable artifact is the defect catalogue — 14 defects established with run evidence, 11 dataset-level and 3 harness-level, 8 fixed, 1 fixed upstream but never deployed to the baked image, 5 still open. exp_rpt_nemotron-junit paid reward 1.0 on 100% of trials with exception_info non-null on all 48, and still paid 100% when relaunched with a healthy agent. swesmith-oracle-filtered scored 2,024 of 2,024 trials at exactly 0.0, proven wrong by a single trial where the agent cloned, patched, and ran pytest to 265 passing and was still scored zero because a test file was absent from the checkout. exp_rpt_pr handed out reward 1.0 on trials whose pytest summary reported only skipped 46.7% of the time, against three simultaneous controls at 0.0%. The sharpest entry is crosscodeeval-java, logged in this campaign as a saturated source and in the TaskTrove changelog as a verifier that always returns zero; both readings were right about their own arm and neither measured Java, because the verifier asserts only that /app/solution.py imports and is non-empty. The policy learned exactly that, climbing 0.2100 to 0.9350 in eight steps — a gamed vacuous verifier, not saturation, and no data-quality verdict is available for the source. A companion comment turns each class into a detection signature readable off the metric stream, which is the part most worth reusing: the exception_info distribution at the first banked step diagnoses four arms at once where the reward screen returned four different wrong verdicts; and because trial rewards stay binary regardless of shaping, the only valid test that reward shaping is live is whether avg_raw_reward × rollouts_per_step is an exact integer — the check that would have caught pass_ratio shaping sitting inert for essentially the whole campaign. Two conclusions were retracted rather than smoothed over, including a reported “+488% over base” withdrawn once step-1 reward was shown to be a biased-low estimate of a run's own base in a fully async pipeline, refuted by falling gradient norm and flat-to-rising entropy.

On the curation side, the 43-source verifier gap opened last week in #7418 is now more than half closed: 27 of the 43 have verifier-equipped Harbor task sets in TaskTrove, taking the pool from 124 to 151 registered subdirectories and adding roughly 160k RL-ready tasks. Nineteen got deterministic gates at no per-rollout cost — codeforces’s placeholder that merely checked an answer file existed was joined to open-r1/codeforces for real test cases and multi-language compile/run/compare; manybugs’s text-diff verifier, which paid out for any edit to the fault line, became exact gold-match; quixbugs’s import-only stub tests with zero assertions became real per-algorithm suites; toolscale’s LLM judge was replaced with deterministic JSON-schema validation. Three took LLM judges and five were mixed. The 16 that remain are documented with reasons rather than left dangling: eight need an interactive runtime rather than a response-string gate, three are broken at source, and mind2web’s gold action sequences exist but are non-trivial to operationalize as a rollout verifier. Underneath that, the data-contract work got filed: #7794 asks for maintained profiles for public non-agentic RL sources — RLVR-MATH, DAPO-Math-17k, DeepScaleR, DeepMath-103K, RLVR-IFeval — pinned by revision, split, prompt field, ground-truth field and reward contract, rejecting incompatible columns instead of silently guessing, and #7799 asks for a normalized, versioned post-training pool whose version is an immutable manifest of exact row membership, so that a run never defines its data boundary as whatever existed before it started. The Delphi RL scaling-law thread #6279 gained the provenance table it was missing, mapping its D1/D3 codes to public Hugging Face sources and recording that the prepped parquets under iris/rl-data/ are the 500-prompt probe subsets — the full-scale parquets behind the headline rlvr7500_w1 and dapo17k_w1 numbers are not there and need locating or regenerating before the Snowball 67B-A2B run #7786 reuses the recipe.

The next campaign is already scoped. #7785 inverts the previous design: instead of sweeping datasets against a fixed objective, it fixes three or four known-good v3.17 sources and sweeps the objective. The diagnosis it inherits is that the objective was unanchored throughout — no Kullback–Leibler (KL) loss, no entropy loss, one policy update per step, and proximal policy optimization (PPO) clipping inert at ratio 0.0, measured at exactly 0.0 across all 37 steps of one arm — after which three deeply analyzed long runs collapsed by different dataset-specific routes to the same cause: silent fallback to sparse binary reward on Tulu3 math, terse one-turn length-capped collapse on Unix, entropy runaway from ~0.32 to ~9 on TACO. Dataset selection is by explicit criteria rather than convenience: difficulty in band for the base model, since sources the base already solves ~90% of pin pass@8 at 1.0000 and kill advantage variance, while sources at ~7% raw pass produce uniform-zero groups that get filtered wholesale. The bar is a full 80-step, two-epoch run with no collapse, no infrastructure failure and no data-quality failure — all three, not any one — and the number to beat is the 0.011 sigma that the strongest arm of the last campaign moved its policy with a working reward signal and a demonstrably alive learner. In parallel, marianna13 posted the week's cleanest curation ablation: filtering R2E-gym by the base model's pass@4 down to a learnable band of ~1.6k tasks from 4.5k converges faster but does not raise the asymptote, reaching the same ~45 pass@1 on out-of-distribution SWE-bench by step 60 — 48k rollouts including the filtering pass against 60k for the raw pool, so the case for pre-filtering is compute efficiency rather than capability, with the ~46 ceiling attributed to the base itself, which never solves 58% of SWE-bench tasks. Franziska Weindel also put a weekly data breakout room on the calendar starting 6 August, giving this line of work a standing venue.

What fell outside the epics this week was housekeeping. Seven agent-authored dependency pull requests rolled the isolated external runtimes — evalchemy, harbor, and MarinSkyRL — and the Marin-owned native package floors and universal lock forward: #7676, #7775, #7832, #7846, #7853, #7870, and the still-open #7892. On #7775 Russell Power asked that these roll-forwards auto-merge and spell out the upstream commit subjects they carry; the later runs do, so #7870 and #7892 read as changelogs rather than bare hash bumps. Russell Power also widened the dockerignore in #7757 to exclude sibling worktrees, downloaded toolchains, nested virtualenvs and node_modules, and Cargo target directories at any depth — the Rust workspaces sit at varying depths (lib/finelog/rust, lib/iris/rust) and the old root-anchored patterns were shipping all of it to the Docker daemon as build context.

Three nightshift multi-cleanup passes landed, each bundling four independent per-subproject cleanups aimed at duplication rather than features. #7778 unified two counter-aggregation implementations in Zephyr that disagreed on AVERAGE — the coordinator averaged per-shard means unweighted while the worker weighted by observation count — behind a single reducer, and lifted roughly 390 duplicated lines of block-partition machinery out of SOAP and Kron into a shared module, verified bit-identical across 80 combinations of parameter shape, small-dimension merging, and block size. #7834 bounded ZephyrCoordinator counter memory: it had retained one snapshot per completed shard and re-merged the whole list twice every five seconds, roughly a million CounterEntry objects on a 100k-shard pipeline inside a coordinator budgeted at 1 GB, and snapshots now fold into a running accumulator keyed by stage, name, and aggregation. #7731 replaced the per-handle bootstrap-state triples in the Iris slice handles with one shared tracker, which also gives manual slices back the failure reason they had been dropping. A separate doc-drift sweep #7893 is open against the design docs, correcting a datakit format spec that still lists the partition_id row column removed in #5879, an actor long-running-operations design describing an API that shipped as a new submit() in #3477 rather than replacing the existing blocking call, and every file-and-line-number citation in the folder, now replaced with paths and symbol names so they stop rotting on the next refactor.

The weekly standup issue #7669 collected seven updates, two of which carry cross-cutting questions. Rafal Wojdyla asked whether Marin should keep two or three B200 racks running the current best MoE config purely to stress the infrastructure and learn its failure modes, and whether to engage the JAX team on MFU regardless of how the Megatron comparison comes back. Russell Power flagged two staffing gaps: an ops rotation that still needs setting up, and an owner for training monitors and cluster tooling.

Zachary Nichols opened a durable JSONL telemetry path #7641 — an actor-backed chunk writer over fsspec plus an nvidia-smi wrapper — validated by a CoreWeave S3 smoke that wrote and read back 10,000 records, and put it up for review. He spent the rest of the week in #marinfold on a re-run of an earlier 1.5B protein model that scored 3.081 contacts-v1 validation loss where the original logged 2.704. Rebuilding against the old source recovered 2.704 exactly, locating the gap in a Levanter loss change #7209 rather than in the checkpoint or the data — which, as Tim O'Donnell noted, may mean GPU runs written off as bad were only being scored on an incomparable metric.

Mark Muchane, an incoming Open Athena hire Will Held credited for the Nemotron code reproduction before his start date, filed a tight Iris/fray series: CoreWeave task pages show no resource data because CKS ships no metrics-server, so the pod sampler 404s silently #7845; a task cannot select its node pool or CPU architecture #7857, with the design note #7860; and fray's GPU type table rejects B200 and GB200 outright #7859, with the tutorial device table extended to match #7858.

In #safety-training, Gustavo Sandoval posted a red-teaming MVP for Marin-8B against Olmo-3-7B-Instruct, with the harness first validated to within ±3pp of Olmo's published table. Default refusal diverges by attack type — Marin 96.0 on DoAnythingNow against Olmo's 77.9, 87.3 against 93.9 on HarmBench — but the load-bearing finding is that roughly 100 public fine-tuning examples drive attack success on both models from ~6–16% to ~99% within ten steps, so neither is tamper-resistant. He offered gradient routing and its ICML successor as a pretraining-side control; Huu Nguyen tied it to persona leakage he sees from Stack Exchange text, and Benjamin Feuer asked for both as public issues.

Three people introduced themselves: Ash, building Aquin to reverse-engineer and interpret models, partnered with LAION and KAIST; Laura, at an early-stage VC and exploring infrastructure for the open-source AI ecosystem; and Michael Siu, a USC undergraduate interning at Kimi on eval and harness after working on OpenThoughts-Agent, which intersects both this week's eval wish-list work and the Kimi K3 report the server spent Monday reading. Six others joined the welcome room without posting an introduction. Reading skewed toward other labs' architecture and optimizer disclosures — K3 dropping rotary position embeddings on multi-head latent attention (MLA), GLM-5's echo of the same gate, Tilde's online Kullback–Leibler (KL) Shampoo — read for what Marin should copy.