Marin: Week of July 13th summary

The centerpiece run — the 67.1B-total / 2.01B-active “Grug” MoE on TPU v4-2048, tracked here and logged day-to-day on #6044 — kept training through the week with no course changes. The active resume run crossed from last week's ~2.11T tokens to about 3.74T — roughly 37% of the 10.07T-token horizon — while holding MFU near 17.8% at the slower step time it inherited from the prior week's post-restart topology change. Train loss sat around 1.38 and Paloma macro loss around 2.36 at the current checkpoint, and the run is still going, tracking toward the ambitious ~August 30 mark for the full 10T.

Larry Dial shared a decomposition of where the loss is now coming from: once past 2T tokens, roughly half the loss decrease comes from lowering the learning-rate temperature and half from genuine learning — inferred by comparing the ~500-step initial-cooldown drop against the ~15k steps the full run takes to reach the same LR. On stability, he reported that router balancing looks healthy for the standard 8,192-seqlen pretrain, with only slight imbalance appearing at 64k context with fewer sequences per batch, and QB routing holding up across scale; routing entropy for the live run is tracked in the run's W&B report. Measured against the run's preregistered stage-1 target — 2.269 Paloma macro loss at the 8T mark, taken before the final LR cooldown — the current ~2.36 at ~3.7T, with LR not yet decayed and only ~37% of tokens seen, is consistent with the projection.

Two smaller threads touched the run's code. Benjamin Feuer's #7247 made the attention QKV head split/merge tensor-parallel-safe by passing an explicit output sharding — byte-identical at today's single model-axis config — while documenting that full tensor parallelism for this exact model stays blocked because no model-axis width divides both the 5 KV heads and the 128,256 vocab; the PR was closed rather than merged. Separately, Larry Dial opened #7201 to track candidate architectures and their MFU tradeoffs for the next hero run expected to start in August; the small d=2048 sweep runs that appeared this week feed that effort, not the live 10T run.

A new hero-run epic, #7170, opened this week to post-train the Snowball 67B-A2B 2T intermediate cut — the smaller-token-count sibling that the 10T post-training run #6705 had asked for so the post-training and inference stack could be exercised early. Its input is the 2T cooldown checkpoint from #6811, which finished at 2.2772 Paloma macro-loss (macro_bpb 0.8242) — essentially matching the 2.269 target preregistered for the full run's first stage, from a checkpoint at only ~2.1T tokens. That gives a usable model to post-train now, well ahead of the 10T run landing. Benjamin Feuer owns the run, on H100.

The centerpiece this week is #7321, the first time the whole chain runs end to end on this model — and it is explicitly a pipeline proof-of-concept, not a statement about model quality. The 67B-A2B MoE was fine-tuned in two sequential Levanter SFT jobs on 64×H100 CoreWeave (stage 1 wildchat chat to step-257, then a thinking dataset chained on top to step-630), exported to an HF-BF16 checkpoint, served on 8×H100 through the marin vLLM fork in Grug MoE expert-parallel mode, and evaluated via the evalchemy two-child serve_and_eval orchestrator — with results durably banked to S3 and verified readable after the serve and eval pods were garbage-collected. The SFT side also folded the bespoke Grug launcher into a shared sft_step + ModelSource flow so the model and data are independent inputs to one pipeline, in #7250. Bring-up and the durable-results rewrite landed in #7303 and #7315.

The preliminary, proof-of-concept-only scores came from a single step-630 checkpoint: gsm8k on a 200-sample slice read ~76.5% flexible-extract versus 38.5% strict-match — the wide gap being the expected format artifact of a chat/thinking model that produces correct answers but not always in the strict gsm8k template. MATH500 first read a provisional 16.0%, but reading the retained traces showed that number was dominated by generation truncation rather than math ability: the long-CoT model kept hitting a 2048-token generation cap before boxing an answer. Will Held reran the same checkpoint on the full MATH500 set after fixing a local-chat generation handoff that had silently fallen back to 256 output tokens; with a 32,768-token context and a matching generation budget it scored 271/500 (54.2%). Investigating a related symptom — the model appearing never to emit the Delphi <|start_think|>…<|end_think|> delimiters — first pointed at the stage-2 SFT, but deeper probing corrected that: the model does emit the delimiters (forced-transition logprobs near 99.95%), and the real cause was serving and measurement, namely the HF export baking the wrong chat template plus vLLM's skip_special_tokens=True default stripping the delimiters from returned text. Fixes preserve the training chat template on export and ship a new delphi-v0-think checkpoint revision; the broader takeaway, relevant to any eval or RL-reward path that must parse those delimiters, is to serve with skip_special_tokens=False.

With the intermediate cooldown closed last week at 2.277 Paloma macro-loss, this week was about putting the step-42150 checkpoint — "Snowball," the 2T cut branched off step 39,000 of the 10T run — to work. Romain Yon mirrored the final checkpoint and its .executor_info into a us-east5 v6e-8 inference zone, verifying all 567 objects (539 GB) copied with zero differences; that followed the CoreWeave S3 copy from the prior week, so the model now sits in the regions where downstream jobs run. On Discord, Benjamin Feuer reported that Snowball had landed with SFT / mid-training, chat templating, vLLM serving, and evals all green, and that it is fast to iterate on — an SFT over 700,000 datapoints in under an hour, evals in about 30 minutes — making it a practical starting point ahead of the full run. Reproduce-and-extend documentation lives at #7321, with post-training tracked separately under #7170.

The open question this week was whether to publish the checkpoint publicly. Jeff Hammerbacher asked whether Snowball would go up on Hugging Face so others could try it; Larry Dial agreed it was worth doing but wanted the config standardized against Hugging Face conventions first. Romain Yon noted that a Grug-to-safetensors export already exists in main for vLLM serving, but the serialization contract needs more attention before external use, and that the two supported inference paths today are the Levanter/JAX code and the Marin vLLM fork. Russell Power raised whether vLLM's out-of-tree model-registration hooks could let Snowball run against stock upstream vLLM; Romain Yon thought that likely feasible for GPUs since most of the fork's code is segregated, while flagging it may not beat simply running the fork via uvx for the basic vllm serve case. The publish work is scoped under #7366.

The compute-optimal 11B-A1.5B dry-run grug-d2048-L24-gqa4-500B-r8-nosim-v2 reached its full 500B-token budget on 64×H100 and closed out this week, right on the July 13–14 window projected last week. Hardware utilization held steady through the tail at ~23.7% MFU, and the run settled at a final training loss of 1.63 and a Paloma macro loss of 2.30 (0.806 bpb). As Isaac Hodes framed it on the thread, this was about exercising the GPU stack ahead of Blackwell rather than shipping an artifact, and on that count it did its job: the run scaled cleanly to 64 GPUs and trained to completion without incident.

The thread itself was quiet after the prior week's d512 ablations — which had already confirmed the GPU stack was behaving as expected and traced the datakit-vs-nemotron Paloma gap to data mix rather than any hardware fault — so no new issues surfaced as the big run wound down. The epic remains open #6716 as hardware-and-loss validation continues toward the Blackwell path.

No new numbers landed on the post-training budget this week; last week’s estimate stands, with Benjamin Feuer having priced the pipeline — reinforcement learning over 10 experts, then multi-teacher on-policy distillation (MOPD) into the student — at roughly 0.25–0.9× a small hero’s 10T pretraining in hardware FLOPs (device-FLOP-seconds, so RL’s low ~1% aggregate MFU is counted rather than hidden), and at only ~5–20% of the 120B big run. The single note this week was a cross-link: the new on-policy distillation teacher-serving system-design RFC #7236 builds directly on this cost analysis, sizing the OPD leg as RL K experts distilled into the student (student-rollout-dominated, roughly RL-scale) per #7074 and working through how to actually serve the teacher — top-K versus full-vocab logprobs, and four candidate serving backends.

Architecture selection moved from estimate to measurement this week. Larry Dial opened #7201, the GB200 Architecture TPS Tracker, to log tokens/second and MFU for every candidate shape at the 15–30B-active scale so the final architecture can be interpolated from real hardware runs rather than back-of-envelope numbers (announced on Discord). By week's end it held a measured shape sweep, all MuonH with Fully Sharded Data Parallel (FSDP) plus intra-rack sharding on GB200: top-4-of-64 (129B total / 14.6B active, 22.3% MFU on 8 racks), several top-4-of-128 variants (250B / 14.5B active, ~20% MFU across 4k/8k sequence and sliding-window options), a fine-grained 8-of-256, and a top-5 arm. The read from the data: at equal active parameters 64 experts beat 128 on throughput — half the FSDP weight-gather — and finer granularity costs more. A separate MLA-vs-GQA sweep found Multi-head Latent Attention no faster than the GQA baseline, so GQA stays. The current best-estimate marker sits on the 4-of-64 129B baseline.

Two throughput levers then opened real headroom. An expert-parallel plus FP8 datapoint in the tracker and an MXFP8 grouped-expert-kernel result in #7282 showed roughly +28% tokens/second over a matched bf16 control — enough that FP8 is now the leading path to pull MFU up. On Friday Dial pushed the first hidden_dim-6144 probes (~360B total / 22.6B active, 48 layers), and the working view is that something of that general size, with MFU lifted via FP8 and better comms/matmul overlap, will make the most sense; the exact hidden dim, expert count, token budget (20T referenced), and the preregistered loss target remain open as the sweep fills in. In parallel the datakit data-quality prerequisites advanced: clustered-store determinism was confirmed and then enforced (#6798 → PR #7177, which now raises on connected-components non-convergence rather than shipping an irreproducible survivor set), and the whole-source-wipeout dedup bug got a per-source canonical-scope fix (#6854 → PR #7197) — both feeding the July-hero datakit release #6037 that will supply the run's tokens.

This commitment gates the end-of-month ~120B-A8B run: B200 model FLOPs utilization (MFU) needs to clear 20% before it kicks off. That bar is now met at rack scale — a single GB200 NVL72 rack reached about 23.3% MFU, comfortably above the 20% target, with the kernel and perf engineering behind it detailed under B200 training MFU & perf #6710 #7012. The single-node figure on 8×B200 remains below 20% at some model widths, so the gain is coming from the rack-scale configuration the run will actually use rather than from the per-node number, and work continues along the fp8 and pipeline-parallel paths in #6710 toward the further 25% target.

The headline artifact this week is a full jusText extraction of the science Focus Crawl. Will Held ran Michael Ryan's jusText fork with the English stoplist across all 4,573 CC-SUPPLEMENTAL-2026-22 WARCs of the Common Crawl (CC), yielding 36,327,068 extracted documents in #7397. The pipeline reads each WARC's CDX sidecar and downloads only coalesced byte ranges for indexed HTML responses, sidestepping most PDF transfer, then decodes and sanitizes the HTML before the fork's sklearn classifier runs, with 66 Genoa actors each fanning out to eight isolated single-CPU subprocesses for 528 concurrent shards. The normalized artifact is 4,573 Parquet files totaling roughly 89.4 GB, and a full pass through marin-community/marin-tokenizer measured 49,702,569,456 tokens. #7382 registers that artifact as a Datakit source by materializing a pointer to the existing GCS object rather than re-running the download, giving the WET-derived and jusText-derived views of the same crawl a side-by-side comparison for the quality ablations tracked in #6570. One pathological page had jusText's line-level dedup, quadratic comparisons over paragraphs, running for eight hours, which prompted a follow-up patch switching to Locality-Sensitive Hashing (LSH) for anything past 300 paragraphs.

Russell Power benchmarked the pooled fast-transformer quality classifier on v6e TPU and found it now cheap enough to score at pretraining scale, in #7187 and #7191. At roughly 55.5K FLOPs per token the forward pass uses only about 1% of a v6e-4's matrix units, so throughput is host-bound, dominated by tokenization and the token-remap loop rather than the chips. A warm shard scores at about 18,000 documents/s per v6e-4 and scales linearly across VMs, putting a 10T-token corpus of roughly 11.1B documents at about 2.7 hours on 64 v6e-4 or 0.7 hours on 256, on preemptible TPU, about 3x fasttext's throughput at 0.69 versus 0.44 quality. Swapping the deployed SentencePiece tokenizer for a tiktoken BPE lifts that to about 26,600 documents/s at matched quality, and the same accelerator-agnostic harness runs unchanged on CoreWeave 8xH100 at about 22,700 documents/s. Because the forward is so nearly free, the classifier could grow 10-20x in FLOPs per token before the device becomes the limit. Left open is how to bucket the scorer's calibrated [0,1] output for mixing: #7124 weighs the current absolute-threshold scheme, whose buckets carry a consistent cross-source quality level but leave thin tails of 1.9% at q0 and 4.6% at q4 on the 100B run, against quantile bucketing, which guarantees usable bucket sizes but erases that absolute, comparable signal.

A wave of new sources landed. #7188 registered NVIDIA's Nemotron Pretraining Specialized v1.2 and Legal v1 as 15 Datakit sources totaling 46,419,444,831 tokens, holding back the two Legal subsets whose released text still carries unresolved <CLAUSE> and <DOCUMENT> placeholders; rehydrating those from the auxiliary source files is now tracked in #7193. #7192 added Nemotron Code v2 at 120.25B tokens, #7363 registered trillionlabs' TheBioCollection as free-text and instruction streams of 33.19B and 18.12B tokens, and #5305 brought in TeraflopAI's SEC-EDGAR, 43.7B tokens across roughly 8M filings, behind a DuckDB rewrite step that strips per-page statistics to dodge a PyArrow bug on page headers over 8 MiB, filed as #5334 against apache/arrow#46404; #7357 then fixed two bugs that had been failing the SEC-EDGAR download outright. On the dedup side, Rafal Wojdyla's agent made fuzzy dedup reproducible across cluster sizes in #7177, tracing the roughly 0.3 ppm token drift from #6798 to LSH-bucket subgraphs whose star-versus-chain topology depended on intra-bucket arrival order, which the shuffle varied with executor count. The same agent opened #7197 to stop the whole-source wipeouts of #6854 by retaining a per-source canonical, since a single global canonical per component had erased overlapping collections like Gutenberg, Library of Congress, and uk_hansard and taken massive_function_calling to exactly 100% removed. #7150 pared experiments/datakit/ down to the reference normalize, tokenize, quality, domain, dedup, decontaminate, store pipeline with a self-contained HTML report per stage.

Looking past the July release, Jeff Hammerbacher filed #7162, a proposal to turn the one-shot focus-crawl ablation of #6570 into a standing per-crawl loop, with each new Common Crawl release flowing through WET ingest, a cheap decontamination annotation pass, and a quality/BPB scorecard into a published report, the Common Crawl Foundation having asked directly about wiring model training into their crawl feedback loop. The proposal frames itself as assembly rather than construction, since the ingestion, decon, and cheap-ladder pieces already exist and run separately; it remains part infra and part open research, not yet landed. Meanwhile the running accounting of the raw pool crossed 23T tokens pre-dedup, with the largest recent additions being roughly 2T tokens each of new code and PDF content alongside AI2's Dolma 3.5 and Nemotron's open-data updates.

This epic's definition of done — serve the full-size GrugMoE checkpoint on both TPUs and GPUs — largely came true this week. The centerpiece is Snowball #7227: a first-class Levanter LmConfig/LmHeadModel that snapshots the June 67B-A2B production recipe and loads its existing Hugging Face BF16 export, so the checkpoint is discovered, loaded, and scored through the standard Levanter stack rather than a bespoke path. It pins one validated recipe (QB-routed 256/4 MoE, GatedNorm, per-head-gated attention, half-RoPE short/sliding-window layers, NoPE full-causal long layers) and rejects off-recipe checkpoints outright rather than silently reinterpreting them; the forward reuses the shared levanter.grug primitives, so it is bit-exact against the training path on CPU. Fitting the real 67B on an 8×H100 slice took a memory fix: the 26 blocks had run in a Python for-loop, which kept every layer's expert buffers live at once and OOMed at 50.94 GiB, so they were folded into a single scan over the stacked blocks. It then matched the golden next-token distribution on 8×H100 — greedy token exact, worst per-token probability error 0.0008 against a 0.008 bound — through both the native compact-mesh load and marin-serve's load_model, closing the serving-investigation issue #7219. Alongside it, #7178 added a Levanter serving backend to marin-serve (--backend {vllm,levanter}), so the same one-liner can serve a model through Levanter's engine or vLLM on the same slice and OpenAI API, closing #7172. Romain Yon announced a v0 of Grug 67B-A2B inference on GPUs landing in main, rough edges and all.

Most of the week's volume was the GPU serving path itself, which had to be built around a resolution problem: stock PyPI vLLM pins torch==2.7.0 against Marin's CUDA-13 torch==2.11, so it cannot live in the workspace lock. The fix, taken on Russell Power's suggestion, provisions CUDA vLLM per-job in a throwaway uvx tool env so its wheel tree never enters uv.lock #7133, closing #7111; the TPU path was then mirrored onto the same isolated-uvx pattern so marin-serve --tpu can run outside a checkout #7107, closing #7106. Stock and forked CUDA vLLM were then unified behind one VllmType launcher #7342 (GPU and TPU now pin the same fork revision), serve_and_eval's GPU fork branch was completed so the 67B export can be served and evaluated #7303, and a boot failure was fixed by adding ffmpeg to the task image, which vLLM 0.25's torchcodec dlopens for libavutil #7141. A cluster of correctness and observability work rounded it out: the export was validated from CoreWeave object storage under data-parallel-8 / expert-parallel-8 (EP) using the Run:ai model streamer to skip a 125 GiB local copy #7137 #7136, the parity contract was widened from one four-token string to 64 real evaluation prompts up to 32K spanning the sliding-window boundary #7283 #7314, native vLLM subprocess logs and Prometheus /metrics were teed into the job log and finelog #7329 #7349 #7323, the Iris proxy timeout was raised from 30s and made per-endpoint so long reasoning generations stop 504-ing #7180 #7173, an undeclared GitPython import that broke marin-serve at load was fixed #7371 #7370, the quick-serve dashboard was rebuilt as a self-contained Vue chat UI #7388, and the --gdn-prefill-backend triton recipe for serving Qwen gated-delta-net models without nvcc was documented #7386 #7373.

The two backends were then measured head to head on fresh 8×H100 nodes #7354. In this cold test path vLLM cut end-to-end case time 1.96× (273 vs 535 s) and prompt scoring 8.4× (15.85 vs 133 s, Levanter's first batch at each shape paying JIT compilation), while using about 10 GiB more high-bandwidth memory (HBM) per GPU — a sampled peak at 92.5% of capacity, the clearest resource risk. Startup, not scoring, dominated both. Correctness held: all 64 representative cases passed for both backends, and across five additional cold vLLM runs, though a separate rank-consistency diagnostic was not reproducible. What it did not clear is strict bit-reproducibility: Romain Yon noted that Levanter's paged key-value (KV) cache hasn't been rolled out yet, so the gap is probably expected. That feeds the still-open logprob investigation #7183: the step-42150 checkpoint has three top-25 results (an H100 golden and two Levanter TPU references) that all agree on the top prediction and the full top-25 token set with no token moving more than two ranks, but differ slightly in logprobs — and because the two TPU paths themselves differ, the evidence does not isolate the drift to the GPU-vs-TPU seam. A companion draft adds a compact Levanter TPU regression on v6e-8 that asserts exact top-25 parity with zero MoE capacity overflow #7182.

On the TPU stack, evals surfaced a burst of prompt-logprob failures that were root-caused and fixed. #7401 bumps the tpu-inference fork to fix bursty whole-batch KeyError 500s on echo + prompt_logprobs (two independent bugs both dropping the true prompt token: a jnp.roll target gather that wraps across a chunked-prefill boundary, and a per-request accumulator wiped on mid-prefill re-add) — before the fix an isolated hammer hit 159 of 400 requests, after it a full MMLU of 56,168 requests ran clean in 3m12s. Two adjacent TPU issues were logged: an EngineDeadError tracing to prompt_logprobs being unsupported for the genuinely multimodal Qwen3.5-9B #7390, and flaky first-placement serve jobs where libtpu's fixed host port 8431 collides with a co-tenant, mitigated with a one-shot reschedule #7392. The maintenance burden behind all this is being cut down two ways: consolidating the vLLM forks so only one patch set has to be maintained #7097, where Romain Yon audited the mlfoundations/vllm delta and Benjamin Feuer gave per-feature verdicts from the CoreWeave reinforcement-learning (RL) paths, landing a qualified port in marin-community/vllm; and a proposal to publish prebuilt TPU vLLM fork wheels so every cold serve/eval/RL job installs a wheel instead of rebuilding from source #7143. The RL track waits on this: Benjamin Feuer flagged he can't start RL until the vLLM branch and fork are synced, with a GrugMoE PyTorch modeling class the remaining gap to loading the checkpoint into MarinSkyRL. Looking further out, a new issue proposes quantization-aware training (QAT) during SFT to ship a natively 4-bit-deployable end-of-year checkpoint of Snowball for ~2× faster, ~4× smaller inference at near-BF16 quality #7404.

The pinned lm-eval-harness fork finally broke. #7267 reported that transformers>=5 removed AutoModelForVision2Seq, which the fork reads at import, so every marin lm-eval evaluator crashed before running a single task — and the same fork sat in Levanter's in-loop eval path. The minimal stop-the-bleed landed first, teaching Levanter's import guard to catch the resulting AttributeError and degrade to an lm-eval-unavailable sentinel in #7278, closing #7277. The real move followed: #7281 (+920/-2365) dropped marin's lm-eval evaluators entirely and routed downstream evals through the evalchemy fork against a served OpenAI-compatible URL — serve the model once with marin-serve, evaluate the group's tasks against that endpoint inside the :evalchemy-tpu container, then tear the server down, the two jobs decoupled by the URL. That decoupling also closed the long-standing #4827. Russell Power framed the switch as “effectively moving all evals towards Benjamin Feuer's approach,” and flagged parallel work on the evalchemy fork to shed the container dependency; the still-open #7270 tracks making that client uvx-installable so the OpenAI-client path drops the image entirely.

Around the pivot the eval surface was consolidated into composable typed artifacts. #7253 (+671/-1683) replaced the default_* eval bundles with one ArtifactStep per EvalGroup, each addressed by evaluation/{backend}/{model}/{group_id}, and in doing so fixed a latent correctness bug: the old default_base_eval issued four Levanter steps that shared a single name, and because StepRunner dedups the graph by output path, three of its four multiple-choice evals — MMLU 0-shot, 5-shot, and MMLU-Pro — had been silently dropped and never run. A sibling collision fell next in #7284, where evalchemy metrics keyed by the bare lm-eval task name let hellaswag's 0- and 10-shot result files overwrite each other; keying each file by its unique upload directory fixed it. The purge that motivated all this closed as #7240 after an inventory found nothing import-broken but plenty of orphaned one-off scripts, and Benjamin Feuer's #7233 added the standalone evalchemy-on-TPU launcher those experiments now sit behind. Test hygiene came alongside in #7259 and #7258.

Two failure modes that let a scored-nothing run masquerade as success were also closed. #7391, filed by Russell Power after an eval whose endpoint died mid-task still wrote an empty results: {} and exited 0, was defended in #7393 with a has_scored_results guard and test that fails any task whose results dict is empty; the deeper fix — making the fork itself exit nonzero — remains upstream in the evalchemy fork. Benjamin Feuer's #7315 closed the mirror-image leak where a succeeded eval wrote its numbers only to a pod-local /tmp path that vanished on teardown, routing artifacts to durable object storage and raising if no results land. Underneath these point fixes, Russell Power raised the question of how the parallel eval efforts avoid stepping on each other in #7345 and answered it with #7405, a full Marin evaluation system — model and task registries, a group launcher, durable per-run records, and an evaldash dashboard — demonstrated end-to-end with snowball plus five open-weight models filling a six-by-ten model-by-eval grid, every cell green, snowball scoring MMLU 0.692 and GSM8K 0.737. Beside it, Will Held kept a direct brokered lm-eval-harness path in #7295, deliberately alongside the evalchemy route so new held-out benchmarks aren't coupled to it, and moved Harbor sandboxes onto Iris as CPU-only, scheduler-visible jobs under gVisor in #7288.

The rest was breadth, much of it still at the design stage. Two harness-robustness proposals from Will Held remain open: #7169 scopes a static first step running MMLU, GSM8K, and HumanEval through one minimal harness contract to measure the harness delta, and #7171 the generic lm-eval-to-Harbor adapter that would make any harness benchmark runnable in-agent — both motivated by the observation that Qwen 3 conserves most of its accuracy in-harness while Qwen 2.5 and Llama 3 do not. The #7090 wish-list gained sub-issues for NUPA direct numeracy #7297, the base-model single-tool-call MCP-Atlas-Easy eval #7160, and StructEval conversion tasks #7161. The perplexity-surrogate design issues stayed open across instruction-following, factuality/abstention, professional-document, and scientific-reasoning coverage #5824, #5826, #5827, #5829, as did strong-model coverage-gap sampling #5006 and the tagged MRCR context-perplexity eval #7203, whose d512 validation run cut final-turn perplexity from 21.09 to 10.40 with context. Rohith Kuditipudi caught that Delphi's published HumanEval was labeled 10-shot but effectively ran 0-shot because lm-eval fixes num_fewshot to zero and rejects the override #7229, and Ruler was wired up and closed #2064. In the evals channel, Percy Liang situated all of this against the basket of 200-plus perplexity evals and hint-mediated forecasting monitored before RL and asked whether the Harbor index is worth curating, and Benjamin Feuer circulated an RFC to give evalchemy a retry mechanism that labels infrastructure errors apart from model errors.

The week's largest theme was moving cluster provisioning off hand-run runbooks and onto infrastructure-as-code. Will Moss's design doc #7151 proposes Pulumi (Python) as Marin's engine for the static cluster substrate — the CoreWeave cluster, reserved NodePools, Kueue/Traefik add-ons, namespace + role-based access control (RBAC), and object storage — with Iris keeping ownership of dynamic per-job dispatch. It merged as part of the first slice #7153, a Pulumi program that provisions the iris namespace, controller RBAC, reserved NodePools, the Kueue gang-admission objects, and the GCP federation-egress IPs directly from a cluster's config. The umbrella issue #7213 tracks folding the rest of the ~9-step CoreWeave rollout into a single pulumi up, leaving only two irreducibly manual steps — installing the console kubeconfig and adding the DNS CNAME, the latter now on Cloudflare so it can be API-managed. Russell Power gated the Pulumi provider SDKs behind a marin-iac[deploy] extra #7358 so the workspace-wide uv sync stops dragging the whole Pulumi stack into unrelated jobs like model training. Will Moss then got Pulumi cleanly importing the first live cluster (cw-us-west-04a, formerly ci-coreweave) with state stored in a GCS backend #7356, the pattern the rest of the CoreWeave clusters will follow; Russell Power extended the same approach to always-on Iris services in #7394, declaring them as Pulumi resources with ducky ported as the first.

A GB200 cluster came online in parallel: cw-us-east-08a was configured with four NVL72 racks — 72 nodes, 288 Blackwell GPUs — plus its egress IPs and federation wiring. On the access side, Rafal Wojdyla's agent admitted a Princeton submitter (wg0420@princeton.edu) to the peer's allowlist #7260 so their jobs federate onto the GB200 cluster; the manual IAP (Identity-Aware Proxy) grant that also had to be issued by hand prompted a follow-up to fold user grants into the IaC workflow #7262. Russell Power fixed a federation resubmit wedge #7184: resubmitting a job id whose prior handoff had already finished on the peer left the parent stuck forever in "awaiting first status report" (the peer answered the fresh delivery as an idempotent replay of the old terminal job and emitted no changelog row), and a routed cancel could not unstick it either. That was patched with a per-handoff nonce, with the general fix — a client-stamped submission uuid on every launch — tracked in #7186; a request to let CoreWeave clusters export a region for constraint satisfaction was closed out #7286. Onboarding friction surfaced too: an MIT submitter found IRIS authorization is tied to a Google-linked identity, and their outlook.com account could not be admitted. Late in the week the new controller wobbled — the cw-us-east-08a controller went unresponsive and k8s briefly broke, Will Held filed #7396, and it was fixed the same day.

The GB200 hardware also produced a genuinely silent failure. Larry Dial's 512-GPU (eight-rack GB200) grug MoE datakit run froze at step 145 of 500 #7344: all 128 ranks logged their last line at the same instant and went silent, with no traceback, no OOM, and no NCCL (NVIDIA Collective Communications Library), CUDA, or timeout message — throughput was healthy at 22.96% MFU right up to the freeze. The reason no forensics existed turned out to be structural. The k8s backend has no per-node worker daemon, so unlike GCE/TPU jobs it never captured the periodic thread dumps that leave a stack-trace trail; Russell Power confirmed finelog held 3.29M periodic captures, none from any k8s pod, and added a PeriodicProfiler to the k8s task provider #7346. The reason the hang itself was silent: XLA's collective-termination watchdog defaults to disabled, so a cross-node NCCL stall wedges the whole mesh with no error and Iris's max_retries_failure never fires. #7350 arms it for GPU runs, defaulting the termination timeout to 600s and setting NCCL_DEBUG=WARN so the abort is actually reported — enough, at least, to learn which collective a future hang is stuck in. A related non-GB200 hang was also closed: a TPU canary that trained to completion but never reached a terminal state, wedged in threading._shutdown by a non-daemon telltale-server thread #7302, fixed by making that thread a daemon #7304.

Two more efforts aimed at visibility and utilization. Responding to Benjamin Feuer's request for a weekly, per-user (and ideally per-issue) rollup of preemptible compute so "where did the compute go this week" stops being answered anecdotally #7353, a design plus a working scripts/ops/compute_report.py landed in #7387. The load-bearing finding, from querying live finelog rather than the schema, is that placement — capacity type, generation, slice chips, and zone — is already encoded durably in the iris.task worker_id string, so TPU chip-hours and the preemptible/reserved split need no new controller emit. Separately, Will Held landed Harbor sandboxes as Iris jobs #7288: each task sandbox now runs as its own CPU-only Iris job, scheduler-visible and bin-packed onto spare host CPUs across the fleet (including TPU hosts), rather than as an unaccounted worker-local docker container. A follow-on #7389 drops the click-context workaround the harbor backend needed and routes public-registry pulls through the existing pull-through mirrors.

The week's headline was a general-purpose chat supervised-fine-tuning (SFT) launcher. Benjamin Feuer's #7231 added experiments/sft, expressing a full SFT run as a parameterized sft_step(spec) -> ArtifactStep[LevanterCheckpoint]: native dataset transform from ShareGPT/OpenAI into canonical messages, then chat tokenize and pack with a pluggable chat template and completions-only masking, then Levanter SFT, then Hugging Face export. The chat template, dataset mixture, model and initialization, sequence length, packing, and cluster target are all fields of an SFTSpec, so nothing is hardcoded to a model family; a Delphi 1e22 config was the first worked example. Days later Benjamin Feuer's #7250 generalized that into a shared sft_step plus a ModelSource protocol, folding the previously bespoke Grug 67B-A2B two-stage chat SFT onto the same seam. ModelSource owns the architecture, tokenizer, weight source, and backend: HfModel initializes from a Hugging Face checkpoint, LevanterCheckpointModel from a native checkpoint or a chained upstream step, and GrugModel plugs the native ring expert-parallel run_grug backend into the same step. The two Grug stages just swap datasets and chain stage one's checkpoint into stage two, where stage one establishes the chat format from the step-42150 base on WildChat and stage two builds a reasoning region from Llama-Nemotron science traces.

Underneath the launcher, Russell Power carried the Levanter “SFT with confidence” cleanup tracked in #7045 a long way forward. #7214 let training resolve num_train_epochs to a concrete step count from the cache's token total, fixing the packing over-training trap where a hand-computed 1-epoch count ran a 1e22 SFT for roughly 6.5 epochs; it closes #7130. #7223 deleted the legacy supervised dataset format in favor of chat, of which single-turn chat is a superset, closing Will Held's #7216, and #7238 removed three vendored chat templates no experiment used. #7209 fixed a crash that had blocked unpacked chat SFT and dropped the never-implemented pack="pad" mode, also plugging a loss leak that trained padded positions to predict padding, and closes #7086. #7210 aligned Levanter's chat renderer to Hugging Face byte-for-byte and added parity tests, surfacing two divergences that reach the token ids inference actually sees. A fail-fast guard in #7239 now raises when a chat template ships no {% generation %} block rather than silently producing an all-zero assistant mask, closing #7237. The confidence has a benchmark behind it: Benjamin Feuer closed #7198 reporting a Levanter/JAX Delphi SFT reproduced at 1e22 on 4 A100s with correctness and model-FLOPs-utilization parity against LLaMA-Factory, MATH-500 at 48.0 against the baseline's 44.2.

Two staging steps made the Delphi recipe reproducible from a clean prefix. #7280 expressed the prepared Delphi checkpoint as an ArtifactStep that renames the reserved-special-token slots to single ids and reinitializes the corresponding embedding rows in one safetensors shard while copying every other file through unchanged, closing #7243; #7337 made the Hugging Face to Levanter conversion a cacheable, weights-only hf_to_levanter step so a large base converts once instead of on every launch, closing #7285. Benjamin Feuer's #7406 then closed an export footgun: the Grug 67B-A2B s2-thinking export had shipped the tokenizer's default template rather than the Delphi think template it trained on, so chat completions never cued the model's <|start_think|>/<|end_think|> protocol even though forced-transition logprobs showed the model emits both delimiters. That step-630 checkpoint was the week's concrete run: in #midtraining Will Held traced most of the MATH-500 cutoffs to reasoning traces not reaching a conclusion, and separately observed the model struggling to emit EOS and looping, wondering aloud whether aggressive pretraining packing had baked that in for SFT to overcome; Benjamin Feuer read it as tunable in post rather than a cause for alarm.

The follow-ups are scoped rather than blocking. #7244 would have the launcher emit a chat-format tokenized cache so the epoch-to-step resolution from #7214 can replace the launcher's hand-computed step count; #7417 tracks two GPU-path bugs surfaced running a 25B dense SFT on a single CoreWeave H100 node, an unsized host that OOMs during initialization and tokenizer staging that rejects cloud-storage paths, neither previously exercised because the recipe had only run on TPU; and the masking-fragility concern in #7087 remains open under #7045. Worth noting against the epic's name: the data-curation work itself was quiet this week, with post-training data curation living on the OpenThoughts side, so nearly all activity here was the SFT launcher and Levanter tooling arc.

A week of proposals and one concrete pool update rather than a landed training-mix decision. On sourcing, Will Held folded two fresh open releases — AI2's Dolma 3.5 and Nemotron's updated open data for Ultra — into Marin's raw pool, which now stands at 23T tokens pre-dedupe, the largest additions being ~2T tokens of code and ~2T of PDF content, all tracked live on the token-count viewer. That is a growth of the candidate pool the July-hero mix draws from, not a change to a committed mix. The week's new standing proposal is Jeff Hammerbacher's #7162: turn the one-shot focus-crawl ablation into a recurring per-crawl Marin × Common Crawl (CC) loop, where each new CC release flows through WET ingest, decontamination annotation, and a quality / bits-per-byte scorecard, framed explicitly as assembly of proven components rather than new construction and leaning on the mixing-via-embeddings surrogate #6969 to price each crawl from a single embedding pass with no proxy training. The Common Crawl Foundation has asked directly whether model training can join their crawl feedback loop; the issue is open for scoping — compute lane, cadence, ownership — and drew no decision this week. The pricing primitive it depends on kept hardening in parallel: Rafal Wojdyla's preregistered holdout test of the content surrogate on the Grug-MoE mix swarm passed, though most of the predictive signal traced to histogram shape rather than cross-bucket semantics — detailed under the Data-selection diagnostics epic #6712, where #7067 lives.

Michael Ryan (Stanford CRFM) posted his weekly sync on the small-model raw-web-to-tokens recipe #2351, continuing last week's finding that Marin's high-quality curation is not Pareto-dominant. He finalized a register-stratified dev set — 1,934 CommonCrawl pages hand-labeled keep / weak-keep / weak-drop / drop across 25 registers, ~75% keep-side — plus a held-separate uniform random-internet slice where only ~30 of 250 pages are keep-worthy, a reminder of how little raw crawl survives. Scored against that human gold, his LLM extraction spec beats every production filter by a wide margin (multi-spec 95.5% accuracy / 85.8 macro-F1 versus FineWeb-CC 71.3, high_quality 65.7, DCLM 57.4, Nemotron 33.5), and a two-stage “filter then chunked-extract” multi-spec variant edges the single-prompt version at the cost of becoming a small system rather than one model-plus-prompt. The load-bearing new result is context length: 54% of the high-quality kept documents exceed the extractor's context window (40–80% on arxiv and fiction), forcing chunking. MMLU stays pre-emergence at this FLOP scale and is not yet reliable signal. He raised open questions for the team — which target metrics best align this work with Marin's efforts, and how small the WARC sample can go, since 100–500 random WARCs read consistently but the DCLM-CORE crossover may need ~1k. Alongside, Huu Nguyen offered collaboration from the Mixture Vitae project and Will Held sent a locality-sensitive-hashing dedup fix for a justext blowup in the extraction pipeline. Two older data issues closed quietly: the CommonPile full-commit-history dataset proposal #3332, and the never-run PDF-dataset comparison #2067, auto-closed after stale cycles.

The week's headline is a throughput answer for the fast-transformer quality classifier, which gives better quality than fasttext but had been too slow to run at pretraining scale on CPU. #7187, opened by Russell Power, set the target: benchmark the pooled classifier on v6e and find out how fast it can score a corpus. The finding that reframes the exercise is that the forward pass is essentially free — the deployed config is ~55.5K FLOPs/token, so a v6e-4's compute ceiling is ~66 G tokens/s and a forward-only microbenchmark tops out at ~670 M window-tokens/s using ~1% of the matrix units. Throughput is therefore host-bound, gated by tokenization and token packing rather than the chips. #7191 builds the benchmark harness around that fact — parallel Arrow row-group reads, a fork process pool that tokenizes off the GIL, and a stager thread that overlaps host-to-device transfer with the forward — and lands the number: a warm 190K-doc shard scores in ~10.5s, or ~18,000 docs/s per v6e-4, scaling linearly across VMs. A 10T-token corpus, roughly 11.1B docs, is ~2.7 h on 64 v6e-4 and ~0.7 h on 256, on preemptible TPU, about 3× fasttext's throughput at 0.69 vs 0.44 quality. The harness is accelerator- and cluster-agnostic: the same command runs on CoreWeave 8×H100 at ~22,700 docs/s, and swapping the deployed SentencePiece tokenizer for a tiktoken BPE, o200k_base, lifts throughput to ~26,600 docs/s at matched classifier quality. With the forward at ~1% of the matrix units the model can grow ~10–20× in FLOPs/token before the chips become the limit, so quality scoring is now cheap enough to run over the full corpus.

On the mixture side, the bucket-indexed-evidence problem from #7067 — that data-mixing surrogates learn a mapping from a bucket's name to loss with no representation of what the bucket contains, so any re-partition strands the evidence — moved from problem statement to a validated result. Working in Rafal Wojdyla's agent session, the content-embedding surrogate was put through a pre-registered holdout test on the grug-MoE mixture swarm: 840 MoE runs over 168 buckets, mapped back to their underlying datakit partitions (store_8ac06c74, 40 lexical clusters × 5 quality tiers, ~10.37T tokens), with 800 runs for training and 40 quarantined for a one-shot test. The surrogate passed its pre-registered gates — test Spearman 0.72, beating the bucket-indexed RegMix-style incumbent, a weights-LGBM at 0.685 — with the honest caveat that on this swarm most of the predictive signal is carried by mixture histogram shape rather than content semantics. A follow-on Codex-collaborative validation campaign corrected the per-run budget to ~100B tokens, quantified where the surrogate is interpolation-dependent, and settled on recommending mixtures at cluster rather than top-1 granularity. Separately, Russell Power closed the embeddings-pipeline scoping issue #4488.

Larry Dial opened a GB200 Architecture TPS (tokens per second) Tracker #7201 to collect measured throughput for every candidate architecture at the ~15B-active scale, so the team can interpolate between shapes to select the hero-run model rather than guess it — each comment pins one architecture with its branch, W&B link, batch, MFU and tok/s, updated as tuning improves. He announced it in the moe channel as the place to weigh the MFU trade-offs across expert count (64/128/256), top-k and granularity (4-of-64 through 8-of-256), MLA versus GQA, multi-token prediction, sliding-window, remat and sharding. The datapoints so far center on MuonH · d5120 · 48 layers at ~14.5B active and 250–360B total, landing ~19–22% MFU on the GB200 2.5-PFLOP bf16 convention. By July 19 Larry Dial pushed the first hidden-dim 6144 point — 359.6B total / 22.6B active — and wrote the run will likely be “something of this general size,” with MFU pulled up afterward via FP8 and better comms/matmul overlap. He also ran a one-rack MLA (Multi-head Latent Attention) versus GQA sweep on the same d5120/128-expert model, with MFU deliberately omitted because the standard model-FLOP formula does not account for MLA's compressed latents, and filed #7407, a bug where the heterogeneous-KV nested-scan forward — the path that would give global-attention layers fewer KV heads — fails to load its CUDA kernel at d6144 on 64 GPUs.

The tracker's sharding question got a sharp answer at high expert count. Rafal Wojdyla's agent measured the 512-expert top-8 config on one 64-GPU GB200 rack #7332 and found that expert parallelism (EP) wins decisively and monotonically: sweeping the Fully Sharded Data Parallel (FSDP)↔EP spectrum, MFU rose from 6.05% at EP1 (the non-EP local backend) to 14.39% at pure EP64, a 2.4× jump. That is the inverse of the earlier E64 top-4 result, where a replica/FSDP layout hid the gradient all-reduce and won; at 512 experts the fine 8-experts-per-rank sharding also let pure EP64 fit and run where the E64 thread had been blocked by out-of-memory and a CUBIN-load bug. With unscaled fp8 on the wire the config reached ~18.2% MFU / ~417k tok/s. Two caveats travel with it: this is single-rack only — multi-rack E512 is untested — and the fp8 is throughput-measured, not numerically validated.

On precision, the committed bet advanced sharply. Matt Wittmann's MXFP8 block-scaled grouped-GEMM work #7282 now trains end-to-end and beats both bf16 and per-tensor fp8: in an apples-to-apples run at 64 GPUs on d5120/48L/128-expert top-5, MXFP8 reached ~27.2% GB200-MFU against its own bf16 control's ~21.3% — +27.8% tokens/s — and +34.8% over the tracker's then-best config, with loss tracking bf16 to ~4e-4 per step. Cross-posted into #7201, it is the largest single MFU lever currently on the table. A first production-shape run initially exhausted memory (an 851 GiB step-0 arena from MoE dispatch/combine buffers across the 48 unrolled layers); Matt Wittmann root-caused it and is porting a scan-over-blocks form, and separately traced the long-standing CuTe compile flakiness to a dependency-packaging conflict rather than node heterogeneity. The end-of-year precision ideas past MXFP8 — native FP4 (4-bit floating point) pretraining #7403 under umbrella #7402, and quantization-aware training (QAT) during SFT — remain proposals, not results; MXFP8 is the shipped path. Separately, Matt Wittmann's Hopper FP8 loss-curve validation #7298 passed its gate, ending +0.0039 in matched-step loss against bf16 over a full 24,000-step run on 64× H100 at ~5.5% higher throughput, and Russell Power merged DenseMixer-style dense router gradients into Levanter's Qwen3-MoE and Mixtral #7212 with a grug follow-up open at #7211.

The Agent-MoE ladder ran its usual gauntlet against the fixed July baseline. Kaiyue Wen's four-stream Identity Hyper-Connections #7409 stayed blocked at Gate 1: matched profiles showed an 8.15% throughput loss at d512 and 10.49% at d768, just past the hard 8% bar, so the v1 variant is terminal and she is stripping observability overhead before a retry. Over-Encoding's n-gram embedding tables #7368 first finished tied-or-worse, but a follow-on Over-Embedding variant cut loss by ~0.025 (~10%) over the baseline at under 5% throughput overhead — a promising single result she flagged in the architecture channel. The SiTU activation #7255 was tabled as a negative and an Inkling relative-position variant #7208 was stopped on an unfused-kernel bottleneck, while @Tschoelll opened a new grug value-embedding PR #7230 setting up a three-arm ablation of the modded-nanogpt value-embedding table. A DeepSeek-style MLA ablation cluster continues under parent #7374, and the architecture channel opened a broader linear-attention question — how to fold Kimi Delta Attention or Mamba-style layers into the GB200 models #7291 — with the caveat that KDA's exponentials are too expensive on GB200 versus GB300. David Hall retired a batch of superseded 10T-scale MoE scaling issues — #4019, #4029, #4038, #4039 and #4043.

Last week B200 model FLOPs utilization (MFU) closed on the 20% bar at 17.8% single-node; this week the bar was cleared on GB200 at rack scale. On the production Fully Sharded Data Parallel (FSDP) driver, Larry Dial traced the recurring 64-GPU step-8 out-of-memory failures not to Muon, Distributed Data Parallel (DDP), or the replica mesh but to a dead allocator flag: TF_GPU_ALLOCATOR is a TensorFlow variable that JAX's PJRT client never reads, so every run had silently used the fragmenting BFC allocator. Switching to XLA_PYTHON_CLIENT_ALLOCATOR=cuda_async landed 23.32% MFU / 362k tokens/s on 64× GB200 (one NVL72 rack, d5120 / 48-layer / 64-expert top-4, replica-2 DDP, batch 1024, MuonH) with no OOM #7012. Scaling past one rack then hit a hard wedge at 8 racks — all 512 ranks going silent around step 11 with a kAllToAll rendezvous timeout — which Dial root-caused via HLO dump to the token embedding table sharding its hidden dim across all 512 GPUs, making the lookup an inter-rack all-to-all whose NCCL first-call rendezvous stalled. Fully replicating the table and doing a replica-local shard_map gather (numerically exact, zero collectives) restored clean scaling: 128 / 256 / 512 GPUs at 23.3% / 22.4% / 22.4% MFU, 2.79M tokens/s at 8 racks — roughly 2× throughput per doubling at flat MFU.

Running in parallel, Matt Wittmann drove a systematic expert-parallelism (EP) investigation on the standalone benchmark, spun out of #7012 into its own thread #7279. The picture at the 64-expert reference config is that EP stays net-positive but the margin compresses to under a point — ring_cute EP4 at 20.83% MFU versus 20.22% for pure FSDP on the identical 64-GPU layout — so for 64 experts FSDP remains the honest production choice, with a ~28% throughput gap to the FSDP driver run still to be decomposed. Two GB200 stack defects were characterized in depth: an intermittent cuModuleLoadData CUBIN-load fault that blocks EP above 8 at batch 1024 (root-caused to XLA's new per-fusion executable layout loading ~5,600 CUBIN modules per device at first execution, with no env or flag mitigation that survived controls), and a deterministic fast-restart deadlock in NCCL clique init. Adjacent threads sharpened the axis question: at 512 experts top-8 Rafal Wojdyla found EP inverts to a decisive win — EP64 a 2.4× MFU jump over the non-EP path, reaching ~18.2% MFU / 417k tokens/s on one rack with unvalidated e4m3 fp8 on the wire #7332; Wittmann's TransformerEngine NCCL_EP port built, ran, and trained at 64-GPU EP8 with MoE-layer chunking, hitting 18.0% MFU against 19.1% for the a2a_cute backend at loss parity #7331; and a source-push EP experiment was sealed as a failed gate after its best case lost ~2× to the existing backends #7333.

The largest single MFU lever this week was MXFP8 (block-scaled 8-bit floating point). Wittmann built a Blackwell-native path end to end #7282, landing on NVIDIA's MIT-licensed cudnn-frontend fused CuTeDSL kernels (grouped GEMM plus SwiGLU plus dual-orientation quantize folded into the epilogue) driven from stock JAX. The load-bearing lesson, confirmed across a dozen microbenchmarks, is that block scaling pays a quantization producer tax that only epilogue fusion removes — a standalone quantizer only reaches break-even. Wired into the grug MoE as a stateless whole-expert-MLP op, the fused path delivered an apples-to-apples +27.8% tokens/s over its own bf16 control at 64 GPUs (d5120 / 48-layer / 128-expert top-5, ~27.2% versus 21.3% MFU) with loss tracking bf16 to ~4e-4 per step, and +34.8% over the current #7201 production best. The quality gate — a compute-optimal small MoE matched on wall time versus bf16 — is tracked separately under #7271 and is not yet started, so this is a measured throughput win pending numerical sign-off.

Two pieces of measurement and operational plumbing underpinned all of the above. fray had no B200/GB200 device-FLOPS entry, so on Blackwell the canonical MFU chain silently returned None and dropped the stat entirely; Rafal Wojdyla added the entry and mapping so MFU logs the standard way #7328, then corrected the peaks to the GB200 NVL72 SKU that JAX actually reports (2.5 PFLOP/s dense bf16, up from an HGX B200 value), an ~11% downward MFU correction across every Blackwell run #7334; iris dev_gpu also learned GB200 #7327, and FlashAttention-4 was wired in as the GPU dense-LM default, 3.4× over the unfused fallback it silently replaced #7015. The 8-rack driver run also exposed an operational gap when it silently froze at step 145 of 500 with no error of any kind #7344; a hung collective is not a process failure, so Iris never retried it. Russell Power added periodic k8s thread-dump capture so a recurrence leaves per-rank forensics #7346, and an NCCL collective-hang watchdog now defaults xla_gpu_nccl_termination_timeout_seconds to 600 for GPU training runs, turning a silent wedge into a crash Iris can retry #7350. Feeding the architecture decision for the September hero run, Dial opened a GB200 TPS tracker #7201 that now holds a menu of candidate shapes measured and projected to 12 racks — a 129B-total 4-of-64 config at 22.3% MFU, several ~250B 4-of-128 variants near 19–21%, a first d6144 / 359B push, and an MLA-versus-GQA sweep in which GQA stays ahead on tokens/s.

The rollout-speed epic put real weight behind its H100 focus this week. Will Held extended the inference broker to launch accelerator-specific vLLM workers, adding an isolated CUDA backend alongside the existing TPU path so the same broker can stand up GPU workers on CoreWeave — they validate their resource shape, infer tensor parallelism from the requested GPUs, and share the pinned CUDA vLLM launcher (vLLM 0.25.1 on CUDA 13, native/Triton sampler so runtime images need no nvcc) #7369. The full Qwen3 GPU eval suite ran clean on CoreWeave — 16,165 matched requests with no drops, timeouts, or rejections. He also opened #7113, still in review, which layers brokered logit mixing on top: a completion façade fronting two brokered vLLM systems (teacher and student) that queries both models concurrently at each token step, mixes the union of their top candidates, and samples from the combined distribution. Rohith Kuditipudi noted the mixer currently assumes both engines share a host — fine for the small models he runs two-to-a-node, but it will need splitting across accelerators for the 1e23 model.

On the throughput side, Benjamin Feuer closed out the rno2a 30B-coder campaign with a node-count win. Measuring MarinSkyRL (fully-async GRPO) on Qwen3-Coder-30B-A3B over the terminal-bench / Harbor agentic environment on CoreWeave H100, the final v0m run validated a 4-node / 32-GPU geometry — policy and reference colocated at expert-parallel 4, plus two inference engines at TP8 — that keeps generation fully fed at half the eight-node footprint of the earlier v0k / v0l baselines #7221. Cutting inference engines from four to two and halving the generation workers, then colocating policy/ref at EP4, held the warm wait-for-generation-buffer at essentially zero, showing the eight-node runs had been over-provisioned on inference. The runs are policy-train-bound rather than generation-bound — policy training is ~70–78% of each compute step under CPU-Adam offload and gradient-checkpoint recompute — so the EP4-combined config was promoted to canonical. A companion result cleared an orjson parse-offload image as fidelity-safe but throughput-neutral, tracing the generation “sawtooth” to a benign batch-tail drain rather than a parse bottleneck.

This Investment epic carries the framework-selection question — evaluate the candidate reinforcement-learning frameworks and settle which one Marin standardizes post-training on. The week's one merged change was a small rollout-correctness fix: #7179 from Russell Power kept the beginning-of-sequence (BOS) token in the fallback chat template that #7178 installs for template-less checkpoints. A rendered chat template is tokenized with add_special_tokens off, so base and midtrained rollout prompts had been coming out one BOS token short of what they were on main; opening the template with the tokenizer's bos_token, the way real chat templates do, restores parity, and tokenizers with no BOS render it as the empty string and are unaffected. On the framework-comparison side, the cross-framework xorl reproduction #6915 — walked back last week under a same-ruler audit — closed with Benjamin Feuer drawing the line between the two candidates: xorl eliminates the behavior-versus-trainer numerical gap (bit-exact sampler and trainer, so PPO clipping never fires), while MarinSkyRL measures and corrects it via token importance sampling (TIS) plus a PPO-clip backstop, a residual that on the reference run is small and quantified (log-ratio ~1.3e-3, clip ratio ~3e-4). He enumerated what MarinSkyRL, or a successor framework, would need to go bit-aligned — batch-invariant sampler/trainer kernels on the vLLM side, matched fp32 lm-head and sampling, and RMSNorm and RoPE-cache alignment — and flagged the one clean recipe lever the audit surfaced: if xorl's eval gains survive the strict parity grader, swapping MarinSkyRL's default optimizer from AdamW to Muon would be worth adopting.

The rest of the week was rollout plumbing toward RL-ing the hero-run model. Romain Yon landed a v0 of Grug 67B-A2B GPU inference on main; Benjamin Feuer flagged that he cannot start RL until the mlfoundations/vllm fork is synced and MarinSkyRL has Grug-compatible training code, and filed #7164 — the remaining gap is a training GrugMoeForCausalLM PyTorch modeling class that MarinSkyRL can load via from_pretrained / trust_remote_code. Separately, last week's 131k-context MoE RL feasibility mapping banked its first result: Benjamin Feuer's #7248 cleared the first RL training step (gs1) of Qwen3-Next-80B-A3B at 131k context on 128 CoreWeave H100s, after fixing an off-GIL routed-expert-replay capture path that had starved the NVIDIA Collective Communications Library (NCCL) heartbeat into a watchdog abort, and raising the Fully Sharded Data Parallel submesh process-group timeout to the worker timeout; the step ran training-bound rather than inference-bound, confirming 80B-A3B naive-map RL at 131k is feasible on that hardware. And on Russell Power's infra track to make all post-training runnable from the Marin repo #7098, Benjamin Feuer landed a first batch of PRs — #7231 through #7234 — wiring a chat-template-parameterized SFT launcher, a TPU evalchemy eval launcher, and two robustness fixes onto Marin's ArtifactStep flow.

On environment sourcing, Jeff Hammerbacher opened #7147 to integrate OpenReward's roughly 500-environment Open Reward Standard catalog into marin.rl by reviving and rebasing the dormant #4661 onto the current MarinEnv adapter interface, with the OpenReward team offering trial credits and an integration guide. Russell Power floated #7245, a side-project to mine a reusable corrections curriculum of common agent failure-recovery patterns out of rollouts, and a bot-filed #7379 proposes replacing the fragile append-while-read JSONL literal log that carries per-token RL training signal with an object-storage-native sealed-segment LogStore shared by the harbor writer and the MarinSkyRL reader. In the reinforcement-learning channel, Benjamin Feuer also noted Megatron outperforming FSDP2 and mused about switching the RL trainer default — an offhand signal rather than a decision.

A quiet week on the curation side, with one pointer that sets the epic's direction: Isaac Hodes connected this RL-data work to #7170, the intermediate-cut post-training on the 67B-A2B 2T MoE. The data curation and experiments tracked here feed the RL leg of that run — the first post-trained Marin MoE — which is now unblocked by the published 2T cooldown checkpoint #6811. No new ablations landed this week; the near-term work is preparing curated RL data for that leg, alongside the parallel OT-collective effort noted in the epic.

Routine maintenance accounted for much of the non-epic traffic. Dependency bumps kept the lockfiles current, including the security and general uv groups #7326 and #7325, alongside a move to torch 2.13 on CUDA 13 #7375, #7330. Automated nightshift sweeps continued their daily dead-code and cleanup passes #7299, #7148, and a batch of hygiene work trimmed the tree further: retiring mypy in favor of pyrefly #7202, Russell Power removing unused experiment config modules #7158 and deleting the broken Levanter EpochDataset #7215, and Rafal Wojdyla quieting aiobotocore credential-log bursts on S3 tasks #7196. A short series of version-plumbing refactors deferred artifact and checkpoint versions to an ambient BuildContext #7252, #7263, with a companion fix for fingerprint-time resolution in reference sweeps #7265.

A few exploratory proofs-of-concept also landed: smallquery, a Ballista-on-Iris distributed-SQL experiment #7272; restart-survivable result caching for ducky from Rafal Wojdyla #7022; and Will Moss prototyping DataFrame-native scatter ingestion that skips the Python round-trip #7200.

External contributors reached into several corners of the stack this week. Michael Ryan (Stanford CRFM) synced his raw-web-data-to-tokens extraction work in #2351, and it was his jusText fork that Will Held ran across all 4,573 science Focus Crawl WARCs to produce the week's 49.7-billion-token extraction #7397. Kaiyue Wen (Stanford CRFM) drove two Agent-MoE architecture experiments against the July baseline — identity hyper-connections in #7409 and an over-embedding table in #7368, the latter cutting loss by about 0.025 (roughly 10%) at under 5% throughput overhead. Will Moss (Airbnb) carried much of the Pulumi infrastructure-as-code push, getting the first live CoreWeave cluster cleanly imported into Pulumi state in #7356 atop his design doc #7151 and first program slice #7153. Rohith Kuditipudi (Stanford) made fsspec hub downloads atomic #7168 and caught that Delphi's HumanEval was labeled 10-shot but effectively ran 0-shot #7229. Further contributions came from timodonnell wiring FlashAttention-4 into the GPU attention default #7015, Tschoelll's grug value-embedding ablation #7230, Matt Wittmann's FP8 loss-curve validation #7298, and sthacher76 restoring legacy RoPE metadata on HF export for transformers<5 #7156.

On Discord, Benjamin Feuer announced that the Snowball 2T cut had landed with SFT, chat templating, vLLM serving, and evals all green #7321. Its provisional numbers then drew outside scrutiny: Jenia Jitsev (LAION/JSC) flagged the MATH-500 score of 0.16 as suspiciously low for the training scale, citing LAION's own MixtureVitae 1.7B result at the same settings. That prompted Will Held to trace it to generation truncation — 87% of the errors were hitting the token cap — and a 32k-context rerun scored 54.2%. Separately, in #news, Kaiyue Wen unpacked the Kimi K3 report, noting its latent MoE, per-head Muon, and quantile balancing — the last echoing the team's own QB routing.

Seven people introduced themselves this week, spanning an independent high-school researcher working on reranking and code-model calibration, a University of Bath postdoc, a voice/NLP engineer, an AI-architecture concept designer at Nexus Studio, and Sergey, who works on protein folding, evolution, and design — a background that intersects the marinfold effort. Two arrivals land next to especially active threads: AlexPalms, an early core technical-staff member at Yoshua Bengio's LawZero focused on large-scale distributed and RL-based training at hundreds-of-GPU scale, which intersects the post-training and RL work; and Huu Nguyen, whose interest in pretraining models to generate their own subsequent training data intersects the datakit pipeline.

Shared reading this week clustered on fresh frontier-model architecture disclosures — Thinking Machines' Inkling, Kimi K3, and Gemma 4 — and their attention, MoE, and optimizer choices.