Marin: Week of June 22nd summary

The H100 cluster crossed from booting to actually training Grug MoE jobs this week, on the back of a batch of operability fixes. Iris GPU tasks now stage the CUDA 13 toolchain so JAX/Pallas Mosaic kernels compile on CoreWeave workers (#6637, closing #6636 and #6600 after the #6601 draft), default a JAX compilation cache so XLA and cross-entropy autotune results persist across runs (#6640, closing #6638), and write Levanter caches and checkpoints to CoreWeave virtual-hosted S3 object storage (#6676, #6677, #6548). A W&B finish() hang that had dropped a passing canary's metrics file was moved into a finally block (#6659, closing #6364). On that footing David Hall built a per-device roofline and attribution dashboard for the d2560 production shape (#6557, #6573), and the team set an explicit bar in a fresh workstream tracker: above 20% MFU at 128 GPUs #6693. The verdict out of the OA meeting on GPU progress was ~19.9% MFU on four nodes with SGD/Adam, with some low-hanging fruit still on the table.

The dashboard made the Muon question concrete. On a two-node run David Hall measured Muon's Newton-Schulz work at roughly 500 ms of a 1,500 ms step, and even at full efficiency it would remain a large fixed cost without much faster interconnect or a different parallelism layout. The single-node grouping experiment #6493 had already returned a negative result: grouped and padded Newton-Schulz neither avoided OOM during autotune nor improved end-to-end throughput, because restore and all-gather boundaries erase the sharding win. Russell Power's pipeline work quantified the same Muon tax under Fully Sharded Data Parallel (FSDP) at +0.92 s per step on two nodes, about a 60% throughput hit that grows as the gradient all-gather crosses the inter-node link. The call, recorded in the same meeting notes, is to shelve Muon on GPU for now: a ~10% step-count improvement is not worth that cost at this interconnect, so #6493's harness is parked rather than pursued.

Pipeline parallelism (PP), last week's leading candidate for both the Muon tax and the first-step memory wall, reached its own verdict. Russell Power drove a thorough zero-bubble investigation (#6532, PR #6534): the manual-threaded schedule is gradient-exact against the oracle and genuinely unlocks memory scaling, fitting an 8M-token batch under 1F1B where FSDP OOMs, and it amortizes Muon (PP's Muon tax is ~3.2x smaller than FSDP's). But it does not beat FSDP on throughput at the relevant scale, running ~0.78x on a single host and only pulling ahead (~1.22x) once FSDP's all-gather crosses the slower data-center interconnect at v6e-32. He could not beat FSDP with 1F1B or zero-bubble on H100, so the work was parked to an external repo and PR #6534 closed unmerged. With Muon and PP both set aside, the chosen forward bet is a Pallas/Mosaic MGPU fused MoE kernel: David Hall pushed a DeepEP-free expert-parallel (EP) dispatch plus W13/SiLU path (#6597, tracked under #6694) through a long string of H100 correctness milestones and steady speedups, with the fused down-unpermute stage falling from ~40 ms to ~2.6 ms, though it still trails the ragged all-to-all baseline and remains correctness-first. Supporting kernel work landed alongside it: the D128 GQA FA4 backward now routes through the native SM90 path (#6613, closing #6632), and a B-tiled fused cross-entropy bounds H100 logit memory (#6572, follow-up #6596).

On the headline question of whether the June 67B-A2B run started: it did, but on TPU rather than on the H100 cluster. The d2560 production shape that hit the first-step memory wall on GPU last week was staged onto TPU v4 instead, where XLA handles Muon's Newton-Schulz cheaply; Larry Dial found TPU Muon throughput good while noting the same approach may not be feasible on GPU. The W&B record shows several 67B-A2B d2560 attempts on v4-512 and v4-1024 slices crashing across 6/24 to 6/27, and as of 6/27 a Muon variant on a v4-1024 slice is running, roughly 178B tokens in at a train loss of ~1.65 and ~13% MFU. So the H100 cluster's June production run was effectively handed off to TPU while GPU work continues on MFU, the Pallas kernel, and the above-20% bar #6693 ahead of the July run (#6706, #6710); the H100 multinode hero-run POC #6716 remains open.

After a quiet stretch, this epic picked back up on two fronts. On the kernel side, #6660 adds Fp8DirectDotGeneralOp, a haliax dot_general that feeds genuine E4M3 operands into the matmul. The existing fp8 path was vendored from Flax and used the legacy quantize-dequantize (QDQ) recipe, where the operands handed to the GEMM are dequantized back to the compute dtype, so whether fp8 tensor cores actually run depends on XLA's GemmRewriter folding the pattern; on Hopper the forward GEMM silently fell back to bf16. The new op vendors Flax's newer direct-quantization primitives, doing explicit quantize then fp8 dot_general then dequantize, with output gradients in E5M2 via a custom VJP so both backward GEMMs also run in fp8. It is opt-in through Linear(dot_general=...) and changes no existing behavior; the PR ships tests for forward numerics against bf16, confirmation that the forward and backward dots genuinely contract float8 operands, and a delayed-scaling state-update check, with the GPU-gated kernel test verified passing on an H100.

On the training-path side, #6699 opens a tracker for an FP8 Expert grouped matrix multiply (GMM) on the MoE path, under the GPU-perf parent #6693. The definition of done asks for a reproducible run command, a timing and MFU delta against the bf16 expert GMM with a target of at least a 40% speedup, plus numerical-stability and memory notes. This came out of an OA GPU-progress meeting that David Hall summarized in #gpu, where the FP8 Expert GMM lands as one line in a broader MoE kernel and fwd/bwd tuning workstream sitting at roughly 19.9 MFU on four nodes. Both items remain open and early; this is the wiring-up phase the epic's definition of done calls for, ahead of the larger stability and throughput checks.

The extension ladder itself stayed parked this week; the visible motion was organizational, as long-context planning folded into the July hero-run architecture effort. A new umbrella issue, #6711 (“Model architecture & scaling recipe (MoE)”), explicitly scopes MoE architecture research “including long-context” and pulls this epic together with the 8k-seqlen pretraining probe #6277 and the long_context_32k sensitivity sweep #6232, routing their conclusions into the hero-run architecture issue #6701 and its loss preregistration. #6701 opened mid-week with a one-line charter to include the long-context plan, so the 4k→32k→64k ladder John Larry Dial sketched last week now reads as one input to the July architecture decision rather than a standalone track. None of the named extension experiments — #6170, #6171, #6194, #6232, #6277 — saw new runs or comments this week.

Two older long-context threads were cleared out during open-issue triage. The original #2062 (“[Epic] Long Context Plan”), the SmolLM3/Olmo-3-style 4k→32k→64k retrofit plan, was closed as superseded by the current roadmap after 80-plus days without human activity — its goals now live in the MoE-native experiments above. The Levanter throughput issue #4276 (“[Levanter] Long-context Training Optimization”) was also closed, with a managed experiment TL;DR appended noting the recorded work stopped at benchmark setup: a clean v5p-32 baseline for 131k-token Qwen3-8B SFT plus queued CP2 / CP2+pdp2 / CP4+pdp4 context-parallel variants, but no steady-state throughput or winning layout on record.

The one piece of live architecture signal came from the moe channel, where David Hall floated sequence-level QB — a router balance loss that forces expert load balance within each sequence rather than only across the batch — as a promising way to handle long context, with the side benefit of being friendlier to the MoE kernel. John Larry Dial noted the proposal builds on a QB v2 that drops the strict top-k-per-token constraint and wondered aloud whether that variant stays kernel-friendly, before adding that it does also address the top-k QB formulation currently in use. Separately on the RL side, the MarinSkyRL retrospective #6656 reported context parallelism (CP) and its rollout-side counterpart (DCP) validated at short context, while 131k-token production remains blocked on degenerate expert-parallel all-to-all routing — a reminder that the long-context push spans both the pretraining architecture and the post-training stack.

No work landed directly under #6045 this week, and that is the expected shape at this point: the substantive 10T-datamix effort lives in its siblings. The mixture science — metric selection, buckets, and the production swarm — was carried in #5359, now closed, and the actual data staging / tokenization for the CoreWeave cluster continues under #6036. The final June mix itself was already handed off last week when #6391 (deliver the final mix for forecasting) closed, so the 10T-for-June production target is effectively met and out of the critical path.

The forward motion this week was the roll-over into July. A dedicated tracker, #6700, was opened for the pre- and mid-training data mix of the July hero run, and #6713 reframes pretraining data curation and mix as a standing area of investment picking up after the July commitment — explicitly pointing at #6700 and #6036. In short: the June 10T mix is produced and delivered, and further datamix iteration is now a July concern tracked through #6700.

The centerpiece of the June milestone went live this week: the 67B-A2B production MoE — the d2560 “Grug” shape, 67.1B total parameters / 2.01B active per token — began its ~10.07T-token pretraining run, tracked in #6044. After the H100 bring-up stalled, Larry Dial moved the run to TPU, starting on a v4-1024 the morning of June 24 and settling on the full v4-2048 (1,024 chips) in us-central2 by June 27. The launched configuration — moe_67b_a2b_d2560_ep1_rep16_bs4096_seq8192_sw2k_v4_2048_muon_10T — runs MuonH on the May Recipe with expert parallelism (EP) of 1 and a replica axis of 16, seq_len 8,192 / sliding window 2,048, and a batch-size ramp from 33.5M to 67.1M tokens at step 15,000 (~5% in) to secure a stable warmup before pushing batch size for throughput; projected wall-clock is ~46–49 days. Three architecture changes were made specifically for this run: partial key offset (PKO) and RoPE on the global full-attention layers were disabled to keep the model close to published long-context recipes ahead of a planned 262k-token extension, logit z-loss was re-added because it helps materially under extreme overtraining on the new data mix, and the learning rate now decays to 5% of peak rather than 0 since context extension and post-training will follow.

Crucially, the run was preregistered. On June 27, before stage 1 completed, Larry Dial filed a preregistered loss target of 2.269 paloma macro loss for the first 8T tokens (evaluated at seq 8192 over 1,024 sequences), backed by a three-point Chinchilla-form scaling fit (L = E + A·C^(-α)) anchored on the june_prep d512/d768/d1024 80%-mark checkpoints, which project 2.258–2.295 across irreducible-loss assumptions of 1.3–1.6. He documented the fit's load-bearing assumptions explicitly: an assumed irreducible loss of 1.4, the re-added z-loss roughly cancelling the loss penalty from the much larger batch, extrapolating from only three points (versus 20-plus for the 1e23 run) to let the run start sooner, and the fact that stage 1 ends at 24% of peak LR. The optimizer hyperparameters all derive from the refit MuonH May Recipe heuristic #5951 (R² = 0.996 on 17 cells).

Most of the week's work in the thread was choosing hardware and root-causing a Muon-specific numerical instability — squarely the “Muon” theme, here on TPU rather than GPU. Per-chip MFU is best on a v4-1024 (512 chips, ~21.5%) because d=2560 shards cleanly under Fully Sharded Data Parallel (FSDP) with EP=1 (2560/512 = 5), but that projects a 78-day runtime; the full v4-2048 (1,024 chips) doesn't divide 2560 evenly and needs replica sharding plus a doubled batch, trading per-chip efficiency for ~2.57M tokens/s and ~45 days. The v4-2048 runs initially diverged, and Larry Dial traced it to a real bug: the sharded Newton-Schulz orthogonalization in Muon was silently dropping whenever the number of matrices (256 experts × 26 layers) didn't evenly divide the chip count — true on 512 chips but not 1,024 — so the optimizer was applying un-orthogonalized updates, which look Adam-like at first and then degrade. Explicitly replicating the Newton-Schulz calculation over the replica axis and patching the silent failure fixed it, reproducing a clean v5p-8 reference run on v4-2048. He flagged that the bug should have errored loudly, and used the postmortem to argue for human review of agent-written prod-scale resharding and a first-200-steps validation script for future runs.

The TPU launch is the flip side of the GPU verdicts that closed last week's threads. Russell Power's zero-bubble pipeline-parallelism (PP) prototype #6532 was proven gradient-exact and unlocks memory scaling, but could not beat FSDP on throughput except once the parameter all-gather crosses the data-center network, so it was parked to an external repo and PR #6534 closed unmerged. David Hall's roofline dashboard #6573 measured Muon eating roughly 500 ms of a 1,500 ms H100 step, so the June 26 OA meeting shelved Muon on the GPU path for now and refocused GPU efficiency on a hand-written Pallas/Mosaic fused MoE kernel #6597 — already ~2.09x over the ragged all-to-all expert-parallel baseline — under the above-20% MFU at 128-GPU tracker #6693. Around the live run, the team opened the July tree: post-training the full 67B-A2B 10T model #6705, an 8B-A1B GPU proof-of-concept #6716, July data-mix #6700 and architecture #6701 planning, and a next-generation 288×B200 run #6689.

No leaderboard or UX surface shipped this week, but the eval engine room saw real plumbing. Calvin Xu opened #6726, a Marin-native evaluator for the OLMoBaseEval Easy Table 9 bits-per-byte (BPB) suite that scores the 51-component set directly in Levanter on TPU, removing the convoluted external Stanford-SC transfer path used for data-mixing checkpoints. On a v6e-8 it reproduces the SC oracle on the full 104-task suite to a macro absolute difference of 1.9e-8, with all 51 components within 1e-3, and derives the model config from the checkpoint's model_type so both Llama and Qwen3 load region-locally; the companion tracking issue #6717 covers flagging any still-missing components so no SC transfer is needed. Alongside it, Russell Power merged #6662, folding the two duplicated, behaviorally-divergent model-loading blocks in save_logprobs and trace_labeled_eval into single-purpose load_levanter_checkpoint / load_hf_checkpoint loaders and closing #6336; the HF-repo resolution now lives once beside the distributed-locked snapshot cache, and inference's quick_serve routes through the same resolver.

The leaderboard-and-automation direction stayed at the planning layer. Benjamin Feuer's #6499 publishes the Eval Manager role definition — the four pieces being a Marin Leaderboard, a Marin Listener that auto-launches evals on newly-landed models, the model-and-eval database behind both, and the eval suite plus per-stage go/no-go gate metric (nothing past Stage 3 advances on perplexity alone). Feeding the same spine, Isaac Hodes filed #6703 to pin down which evals get run, and whether they are ready, inside last week's zero-touch automated-eval system spec #6503. On the GPU side, Romain Yon's #6566 scopes a first realistic OpenThoughts-Agent eval smoke on a CoreWeave H100x8 nodepool via Iris, validating the OpenThoughts GPU image, vLLM, and the Harbor/Daytona path end to end rather than chasing score parity.

A backlog sweep of long-standing eval bugs also closed out: the eval_harness crash on HF tokenizers calling the missing encode_batch #4943, the default_eval generation-task out-of-memory failures #2033, the outdated extract_model_name_and_path appending a stray hf suffix #2439, and two speedrun-leaderboard items — the W&B training-time discrepancy #1767 and surfacing token counts in the leaderboard #2255. Ahmed Ahmed's open-weight LM-judge search #4790 also landed its conclusion: zai-org/GLM-5.1 as the default open-weight judge, matching GPT-5.1 rankings within ~0.009 median Spearman at roughly 16x less cost, with pair-judging carve-outs for known weak rubrics. On Discord, the friction of running evals at scale showed through in the data-mixing channel, where Will Held shared optimized-mix-versus-proportional scaling on the Uncheatable Eval macro BPB and the "big three" pretraining evals, and noted to Percy Liang that generating the full ~200-eval set at once (across 840 data-swarm models) had previously crashed Iris. Separately, Russell Power enabled customizable Docker environment privileges so sandboxed eval setups can run locally without sourcing out to Daytona, and a new contributor opened #6537 against the eval codebase for help-wanted issue #6238.

This epic stayed quiet again: no GPU-specific GrugMoE serving work landed this week, and the GPU vLLM path still has no checkpoint-loading, correctness, or performance work of its own. The week's one piece of GrugMoE serving activity, #6732 from Romain Yon, repins Marin's vllm and tpu-inference forks to preserve the GrugMoE serving overlay, and it was validated end to end on a TPU v6e-4. That keeps it on the TPU inference path rather than this GPU epic.

Two cross-cutting MoE inference threads also stayed warm but land elsewhere: the chunked-routing throughput sweep in #6509 and the packed-vLLM rollout study in #4286 both run on TPU v5p. The GPU-flavored GrugMoE discussion this week was all training and kernels — B200 FlashAttention-4 attention shapes, FP8 recipes, and MFU/roofline profiling — which is GPU training, not serving, and sits outside this epic.

The epic's headline goal landed this week. #6664, authored by Romain Yon and merged on June 27, adds correctness-first GrugMoE support to Marin's TPU vLLM path: it exports a Levanter GrugMoE checkpoint into a vLLM / tpu-inference-compatible artifact, pins Marin-owned vllm and tpu-inference fork commits carrying the GrugMoE TPU overlay, and adds an opt-in real-checkpoint end-to-end (e2e) test under tests/vllm/. A fresh run on a v6e-4 slice in europe-west4 loaded the real checkpoint and produced identical greedy output from the vLLM serving path and the Levanter/JAX reference (both decoded the same deterministic continuation), with the e2e suite passing. That is the epic's definition of done — checkpoints load and serve, and correctness is pinned against the reference — reached for one checkpoint and one short-decode prompt. Romain Yon is explicit about the remaining surface: this does not yet validate throughput or latency, tensor- or pipeline-parallel (TP/PP) serving, broad context windows, router replay, or routed-expert/logprob exposure through the OpenAI-compatible completions endpoint.

Fork maintenance ran alongside the feature. #6732 (merged June 27) repins the vllm and tpu-inference dependencies onto new GrugMoE fork commits rebuilt on the fork SHAs Marin used before #6664, deliberately keeping the GrugMoE serving capability while dropping unrelated newer upstream fork churn; its Marin-side diff is confined to pyproject.toml and uv.lock. Two refresh-track efforts remain in flight: #6733 (draft) bumps the stack to tpu-inference v0.23.0 plus its tested vLLM last-known-good build but holds CI green only by keeping the Marin forks accepting Transformers 4, and the new issue #6734 calls that Transformers-4 bridge a temporary state rather than the steady one. The follow-on #6735 moves Marin and Levanter onto Transformers 5 / HF Hub 1.x, porting the durable fixes from the now-closed #6483 and relocking to transformers==5.12.1. #6724 writes down the post-merge protocol for promoting the fork main branches after a Marin PR lands.

On the eval-path side the work is adjacent rather than wired. #6556, from Russell Power and merged June 22, ships marin-serve, a one-liner that boots vLLM on a single-host Iris slice behind a dashboard and an OpenAI-compatible API — the natural front door for poking at a served GrugMoE checkpoint once perf work begins. In #infra, Russell Power demonstrated his Tunix evaluator running as a one-off inference service on idle v6e capacity, and Romain Yon replied that he wants to use exactly that to play with Grug vLLM inference; earlier he had posted the #6664 draft in #code-talk asking for a direction check (against the moe_may_compute_opt_d512_ep1-05c39b checkpoint) before further de-slopping. The historical scaffolding for all of this — the move to Marin-owned forks #4357, the Ray multi-host vLLM exploration #3792, the JAX bump #4502, and the CPU/vLLM dependency-conflict cleanup #6481 — is all closed, leaving performance, parallel serving, and eval-harness integration as the visible next steps.

Last week's intruder-coherence hill-climb gave way to two cleaner local-metric efforts. The headline is the document-quality scorer that ranks Datakit documents: Russell Power's agent-driven thread #6739 asked whether a cheap "fast-transformer" (embed → pool at 64-token boundaries → a few layers → regression head) could beat the deployed v0 fasttext quality classifier while staying under ~1M floating-point operations (FLOPs) per token. Scored apples-to-apples on the same Claude-Sonnet oracle-labeled holdout (961 docs) with the same AUC / Spearman harness, the winning config landed Spearman 0.703 vs fasttext's 0.641 and AUC 0.875 vs 0.846 at 0.41M FLOPs/token, and matched or beat it on a calibrated F1 (0.751 vs 0.718) — a win on every metric, shipped as PR #6741. Two findings sharpen the precision/recall picture: most of the gain comes from learned embeddings plus multi-statistic pooling rather than the attention layers (an L=0 "neural bag-of-words" runs at 12K FLOPs/token and nearly ties the best), and the scorer is data-limited at ρ≈0.70 — naively adding free Nemotron-CC quality-bucket labels did not help, so the bottleneck is more diverse oracle-style labels, not more web-quality ones.

The Datakit Testbed's ranking protocol — the local-metric proxy for whether a data transform actually helps — also produced its first verdicts and closed out. The no-dedup baseline #5308 established a tight noise floor (eval/loss 3.3967 ± 0.005 across four seeds), against which the default fuzzy-dedup arm #5309 won by roughly 3-5 sigma, largest on structured text where boilerplate duplicates cluster. The 50%-duplication negative control #5310 returned a null result at 1e18 FLOPs: at this undertrained scale the model is data-limited and exact duplication itself does not register, though the related drop-half calibration did show clear harm. Russell Power closed the parent testbed #5200 and all three arms during open-issue triage, and David Hall's agent backfilled experiment TL;DR blocks on each. One caveat is flagged for later: the dedup win is not a clean quality isolation, since removing duplicates also shifts proportional mixture weights, so a pinned-mixture follow-up is on the table.

On the plumbing side, the team took on the clustered store's file-count problem: the ~22TB, 10T-token store fans out to roughly 6M files (~14.2M tiny leaf caches) under default flags, which hurts both Zephyr jobs and the dataloader #6687. Rafal Wojdyla's agent opened a shuffle-based store build #6725 that groups each (cluster, quality) bucket into one large materialized cache — benchmarked at ~17TB in-region scatter with no egress — alongside a draft web visualizer #6641 for browsing the clustered-store output, and the stale tokenize file-scan issue #4411 was retired. In Discord, Will Held shipped a quality viewer over the data filter to eyeball what the classifier keeps, and Pranshu Chaturvedi reported tokenizer experiments on numeric handling — right-to-left digit alignment and capping long digit strings — finding the gpt-oss tokenizer benefits from the new numeric logic while Llama does not.

The week's headline for the CoreWeave data pipeline was a storage decision. Rafal Wojdyla ran a controlled object-store micro-benchmark on the cw-us-east-02a cluster, pitting CoreWeave's native object store against the Cloudflare R2 store the epic had tentatively leaned on, and closed #6671 with a clear recommendation: standardize pretraining data on the CoreWeave store. Across both a single pod and an 8-replica fan-out, CoreWeave beat R2 on every axis measured — reads, writes, listing, and tail latency — often by an order of magnitude; the 2.85 GB benchmark corpus wrote in 2.7 s on CoreWeave versus 37.9 s on R2. Reads are best routed through LOTA, CoreWeave's node-local read accelerator, which reached roughly 65 GB/s aggregate on warm re-reads, while writes go direct to cwobject.com. Neither store throttled under fan-out, so the case for CoreWeave is speed and tail latency, not error rate. The cost flagged against the epic's confirmed-regions item is that the CoreWeave bucket is regional to US-EAST-02A, so any non-CoreWeave consumer such as a GCP TPU cluster pays cross-cloud egress, and Iris still needs a CoreWeave access key wired into its task environment.

That decision reframes the staging work that remains. To unblock the megagpt-speedrun H100 runs before native CoreWeave data existed, a throwaway R2→cwobject mirror plus an S3 monkeypatch stack was built in the experiments repo; #6688 was filed this week to delete all of it once #6036 lands tokenized data generated natively on the CoreWeave store, while upstreaming the one durable piece — a virtual-hosted build_kvstore_spec for a custom S3 endpoint, on tensorstore ≥0.1.84 — into Levanter so that no future CoreWeave job needs monkeypatching. Two new umbrella issues reorganized the road ahead: #6713 covers ongoing pretraining data curation, mix tuning, and productionizing the CoreWeave pipeline, and #6715 covers cluster reliability. The latter is timely — Benjamin Feuer reported CoreWeave node-health-check sweeps cycling about every 25 minutes and bouncing his gang-scheduled jobs, which Russell Power traced to a priority-and-quota interaction and addressed by restarting the cluster so Iris jobs always schedule above the health-check priority.

Source registration and tokenization both moved. Will Held landed ingestion of Common Crawl WET — the crawl's plain-text extract — via the common_crawl_wet Datakit downloader, and registered two new corpora: the science-focused focus crawl CC-SUPPLEMENTAL-2026-22 and a token-matched sample of the general CC-MAIN-2024-18 main crawl, llama3-tokenizing both into fixed token pools for the quality ablation tracked in #6113 and #6570; early ladder rungs put the focus crawl decisively ahead on scientific evals such as arXiv physics and S2ORC and behind on broad web and Paloma macro loss. In parallel, Pranshu Chaturvedi advanced the numeric-aware tokenizer variants behind #4915 — digit-run isolation plus right-to-left, place-aligned three-digit grouping, capped at 510 characters to avoid pathological regex backtracking — wired into a Datakit tokenizer sweep in draft PR #6571 and feeding the MoE tokenizer-sensitivity experiment #5821, where the gpt-oss place-digits variant currently posts the lowest bits-per-byte. Michael Ryan reported continued progress on the small-model raw-web-to-tokens pipeline in #2351, narrowing classifier and extractor choices toward a throughput-versus-quality frontier.

The week's one merged change here was #6537, which adds surface diversity to the string_slicing generator in the PPL circuit-coverage v2 suite. The task still computes repr(text[start:stop:step]) with target-only supervised records, but generated examples now rotate deterministically across five forms — the original Python statement plus function-call, inline-subscript, pipe-delimited, and JSON-field shapes — keyed off the row index so small local samples cover every variant without disturbing the task RNG. The raw source and executor names move from compact_v1 to compact_v2 because the materializer writes the full 27-slice bundle from a single step, so the bundle version has to be the cache and provenance boundary. The change came from new contributor ma08, who scoped it to a single task to match the bounded first-contribution shape and flagged it in code-review, offering follow-ups for the other listed families.

That PR closes out the first slice of #6238 (template diversity for existing tasks) and leaves the sibling #6236 (new circuit/task families) open, where wasup-yash asked about contributing. Both descend from the #6070 circuit-coverage gap dashboard, which showed Marin 32B trailing Qwen3 32B on short deterministic continuations — arithmetic, string and token manipulation, exact character tracking, and literal formatting. On the documentation side, David Hall's #5318 was closed by Russell Power after the perplexity-gap workflow was written up at docs/references/perplexity-gap-analysis.md, so the raw-validation, score-cache, and comparison entrypoints now have a stable home separate from the LM-eval tutorial.

Looking past the circuit probes, Isaac Hodes opened #6712 as an umbrella for data-selection diagnostics — strengthening the PPL and eval signals that predict downstream quality, with rank correlation between a PPL proxy and downstream evals as the metric to improve — picking the thread back up now that the July commitment and hero-run work is settling. Alongside that, two older agentic-proxy investigations were tidied up: #5401 (the 300M agentic-coding bits-per-byte, or BPB, eval) was closed with a results TL;DR finding success-only agentic BPB a usable signal at high signal-to-noise, while #4389 (a soft proxy for agentic benchmarks) was closed with the work consolidated into #4550, which Rohith Kuditipudi said he would update shortly.

With the June mixture finalized and handed to forecasting last week, the epic closed out: metric selection #5362 wrapped up when Will Held landed the optimization objective — a hinge on task-versus-proportional loss plus code and math task-loss terms and an L2 anchor back to the proportional mixture. The week's visible payoff was scaling validation. In data-mixing, Will Held posted the first optimized-versus-proportional scaling curves on Uncheatable Eval macro bits-per-byte (BPB), with the optimized mixture's gains dominated by code and arXiv speedups, then added a three-way comparison against the old Marin mix across the "big three" pretraining evals. Percy Liang flagged the crossovers as exciting and asked for a way to regenerate the full eval set in one pass.

Underneath, Calvin Xu drove a large scaling-validation push across the optimized-mixture sub-issues. On the Delphi ladder (3e18 to 1e21 FLOPs) he validated DSP- and OLMix-family mixtures optimized for Uncheatable Eval in #6602 and #6608 and for OLMoBaseEval Easy in #6603, stopping the canonical DSP cells once the early rungs showed effective-exposure DSP scaling better on uncheatable-eval BPB. He closed #6611 after assembling the 60M and 300M OLMoBaseEval Easy panel — 736 runs by 117 BPB metrics — and fit both methods on the paper-style 51-component Table-9 macro: effective-exposure DSP reached out-of-fold (OOF) Spearman 0.918 against 0.380 for the paper-faithful OLMix reference, with Marin's two-phase OLMix extension at 0.891. Supporting work landed proportional and UniMax-8 baseline scaling anchors in #6607 and a reliability-weighting study in #6665.

To get these checkpoints off the external Stanford-SC eval path, Calvin Xu also stood up a Marin-native OLMoBaseEval Easy Table-9 BPB evaluator that scores the 51-component suite directly in Levanter on TPU, tracked in #6726 and #6717, reaching roughly 1e-7 parity with the SC oracle after defaulting the executor path to fp32. Planning for the next cycle opened with #6700, the July pre/mid-training hero-run mixture, alongside #6712, a continuing data-selection-diagnostics track. As a directional alternative to mixture sweeps, Rafal Wojdyla reported scaling results for MATES/Group-MATES influence selection in #6521: the cheap in-context-learning proxy's selection advantage grows monotonically with model and pool size — ΔNLL on a GPQA probe rising from +0.04 at d512 to +0.089 at roughly 3B and reproducing across seeds — and at scale plain top-k MATES matched or beat the diversity-aware Group-MATES variant. He framed it as promising but not yet a head-to-head against a tuned RegMix mixture at equal compute.

The week's recipe-level news is a hardware split on Muon. On the GPU side, David Hall's roofline dashboard #6573 showed MuonH eating roughly 500ms of a 1,500ms step on a two-node H100 run, and the Open Athena GPU sync #6693 resolved to shelve it for now: a ~10% step-count gain does not pay for the interconnect cost without much faster networking or pipeline parallelism (PP), so the grouped-bank Muon harness #6493 is parked. Russell Power's 1F1B and zero-bubble PP prototypes could not yet beat Fully Sharded Data Parallel (FSDP) #6532, so that escape hatch stays open. On TPU the picture is the opposite: Larry Dial measured MuonH as essentially free, about 1.47M tokens/s at d2560 on a v4-1024, matching AdamH because XLA hides the optimizer cost #6493.

That split matters for the scaling recipe because the best point on the effective-speedup frontier is still a MuonH run (the “muonh-may-recipe” d1280 configuration, roughly 4.2x), so the TPU recipe keeps Muon even as the GPU hero-run path drops it. No new per-budget frontier record was set this week; the agent-MoE completions were validation and long-context work rather than new bests, including d1024 runs at 8k sequence length with a 2k sliding window and d512 re-validation at 4k, all landing below their prior frontier points.

On the tuning side, Ahmed Ahmed opened a thread on Muon and AdamH hyperparameter transfer (#5951, #4225); Larry Dial walked through why beta2 is floored at 0.95 and Muon momentum pinned at 0.95, with the LR doing most of the tuning work. Relatedly, Will Held reported that tuning betas lifted the loss-prediction R² at 3e20 from 0.75 to 0.978 with no divergent runs, directly the predicted-versus-actual val-loss metric this epic is trying to tighten.

Still no preregistration was filed. The line was instead reorganized under a new umbrella issue, #6711 (“Model architecture & scaling recipe (MoE)”), which routes this work into the July hero run, feeding its architecture #6701 and a dedicated loss-preregistration slot #6702 that remains an open, empty placeholder. The run-planning scaling-heuristics summary #4147 was closed.

The Delphi post-training line had its most public moment of the milestone this week. On June 23, Benjamin Feuer stood up Marin's first publicly served post-trained model — the 1e22-p33m67 Delphi SFT checkpoint on a live endpoint — and invited the community to red-team and vibe-test “baby Marin,” drawing hands-on probes from Tim O'Donnell, Russell Power, and others, with Percy Liang noting it had been a while since the team last stood up a model. Days later the study #6279 crossed from SFT picks into its first reinforcement-learning result. Ahmed Ahmed and Benjamin Feuer landed rlvr7500_w1, a forced-thinking RLVR run (GRPO via MarinSkyRL, batch 256 / G=16) on the 1e22-p33m67 SFT checkpoint, carried to step 140. Measured against the SFT start, it lifts MATH-500 from 45.0 to 53.4 (+8.4) and gsm8k flexible-extract from 64.1 to 68.5 (+4.4) — both format-robust columns move, so it reads as a real capability gain rather than a formatting artifact. AIME24 regresses (4.9 to 1.5), which they diagnose not as a plumbing bug but as forced-thinking-at-4k truncation: the longest chains-of-thought pile against the 3584-token generation ceiling and cut off before the boxed answer, so the verifier extracts nothing. Alongside the run, the grid harvest is now complete — the full 54-cell gsm8k flexible-extract SFT grid, the complete 9-scale BASE pass@k grid (pass@{1,8,32,128}, N=128), and a consolidated 129-repo model map whose Hugging Face ids were all verified live against the HF API.

Benjamin Feuer filed #6643 to fully document the p*m* midtraining procedure behind those 27 checkpoints — the K=0.20 continued-pretraining recipe, per-scale token budgets, learning-rate schedules, and base initialization — since the entire #6279 study initializes from them, yet the procedure itself isn't written down anywhere reproducible. On the scaling-laws side, Ahmed Ahmed and Will Held compared notes on the reference-model DPO sweep: tuning betas rather than holding them fixed took the 3e20 fit's R² from 0.75 to 0.978 with no divergent runs, with the caveat that some Delphi runs likely crossed the critical batch size under a constant-step setup.

Isaac Hodes opened a cluster of forward-looking post-training epics for July and beyond: #6705, the first post-trained Marin MoE — a hero run on the full 67B-A2B 10T model, owned by Benjamin Feuer on 512+ H100 (64 nodes) on CoreWeave — explicitly framed as starting on smaller token-count cuts before the full run begins later in the summer; #6714 for SFT data curation, targeting instruction-following lift on IFEval; #6707 for RL data curation and ablations, coordinating scope with the OpenThoughts collective; and #6626, a versioned Marin chat/response format library to own render and parse in both directions. The team also outlined its near-term RL-infra direction — continuing on the SkyRL fork while exploring a consistently-JAX path aligned with the pre-training stack — and onboarded a new RL collaborator to repro the Delphi midtrain and MarinSkyRL stack. On the eval-plumbing front, Romain Yon got a realistic OpenThoughts-Agent eval smoke running end to end on CoreWeave H100 via Iris #6566, and an open-issue triage pass by Russell Power closed several now-stale post-training items (#1175, #4426, #4556, #5715).

The week's cross-cutting work was dominated by the Iris cluster manager, where Russell Power landed a multi-backend architecture: a meta-scheduler that drives a collection of task backends behind a single dashboard #6730, a remote backend transport #6731, a lease-based EndpointService for service registration #6728, and the dashboard's backend-scope and overview tab #6740, all following the design in #6718. Scheduling grew more robust in parallel: blocked gangs can now preempt CPU-only squatters on their hosts #6675, task failures count cumulatively so coscheduled gangs fail cleanly instead of crash-looping #6673, and jobs whose resources exceed every matching pool are rejected at submit time #6541.

Iris's Kubernetes path was hardened as well — controller state is now persisted #6585, a bulk pod-metrics query cuts controller load #6568, and pod-log pulling was replaced with a Finelog log-shipper sidecar #6619. The controller dashboard was compacted and decluttered #6683, with Will Held restoring the fleet overview and per-node job links on the capacity tab #6599. On observability and cost, Russell Power added a cost_manager that pulls daily provider costs into Finelog #6555 and closed several cross-region read-budget escapes #6692, while Rafal Wojdyla built out the status-page probes panel for synthetic-canary health and worker/job gauges #6565, #6559.

Platform plumbing rounded out the week: Marin's client and server auth were unified onto rigging #6593, Iris's Identity-Aware Proxy ingress moved from a Cloud Run proxy to a shared Google Cloud load balancer #6591, and Calvin Xu added opt-in IAP SSH #6407. The executor was reworked to build steps from lazy typed artifacts #6649. On continuous-integration and dependency upkeep, GitHub Actions runners were upgraded to Node 24 #6648, a stale CPU/vLLM dependency conflict was removed #6646, and the recurring nightshift passes continued their automated cleanup of documentation drift and dead one-off experiments #6680, #6644.

External GitHub activity clustered on the optimizer and GPU bring-up fronts. Kaiyue Wen opened a burst of Muon-variant prototypes — activation-aware Muon #6538, left-preconditioned Muon #6588, a Shampoo-Muon hybrid dubbed “Mudam” #6589, and a curvature-corrected variant #6634 — feeding the same optimizer line the scaling-recipe work is weighing. mcwitt merged #6473, which reserves ad-hoc CoreWeave H100 dev pods through Iris as a GPU analog of dev_tpu, and authored the direct-quantization fp8 dot_general op #6660 the architecture work picked up. Two first-time contributors landed circuit-task work: ma08 shipped surface-form diversity for the PPL string_slicing task in #6537, and wasup-yash asked into the open new-task-families slot #6236.

On Discord, two threads drew discussion beyond the engineering covered above. In #marinfold, Tim O'Donnell posted the sub-project's weekly update: a tuned contact-map model now folds proteins far from its training set where a nearest-neighbor baseline scores zero, evidence that MarinFold has learned structure rather than memorized it, even as that baseline still beats it on near-duplicates. And in open-questions, a community member, Eren, asked for help diagnosing a plateaued training loss on a modified Qwen3-8B he had been pretraining for five days on 8×B200 GPUs.

Four people introduced themselves: a TU Munich particle-physics master's student applying LLM architectures to CERN/ATLAS data, a computational-biologist-turned-ML-engineer, a recent NYU MS graduate who worked on rubric generators for LLM judging, and an MIT EECS postdoc working on AI for healthcare. Joël's hands-on work with the Muon optimizer and value embeddings intersects with this week's optimizer-variant push, and Jacob Silterra's background intersects directly with the MarinFold protein-folding effort. The week's shared reading ran toward data-selection signals and the dynamics of SFT-to-RL post-training, anchored by the OpenThoughts-Agent data-recipe release.