Marin: Week of July 6th summary

The centerpiece run — the 67.1B-total / 2.01B-active “Grug” MoE on TPU v4-2048, tracked here and logged day-to-day on #6044 — kept training through the week, crossing from last week's ~1.47T tokens to step 39,000 (~2.114T, about 21% of the 10.07T horizon) while holding MFU near 18.6% on the resumed resume15k_v2 run, which is still going. At that mark Larry Dial branched off the planned first intermediate cooldown #6811: a short 3,150-step learning-rate decay from the step-39,000 checkpoint that also stretches context 8× (sequence length 8,192 → 65,536, batch cut 8× to 1,024 so tokens-per-step stays 67.1M) and switches to the phase-1 (mix-2) data. The point is to hand mid-training, SFT, RL, and the inference stack a usable intermediate model well before the full 10T checkpoint lands, which is targeted for around August 30; a second intermediate cooldown at 5T is planned for early August. It finished at step 42,149 (33.9h wall-clock) at Paloma macro loss 2.2772, bits-per-byte 0.8242 — down 0.109 (−4.6%) over the ~211B cooldown tokens.

That 2.2772 is the striking number. The run's preregistered stage-1 target — scaling-law predictions filed before the run launched — was 2.269 Paloma macro loss evaluated at seqlen 8,192 on 1,024 sequences, at the 8T-token mark — stage 1 is the first 8T of the 10T run (~80% completion), and its checkpoint is taken before the final learning-rate cooldown, with the model still at ~24% of peak LR there per Larry's own projection. This 2T cut lands at essentially the same number from a checkpoint at only ~2.1T tokens, but the comparison is deliberately not apples-to-apples: the cooldown fully decays LR, switches to the phase-2 mix, and — unlike the seqlen-8,192 preregistration — evaluates at 8× context (65,536). So it doesn't retire the 8T preregistration; it is a strongly favorable early signal. Larry flagged a further wrinkle himself: the code eval is sensitive to the exact eval token count (it is ordered by language), so a different max_eval_batches can make even like-for-like reads not quite apples-to-apples. Will Held noted that Marin's lowest-ever loss is 2.202 from the 32B model and wondered whether the cooldown could catch it in dramatically fewer FLOPs; Larry Dial reckoned probably not but credited the updated data mix for helping a lot on code relative to the earlier Mantis cooldown. The loss-registration path itself was formally closed out this week in #6046, which points back to that same preregistration.

The context extension rode on a YaRN attention-scale probe. Because this model sets disable_long_rope and uses 2,048-token sliding windows, the RoPE-frequency rescale that “real” YaRN adds buys nothing; what matters at 8× longer context is that the attention softmax sharpens, so the fix is a temperature scale on the query–key product (qk_mult). Larry Dial ran a 6-arm, 20-step sweep (all-layers vs long-branch-only, each at scale coefficient 0 / 0.1 / 0.2) and picked all-layers coef=0.1 — qk_mult = 1.5703 — which tied for best final Paloma among the arms. He flagged that the longer context helps evals immediately but costs ~30% in tokens/second from quadratic attention, and that load balancing degrades with only one sequence per chip across 1,024 chips. The qk_mult_long_scale config field added for the sweep was then reverted on the branch, because adding fields to GrugModelConfig shifts the executor hash and would break the pretraining launcher's checkpoint auto-resume — a reminder of how tightly the live run is coupled to code identity.

The week's run incidents were all host and hardware, not the model. A crash at step 31,962 and Iris restart landed a worse TPU topology and permanently slowed steps from 26.1s to 30.0s (+15%); David Hall traced it to topology and is adding topology logging so future slices can be requested deliberately. Restarts from high preemptions and barrier-timeout startup failures on the v4-2048 recurred, and Larry Dial had earlier planned to manually kill the run near 2T for exactly this first cooldown. Meanwhile the inference and export stack for the intermediate checkpoint came together: #7116 (merged) vendored the cooldown launcher and added an 8×H100 BF16 inference regression that loads the step-18000 checkpoint and checks next-token logprobs; #7125 fixed the CI redness it exposed (skip the test when CoreWeave kube-config is absent, and treat pytest's “no tests collected” exit 5 as success); and the still-open #7117 adds a vLLM-compatible BF16 Hugging Face export regression with a whole-tree SHA256 over the 39-shard artifact — the 134GB export verified, though a rerun of the inference assertion hit an NCCL AlltoAll error. With #6811 now closed, Larry Dial has already opened questions for what to do with the eight cooldown checkpoints, from checkpoint averaging to swapping MuonH for Adam.

The July hardware-validation epic came alive this week: #6716 brought the Marin 11B-A1.5B model — a d2048 mixture-of-experts, 24 layers, 64 experts routed top-4, grouped-query attention (GQA) at 4:1, ~1.53B active / 10.6B total params — up on 64×H100 on CoreWeave (cw-us-east-02a, production priority). The point isn't the artifact: the run is deliberately pushed to 500B tokens, well past the model’s compute-optimal point, to shake out the GPU training stack and exercise the hardware ahead of the larger Blackwell runs — as Isaac Hodes put it, "this is more about exercising the hardware instead of delivering an artifact." It has held ~23.8% MFU (~1.37M tokens/s, ~1.53 s/step at batch 512 × seq 4096), with completion expected around July 13–14. One caveat worth keeping in mind: the FLOP counter treats attention as fully quadratic and ignores the 2048-token sliding window, so the reported MFU is slightly optimistic on the attention term.

Alongside the scale-up, Larry Dial ran two d512 compute-optimal chains to check that the GPU stack tracks the established TPU feature-ablation chain. On CoreWeave 8×H100 with datakit data, final Paloma macro loss walked down cleanly with each stack change — pure main 3.7103, +vectorized-map (VMAP) w_gate fix 3.6954, +256 experts 3.6359 — confirming the GPU path behaves as expected. Most of the residual gap to the TPU reference chain traces to data rather than hardware: those TPU ablations run on the nemotron mixture, which lands lower on Paloma at this compute-optimal d512 point.

That data gap opened a useful thread with Will Held. datakit is tuned toward code and math, where Paloma scores comparatively low relative to both the project's goals and Uncheatable-eval; re-scoring the same runs on the shared Uncheatable cache flipped the ranking, with datakit ahead. The read is a natural-language-vs-code trade-off that sharpens as models approach compute-optimal — closer to the token:param frontier the mixture matters more and yields cleaner signal, whereas the deep overtrain had largely saturated Paloma for this model size. (The heavier B200 kernel, fp8, and pipeline-parallelism engineering underpinning these GPU runs lives in the B200 MFU epic #6710.)

Closed. The intermediate cooldown finished — a 3,150-step decay from step 39,000 of the 10T run that landed at 2.277 Paloma macro-loss (bits-per-byte 0.824) from a checkpoint at only ~2.1T tokens — a strongly favorable early read against the 2.269 macro-loss target preregistered for the run’s stage 1 (the 8T-token / 80%-completion mark), though not apples-to-apples: that target is evaluated at seqlen 8,192 and before the final learning-rate cooldown, whereas this 2T cut fully decays LR, switches to the phase-2 mix, and evaluates at 8× context. The full trajectory and context-extension details are covered in the June 67B-A2B hero run entry #6704. The final step-42150 checkpoint was verified and copied to CoreWeave S3 and a v6e inference zone by Romain Yon, so mid-training, SFT, RL, and the inference stack can start against a usable intermediate model well before the full run lands.

This commitment gates the end-of-month ~120B-A8B run: B200 model FLOPs utilization (MFU) must clear 20% before it kicks off on 2×NVL72. It is not there yet, but the gap is closing — the engineering detailed under B200 training MFU & perf reached 17.8% single-node on 8×B200 by wiring Tri Dao’s QuACK/SonicMoE grouped kernel into the MoE experts, with model width (not batch) identified as the live lever #7012. Work continues toward the 20% bar via the remaining fp8 and kernel paths tracked in #6710, which is separately pushing MFU toward the 25% target for the September run.

Benjamin Feuer filled in the post-training cost estimate this week. Costed in hardware FLOPs (the pretraining plan’s currency, so reinforcement learning’s low ~1% aggregate MFU is counted rather than hidden), the pipeline — reinforcement learning (RL) over 10 experts, then multi-teacher on-policy distillation (MOPD) into the student on the Nemotron path — is dominated by the RL leg, whose cost scales with the experts’ hosting footprint. The headline: post-training a small 67B-A2B hero costs roughly 0.25–0.9× its 10T pretraining in HW-FLOPs (about 1.2–3.9× the cheaper 2T-cut), and only ~5–20% of the 120B big run; a ~5× environment speedup divides all of it by five. In literal tokens it is small — on the order of 10⁹–10¹⁰, under 0.5% of a 10T pretrain — though that undercounts, since each post-training token is far costlier to produce #7074.

This commitment is to lock in the data mix, the architecture, and a preregistered loss target for the ~120B-A8B B200 run that kicks off at the end of the month on 2×NVL72. It is still open — no decision was recorded this week — but its inputs are actively in motion: the B200 MFU bar #6706, the post-training cost estimate #7074, and the architecture and data-mix work under the investment epics below. Preregistering the loss target, as was done for the 67B-A2B run’s stage 1 before it launched, is the piece that turns the decision into a testable prediction.

The cross-cluster federation substrate that shipped inert last week went live. Russell Power added the identity, authorization, and trust layers a marin↔CoreWeave rollout needs: every job now carries a submitting_user principal resolved once at submit and inherited across a handoff, a per-cluster allowlist decides who may federate to a given peer, and a parent presents a short-lived aud="federation" Ed25519 token that the receiving cluster verifies against the peer's configured key #7034 — then turned the wiring on so a job submitted to marin or marin-dev can run whole on CoreWeave and be watched from where it was launched #7064, behind one idempotent networking installer that stands up Traefik, cert-manager, and a single IP-locked ingress #7043. Placement moved off the submit path: instead of piling every job onto the first peer that merely could host it, federated jobs now park as QUEUED_HANDOFF and drain on the control tick to peers that actually report free capacity #7108, and endpoints served by a job on a child cluster are mirrored up and reachable through the parent's /proxy, so a GPU serving job pinned to CoreWeave is usable without tunneling to the child #7109. Rolling marin-dev onto the queue immediately surfaced a gate bug — a rebuilt handoff arrives with an empty client_revision_date that the peer's freshness check reads as ancient and rejects forever — fixed by exempting federated handoffs from the CLI-freshness gate #7132, part of the broader cluster-queueing and federated-availability effort #7099. Turning federation on also reshaped logging: the in-process controller relay that tailed each CoreWeave finelog and pushed rows to marin fell steadily behind on busy clusters, so forwarding moved into the finelog server itself and now ships every table — logs plus the iris.worker/iris.task stats — past a durable per-namespace cursor #7091, closing #7078.

A run of canary and multislice incidents got root-caused and closed. The daily Grug multislice smoke found that when the second v6e slice couldn't be scheduled for lack of capacity, slice 0 booted, sat in the Megascale readiness barrier for the full 300s, and crashed the gang #7028; since slice provisioning alone can exceed 300s, the barrier default was raised to an hour #7030. A separate two-slice TPU job hung in distributed init because the TPU path called no-arg jax.distributed.initialize() and bypassed Iris' coordinator wiring #7041; David Hall routed Iris TPU jobs through the registry-backed coordinator with explicit world size and rank #7042. The long-standing coscheduler collision — two gangs dispatched seconds apart landing on identical host sets and racing the JAX coordinator port into a preemption death loop, an estimated ~1,100 v5p host-hours wasted — was finally closed out #5470. On CoreWeave, multi-node NCCL (NVIDIA Collective Communications Library) bootstrap on cw-rno2a found no routable interface #7018 (resolved by the prior week's exclude-list fix #6941), and the GPU canary's cuDNN precedence-restore aborted the job when an --offline reinstall missed the uv cache, dropped in #7031 with CUDA 13 cuDNN precedence restored in #6947. The 67B hero run hit the same class of gremlin operationally: Russell Power traced a stuck restart to a crashed server failing to rebind port 8431 because the libtpu socket flags weren't set, and later a finelog deploy briefly broke the job display before being fixed the same day, while Larry Dial chased DEADLINE_EXCEEDED barrier timeouts and preemption spikes on the v4-2048.

The Executor is gone. Russell Power's long-running migration off the eager, import-time content-addressed DAG landed #6649 — a net roughly −14k lines across 276 files — replacing ExecutorStep module globals with lazy, explicitly versioned ArtifactSteps built at run time, so a code edit no longer silently forks an output path and the important training decisions stop hiding behind default_* wrappers. On the new path, StepRunner persists full artifact records (name, deps, config, provenance) #6901, the loss-era experiments were revived as ArtifactSteps #7029, and the build commit and launch identity are stamped into the job env so remote-run records get real provenance #7000, #6902. The retirement also settled a deadlock: running a Levanter step as a single-program multiple-data (SPMD) job under srun hung because the per-step distributed_lock let exactly one rank enter the step body while the others spun retrying, so the collective mesh never formed #7080; with the Executor retired the practical fix is to bypass StepRunner in SPMD ranks, which Benjamin Feuer confirmed unblocked a 16-rank dense SFT, and StepRunner now logs a loud warning when it starts on a secondary Iris task #7082.

Two cross-cutting cleanups tightened the rest of the plumbing. Server auth was unified onto one declarative, deny-by-default stack: EdDSA JWTs minted per cluster and verified statelessly against a published JWKS, a distinct audience per surface so a token for one plane is rejected at another, and the plaintext static-token provider removed at the source #6948 — closing the leak that had shipped controller secrets in plaintext ConfigMaps and GCE startup metadata #6873, alongside an infra permissions/auth audit #6942 and off-cluster headless service-account IAP access #6978. A deploy footgun surfaced along the way: a stray IRIS_SIGNING_KEY left in an operator's shell outranked the pinned Secret Manager reference and silently became the cluster's signing identity, so every handoff and log relay would fail authentication with nothing pointing back at the shell #7081. On CI, the unified import-driven unit workflow #6833 became the single unit-test gate: it now selects tests from the pull request's own diff at module granularity #7068 and unties the __init__ re-export hubs that let a one-line submodule change select nearly the whole levanter suite #7075. Underneath, file and object handling converged onto rigging.filesystem.StoragePath #6994, #6981 (the module split into a package first #7093); finelog gained bounded compaction merge memory #7056 and a fix for a compaction wedge on Arrow's 2³¹ offset overflow #7049; and on the data plane Zephyr grew more robust on CoreWeave object storage — parquet written through native pyarrow filesystems #7100 and its worker-gang cumulative failure budget raised from 10 to 200 so a soak isn't torn down by a handful of preemptions #7122 — while datakit picked up finite S3 timeouts to fix a wedged CoreWeave commit hang #7017.

This was a consolidation week for the SFT-data area rather than a shipping one. The long-running top-level synthetic-data tracker #3192 — open since March and covering which open datasets to aggregate, which to generate, and which to relabel with better teachers — was closed on July 7, with Will Held having asked to close and split it into successors with proper definitions of done and Isaac Hodes noting a related issue will likely be reopened later. The epic itself carried a single note this week: on its own thread Isaac Hodes relayed Benjamin Feuer’s read on data sufficiency — “quite comfortable about this, I think we have plenty.” The practical center of gravity moved to two concrete workstreams: making SFT on Levanter trustworthy, and keeping the agentic-trace supply flowing.

The week’s one merged deliverable was the SFT backend itself. #6839 added axolotl as a first-class SFT backend for OpenThoughts-Agent, built on a marin-community/axolotl fork and validated end to end on a GH200 (aarch64) host. The motivation is jinja-as-ground-truth SFT — train and serve on a single chat template — which closes the “template split at save time” footgun that produced 0% SWE-bench out-of-distribution results through the earlier LLaMA-Factory save path. The fork ports a delphi.jinja chat template (Llama-3 turn format plus reasoning and tool-call tokens) and three plugins, of which the load-bearing one embeds the chat template into tokenizer_config.json across every per-checkpoint directory the naive save flag misses. The Delphi masking canary — assistant answer plus <|start_think|>…<|end_think|> reasoning masked correctly on a real Delphi checkpoint — passed, and the patches were merged to marin-community/axolotl main, with the OpenThoughts-Agent submodule pin left valid.

The larger new thread is Levanter SFT reliability, tracked under Russell Power’s #7045 (“SFT with confidence”), whose goal is token-identical-or-defensible SFT behavior versus a Transformers-based toolkit on chat templates and tool-call results. Benjamin Feuer filed a precise cluster of gap reports against it. #7130 is the most consequential: train_lm has no config-level epoch cap — the EpochDataset primitive exists but is never wired in — so 1-epoch SFT is a hand-computed, packing-dependent step count that is easy to get ~6× wrong, which bit the Delphi 1e22 SFT parity run when 34,722 steps turned out to be ~6.5 packed epochs and near-zero loss was the symptom. #7086 reports that chat-format SFT cannot run unpacked at all — pack="pad" raises NotImplementedError, so there is no padded one-sample-per-example escape hatch — and its companion #7087 flags the correctness fragility that follows: packed SFT replaces a trivial != pad mask with coupled segment_ids block-diagonal attention and {% generation %} loss spans, exactly where cross-contamination bugs would hide, and asks for a regression test asserting no attention leaks across segment boundaries and loss only on assistant spans. #7088 requests a DenseMixer-style dense-forward router gradient for MoE SFT: the sparse forward computes only selected experts, so a non-selected expert’s router-logit gradient is exactly zero, and Feuer argues that re-routing among already-specialized experts — not rewriting them — is the dominant available lever in post-training, making that withheld counterfactual signal most valuable precisely here.

On the supply side, the agentic-trace index #6191 kept growing: Benjamin Feuer posted consolidated metrics for the wound-down Qwen3.5-122B @ 32k campaign (49 datasets, 258,275 trials, now carrying per-trace mean turns, tokens, and Harbor verifier reward) and a new Qwen3.5-122B @ 131k opencode cohort (22 datasets, ~41k trials). That index became the onboarding spine for the data area’s new members: in data-curation Feuer pointed Franziska Weindel at the data-mixing channel for midtraining data and TaskTrove plus #6191 for RL data, asking her to map out which capabilities lack good datasets — noting the recent MiniMax tech report lists knowledge domains essentially unrepresented in TaskTrove. The framing echoes Will Held’s point in the midtraining intro thread that naturally-occurring agent traces remain the biggest shortage in the open-data ecosystem, and his worry that too many teacher rollouts risk entropy collapse — pushing him toward harvesting “natural” traces from artifacts like PR reviews and follow-up commits.

The week’s biggest mixture-optimization result — an embedding-based surrogate (mixing_via_embeddings, #6969) that re-featurizes a mixture by the content it induces and prices a genuinely never-swept bucket — is covered under Data-selection diagnostics below.

On tokenizers, Russell Power closed the tokenizer-research experiment #6796 with a grug-MoE bake-off built around feBPB — a FLOP-equivalent BPB that prices each tokenizer's serving cost at the ~250B-total / ~20B-active deployment MoE, not the small proxy, since the vocab-dependent LM head is 30–50% of proxy FLOPs but only ~2–3% at deployment. The SuperBPE soak first read as 5 of 7 arms beating the Llama-3 baseline at a common 6e19 budget, but a same-day correction showed the win is budget-dependent and reverses at scale: the 128k SuperBPE arms start level with Llama-3 and fall progressively further behind as budget grows (the gap roughly doubling out to 2.6e20), leaving only the trained 64k-fixed arm ahead and only at low budget. In the tokenizer channel, Will Held argued for adopting the GPT-OSS tokenizer to ease pass@K forecasting and on-policy distillation (OPD) from strong teachers, while Rohith Kuditipudi noted ~98% of Llama tokens have direct Qwen counterparts for regular text (math and code aside). Yiyuan Li's Marin-vs-Qwen3 coverage comparison #5842 closed as completed. Pranshu Chaturvedi's digit-variant sweep #6571 stays in draft after he found a bug — the wrong model size was passed to the digit arms, handing them an unfair ~0.03 bpb edge — forcing a rerun of the matched 16-cell comparison.

The 10T June-run datamix #6045 closed as completed on July 7, its definition of done met. But Michael Ryan's weekly sync on #2351 delivered a sobering reality check on curation: at compute-matched DCLM CORE eval, Marin's high_quality curation is not Pareto-dominant even at the 3k scale — the earlier apparent win relied on scale and had been hidden by scoring only at each run's minimum uncheatable-eval val-loss point. A gap analysis across the uncheatable and OLMo subcomponents put Marin strong on code, on par at math and Wikipedia, but with real gaps in fiction, QA benchmarks, and news. To attack this he has labelled a 935-document dev set (keep / weak-keep / weak-drop / drop, plus must-keep and tricky edge cases), drawn by cross-pipeline disagreement and stratified by register so both kept and dropped documents appear in every register.

Two mixture threads moved into scoping for the next runs. Calvin Xu opened #6972 to scale the best one-phase OLMix KL settings from the 3e18 sweep, which showed the best KL is target-specific — KL=0.1 for uncheatable BPB, KL=0.005 for OLMoBaseEval Table-9 macro BPB — launching both winners at 2e19, 3e20, and 1e21 with native Table-9 eval attached. And Jeff Hammerbacher filed #7128, a proposal for a ~40–60B-token medical midtraining corpus as a GAP-Replay analog assembled mostly from in-catalog, already-tokenized sources (Common Pile PMC and pubmed, the biomedical peS2o slice, EPFL clinical guidelines) — companion to the medical SFT pilot #7127, and grounded in the Delphi finding that midtraining supplies capability while SFT elicits it.

This week the epic's mixture-optimization side got a sharp problem statement and a validated fix. #7067, filed from Rafal Wojdyla's agent session, names the structural weakness in how the team stores data-mixing evidence: it is bucket-indexed. A mixture swarm (the qsplit240 sweep is roughly 240 mixtures times 2 scales, hundreds of proxy training runs) learns "weight on bucket #7 to loss," but bucket #7 is just a name and the surrogate has no representation of what the bucket contains. So any change to the bucket layer silently strands the evidence: a dedup pass, a quality relabel, a source refresh, or any datakit re-partition invalidates the learned coefficients, new data has no coordinates to be priced at, and swarms run under different bucket eras cannot be pooled into one training set. Because the bucket layer changes far more often than the corpus, mixture decisions and data curation run on different cadences, and every gap between them is currently bridged by heuristics. What a fix has to deliver, per the issue: sweep reuse under redefinition, cheap pricing of genuinely new data, and pooling across bucket eras.

The proposed fix, #6969 (mixing-via-embeddings), re-featurizes a mixture as the token-weighted histogram it induces over a frozen embedding codebook (h = V·w) and learns the surrogate over that content distribution instead of over bucket names, so bucket identity is consumed in featurization, old sweeps re-featurize under any future bucketing, and a new dataset is priced from one embedding pass plus a CPU refit. The validation is the notable part, and it was preregistered end to end: retrodictive gates on the existing swarm with no new proxy training (an information audit, a content-to-domain-value leave-one-domain-out gate, and held-out-dose retrodiction, each judged against shuffled and matched-random controls under a pre-committed two-basis kill rule) all passed, and then a live pre-registered test priced a genuinely never-swept bucket, dolma_starcoder, proposed a mixture, and realized 0.9410 uncheatable-eval bits per byte (BPB) against olmix-reuse 0.9495, token-proportional 0.9759, and the sweep's own best-ever run 0.9554 (a −0.0145 margin the significance check put at roughly 8σ versus measured 300M repeat noise). The whole integration cost one embedding job plus minutes of CPU and three validation runs, no re-sweep. The honest caveat is preregistered too: the optimized point was optimistic by about 0.031 BPB (winner's curse), so a trust-region / lower-confidence-bound proposer is the flagged next step. Will Held pushed the swarm's sweep pointers to a public HuggingFace dataset so the regression work is open to anyone.

On #6096, last week's headline result — a base model's BPB on verified math-reasoning traces predicts its post-RL math accuracy — turned this week toward the loop the epic exists to close. Isaac Hodes raised the pre-to-post feedback loop for the team agenda, and Rafal Wojdyla distilled it to two questions: what should pretraining experiments hill-climb to make post-training easier, and how should post-training results feed back into pretraining data choices. Will Held pushed on the first in a scaling-laws thread, sketching how "compute optimal" should be redefined for an RL world: rather than fitting the hero-run scaling ladder to pretraining loss, fit it to pass@256 on agent-relevant skills, on the simplifying assumption that a pretrained model's pass@256 converts to pass@1 under enough RL — which would also let overtraining be priced against the RL-cost savings it buys (for Delphi, 5x overtraining costs only about 3x more pretraining compute, likely worth a 5x cheaper RL recipe). No experiment has closed the loop yet, but the question this epic was created to answer is now concretely on the table.

With the May Recipe merged to main and the grouped-query-attention (GQA) versus Multi-head Latent Attention call largely settled last week, this epic turned to the question the recipe leaves open: getting MoE training fast on GPU hardware ahead of the September large run. David Hall opened a GPU Upskilling Master Plan #6998 laying out four work streams — an XXB-A2B target on Hopper (MFU ≥ 20–25% on 128–256 H100s), a Blackwell A8B smoketest, MXFP8 training on Blackwell, and a fused expert-parallel (EP) MoE kernel — and naming the Hopper obstacles plainly: high sparsity plus Adam taxes high-bandwidth memory, the MoE block is slow, and Muon carries high overhead at high sparsity, with pipeline parallelism (PP) the real fix. Larry Dial's GPU MFU Learning Path #6979 built the Hopper story up from scratch on a single 8×H100 node: an lm_head-only cross-entropy benchmark at 50.3% MFU (Liger chunked CE, one chunk per device), a full dense d2560/26-layer model at 56.2% (FlashAttention-4 CUTLASS, ZeRO-1, scan-over-layers with full rematerialization), and then an itemized progression as each correctness feature that makes it a real trainable MoE is switched on — from a 34.4% throughput probe down to 26.7% for the full model.py, with real Muon's Newton-Schulz orthogonalization the single biggest give-back at ~3.7 points and each norm only ~0.4–1 point. He also caught that an earlier step-10 out-of-memory was the per-parameter-norm watch callback unstacking all 26 scanned layers, not a genuine memory ceiling.

The multinode result was the headline: model.py weak-scales essentially flat from 1 to 8 nodes — 26.7% at 8 GPUs to 26.5% at 64 H100s and global batch 1024 — because replica/DDP hides the once-per-step gradient all-reduce over the data-center network while keeping the frequent Fully Sharded Data Parallel (FSDP) all-gathers on intra-node NVLink, whereas sharding the model across nodes costs about 4 points. Dropping the total expert count from 256 to 64 (still top-4) lets the full model fit one node, which Larry Dial proposed as the H100 “working copy,” scaling total experts up only as a model shape and hardware allow.

Will Held carried the same recipe onto Blackwell in #7012, wiring Tri Dao’s QuACK SM100 grouped general matrix multiply (GEMM) — the SonicMoE kernel — into the expert MLPs to lift single-node 8×B200 MFU to 17.8%; that kernel work, plus the fp8 and pipeline-parallel threads, is covered in full under B200 training MFU & perf below.

The master plan's named real fix is pipeline parallelism, and two threads advanced it. David Hall's fused EP kernel #6597 — in the spirit of SonicMoE but handling EP over NVLink — is, he noted in the gpu channel, showing promise at getting genuine overlap with the all-gathers where nothing else had; and on July 10 he flagged that an agent was making good progress on JaxPP pipeline-parallel MoE training #7024, declaring “Hopper is back on the menu.” On the GQA path the recipe now runs, Romain Yon merged #6965 (closing #6964) to align Grug MoE's XSA value correction to the actual attention-output partition spec, since GPU attention backends can expose either head-sharded or head-dim-sharded outputs. Kaiyue Wen opened a speedrun PR #7118 adding two error-aware Muon feedback policies; in a single-seed 40-run sweep, Hessian-corrected Muon beat a fresh Muon control at four of five learning rates by a mean −0.000587 C4-en bits-per-byte — a single observation, not yet a replicated optimizer win. Finally, the golem kept retiring superseded MoE experiments with managed TL;DR summaries — the MuonH and Muon/AdamH optimizer studies #5596, #5167, and #4034, the tanh(pos/10) loss-reweighting test #5306, the expert-count and 3e18-hparam sweeps #4030 and #4018, the take-until-null routing follow-up #2851, and the 8k-sequence-length experiment #6277, which had found 8k gradient trajectories essentially identical to 4k at 2.5% lower throughput because the nemotron mix is overwhelmingly short.

Epic #6710 is the July milestone’s ongoing GPU-performance investment: push NVL72 (B200) model FLOPs utilization (MFU) toward the 25% target set for the September ~512B-A16B run, with the nearer 20% bar for the end-of-month ~120B-A8B run tracked as commitment #6706. The 64×H100 d2048 stack-validation run that cleared ~23.8% MFU this week is covered as a hero run under Hardware and Loss Validation on H100s #6716; the work here is the Blackwell push. On a single 8×B200 node, Will Held reproduced the H100 config at 12.5% MFU, established that batch is a dead lever (the workload is arithmetic-intensity-starved, not batch-starved) while width is live (16.2% at d5120), then wired Tri Dao’s QuACK SM100 grouped general matrix multiply (GEMM) — the SonicMoE kernel — into the expert MLPs via a torch-free CUTLASS shim, lifting whole-model 8×B200 MFU to 17.8% at d5120 (14.9% at d2560), with the QuACK gated GEMM alone at ~1,175 TFLOP/s, ~52% of peak #7012.

The fp8 compute lane crossed from kernels into a real training path. Building on last week's grouped-GEMM work — the preregistered experiment #6824, the Fp8RaggedDotOp in #6880, its backward-pass tuning #6930, and the wire-format spike #6911 — Matt Wittmann's draft #7079 threads the stateful fp8 ragged-dot ops through the grug Mixture-of-Experts (MoE) expert MLP and adds opt-in fp8 over-the-wire collectives — E4M3 forward and E5M2 backward on the permutation legs only, reductions staying bf16 since decomposing the reduce-scatter into an fp8 all-to-all measured 0.885× against NCCL's hierarchical path — all behind a single GrugModelConfig.fp8 switch with the matching train-step state handling for delayed scaling. At the production d2560/E256/K4 shape the full MoE layer measures 1.53× vs bf16 at 2-node expert-parallelism (EP) 16 over InfiniBand on the ring backend (1.65× on all-to-all). Defaults stay bf16; the gate before flipping any default is loss-curve validation on a real trajectory, tracked in #6699. With the kernels proven out, Isaac Hodes closed the original “try out fp8 training” issue #6048 as done and the June H100x8 MoE MFU tracker #4302, which had already been superseded by the current GPU MFU threads.

The largest single effort was pipeline parallelism. David Hall's #7024 stood up JaxPP pipeline-parallel (PP) MoE training across four 8×H100 nodes on RNO2A, pivoting to an explicit jaxpp.experimental.mpmd path with stage-local weights and optimizer state after the automatic schedule stayed blocked on JaxPP sharding inference. The best measured point — explicit std_1f1b, batch 8192 / 256 microbatches, four six-layer stages, ring EP, CuTe FA4, eight-warp Pallas-Triton grouped GEMM — reached mean MFU 18.26 at ~414k tokens/s, saturating pipeline occupancy about 1.74 points below the 20 bar. What followed was a disciplined week of negative results, each gated on H100 A/B evidence before scaling: a latency-hiding scheduler flag (+0.4%, within noise), a pure-XLA output-oriented ring combine (−52%), explicit Triton routing kernels (performance-neutral), transfer-priority task ordering (−2.9%), an input-gradient-first backward split (−29%), and an exact two-chunk bulk ring (−14% fwd+bwd) all failed to close the gap. The most promising lever — an EP-local QuACK/Sonic grouped-MLP adapter — cleared the performance gate (1.16× forward, 1.108× fwd+bwd) but failed output parity; a standalone exact-shape repro pinned the ~2.6% mismatch to QuACK's fused approximate SwiGLU (fast exp2 and reciprocal), turning it into a semantic-policy question rather than a bug. That concurrent-QuACK path also had its own blocker root-caused in #7110: a shared TVM-FFI handler hangs under concurrent multi-device calls at Fully Sharded Data Parallel (FSDP) 3, minimally reproduced with three experts on 8×8 matrices, and fixed with a per-handler mutex validated at 32-H100 scale (though at 13.96 MFU it is not itself competitive with ring EP). David Hall told the GPU channel the JaxPP progress means “Hopper is back on the menu”.

The fused-kernel research advanced on both architectures. David Hall's Pallas Mosaic GPU source-push MoE under #6597 — which he described in the GPU channel as pushing tokens device-to-device in SMEM-sized chunks to get real overlap with the all-gathers — now completes the exact EP8 target forward and backward through one JAX-native custom VJP with consistent shardings (integrated fwd+bwd at 242.8 ms, ~31.9 TFLOP/s per rank), after a long sequence of CTA-schedule redesigns, with serialized operand movement around WGMMA named as the remaining bottleneck. The Blackwell sibling #6933 cleared its 300 TFLOP/s-per-rank forward target once the staged MLP call is outer-JIT compiled — 666 useful TFLOP/s per rank steady-state, the earlier ~155 figure having been wrapper and staging overhead — and its backend is being integrated in draft PR #6970. A parallel transport study #7114 evaluated a CuTe/NVSHMEM push/pull transport as an alternative to Mosaic source-push: symmetric allocation, device-only push/pull, zero-copy JAX aliasing, and warp put_signal at 5.30 µs / 1.16 GB/s per PE all proved out, but the decision was to keep Mosaic — transport overlap more than halves concurrent GEMM throughput and the JAX/XLA custom-call path will not compile against the tested CUTLASS/NVSHMEM stack. On the attention plumbing, @timodonnell found in #7013 that levanter's GPU default attention backend was NVTE (Transformer Engine), which no marin environment ships — the gpu extra installs flash-attn-4 instead — so every dense-LM GPU run silently fell back to an unfused reference path that also OOMs at sequence 8192; his fix #7015 adds an FA4 backend wired to the already-shipped FA4/CuTe kernels and makes it the GPU default, 3.4× faster than the fallback (11.3 vs 38.9 ms/iter on an H100). Two supporting PRs merged early in the week: #6910 keeps FA4 metadata shard-local before entering the shard_map to avoid a global-metadata mismatch during lowering, and #6872 mirrors grug profiler captures to /profiler rather than uploading them as W&B artifacts. The older single-shape d2560 push #6367 stayed open with its best clean 8-node no-tensor-parallel smoke still at ~1.8 MFU, effectively handed off to the pipeline-parallel path in #7024.

This epic is a forward-looking investment area #6709, opened June 26 to pick up after the July commitment and hero-run work: speed up rollout generation so reinforcement learning (RL) is not inference-bound, focused on H100 GPUs where most of this year's RL will run. Its stated first task is to measure and document the current serving baseline before a target is agreed. No sub-issues, PRs, or design docs landed against it this week — the concrete activity was a single serving-reliability bug and a set of Discord signals about where RL step time is actually going.

The bug, #6983, is a good illustration of why this epic exists. Will Held filed it after byte-level ensemble decoding — tens of thousands of one-token completions with 128 logprobs — drove a brokered vLLM setup (local proxy → broker actor → inference worker → vLLM) into a state where clients timed out while the engine sat idle and answered direct requests in milliseconds. The initial theory was a lease/slot-leak death spiral in the worker↔broker accounting; a review pass instead pointed at the worker→vLLM httpx connection pool exhausting under sustained load. Will Held built an isolated reproducer that wedged reliably in about 24 minutes. But Romain Yon's instrumented rerun overturned the diagnosis: splitting the failure bucket into 4xx/5xx showed the zero-success plateau was really fast vLLM HTTP 400s once the growing prompt crossed marin-8b-base's true 4096-token position limit (an Out of range float values are not JSON compliant: nan body, masked by VLLM_ALLOW_LONG_MAX_MODEL_LEN plus an oversized max_model_len), with the broker/proxy path clean (dropped_responses=0, rejected_requests=0). Will Held withdrew the reproduction claim and hardened the harness — split counters, first-bad-response diagnostics, a context cap, and a distinct CONFOUNDED exit when a plateau is 4xx-dominated rather than timeout-dominated. Net: the original field wedge (brokered clients stalling for over 30 minutes against an idle, healthy engine, no task restarts) remains genuine but unreproduced, and the silent NaN→400 behavior past max position was flagged as possibly deserving its own issue.

The demand signal for the epic showed up in the reinforcement-learning channel. Benjamin Feuer reported that 131k-context RL on Mixture-of-Experts models (MarinSkyRL) is now plumbed on CoreWeave: Qwen3-Coder-30B-A3B and Qwen3.6-35B-A3B each fit on 64×H100 (8 nodes) at roughly 2.5 hours per step — “slower than I would like but not insane” — with about 10M tokens generated per step, and the full sharding-geometry feasibility grid and four root-caused bugs written up in #7052. That generation cost is exactly the lever this epic is meant to pull. Relatedly, on a question about whether vLLM's native RL APIs would be required to RL the incoming GrugMoE model, Benjamin Feuer answered that they are not needed short-term. Quiet week for the epic itself — it is still at the baseline-establishing stage — but the serving stack it targets is now under real RL load.

This Investment epic carries the framework-selection question — evaluate candidate reinforcement-learning frameworks, close the parity gaps in checklist #6341, and settle which one Marin standardizes post-training on — and this week the strongest datapoint from last week's summary got substantially revised. Last week the cross-framework reproduction of the MarinSkyRL rlvr7500_w1 Delphi run in Together AI's xorl #6915 read as matching and exceeding the reference math lifts (+12.3 MATH500 / +8.8 gsm8k) on a fraction of the hardware. Russell Power closed the issue on July 6 as not immediately actionable, suggesting any worthwhile features be raised against MarinSkyRL directly. Then on July 10 Ashwinee Panda posted a fidelity-audit correction: the headline numbers were not a same-ruler comparison — the original run used Muon rather than AdamW, a different prompt/grader reward pair, and a more permissive evaluation grader. After closing the auditable gaps (byte-level prompt rendering, a token-id-identical dataset of 7,498 common rows, AdamW with β=(0.9,0.999) and FP32 moments, the reference grader and greedy-evaluation convention), the parity run lands at roughly +0.8 to +1.4 MATH500 against the reference's +8.4, and +7.6 to +8.4 gsm8k against +4.4, with AIME regressing under the parity contract. The corrected read: total MATH500+gsm8k transfer is broadly comparable (~+9–10 versus +12.8) but allocated differently — the xorl run is gsm8k-heavy where MarinSkyRL was MATH500-heavy.

The result that survives the audit intact is a train/serve correctness one: all 141 uninterrupted parity updates ran with behavior_k3 = 0.0, reproducing the reference's start, climb, and plateau, which Ashwinee Panda was careful to frame as a stability and on-policy-fidelity property rather than an explanation for better held-out evaluation. The earlier apparent gains remain confounded across five candidate causes — Muon versus AdamW; normalizing over retained non-zero-advantage records instead of keeping zero-advantage records in the denominator (an approximately 2.5× larger effective late-run step size); β₂=0.95 with BF16 moments versus 0.999 with FP32; independent per-step draws instead of epoch ordering; and the permissive prompt/grader pair — none of which has isolated causal evidence yet. The clean next step named is a one-variable-at-a-time ablation series holding the parity prompt, reward, and evaluation contract fixed.

On the in-house side, MarinSkyRL kept maturing at scale. Benjamin Feuer announced in the reinforcement-learning channel that 131k-context RL on Mixture-of-Experts models is now plumbed on CoreWeave, backed by #7052 — a step-time feasibility grid at 131k across expert-parallelism (EP), Fully Sharded Data Parallel (FSDP), and context-parallelism (CP) geometries for Qwen3-Coder-30B-A3B and Qwen3.6-35B-A3B on 8×H100 nodes. Getting a clean number out of each geometry surfaced and root-caused four independent bugs (an engine-readiness init race, a memory ceiling, a CP>1 kernel gap, and a storage-migration breakage), pinning the feasible geometries and mapping two infeasibility boundaries (EP16 cross-node all-to-all latency, and 35B-EP8 activation memory). The 30B models fit on 64×H100 at ~2.5 h/step (about 10M generated tokens per step), and Qwen3 Next 80B A3B — the closest public analog to the incoming Marin 67B-A2B — is expected to need 96–128×H100. Benjamin Feuer judged 131k to be about the hardware limit without additional cleverness, but likely sufficient for this year's model slate.

Two smaller framework threads: Romain Yon asked whether the MarinSkyRL setup will need vLLM's new native RL APIs to RL the GrugMoE model short-term, and Benjamin Feuer's answer was no, not short term — with Romain Yon noting SkyRL is an early adopter of those APIs. The evolving framework guide #6162 and parity checklist #6341 saw no new edits inside the window, and Tai Vu's objective-runtime trainer refactor #4766, brought under this epic last week, remains in review with no new activity since June 30.

#6991 is a careful negative-to-positive result from Will Held: running the SimpleRL-Zoo zero-RL recipe — Group Relative Policy Optimization (GRPO) with no supervised warm-up — on the compute-optimal Marin Delphi-25B (delphi-1e23-25Bparams-628Btokens, Qwen3 architecture) over hard competition math (MATH levels 3–5). The headline is that on a weak, un-instruction-tuned base the prompt and output format, not the RL algorithm, are the dominant lever. Plain-prompt RL collapses outright — reward sits flat near 1%, then the model length-hacks to the token cap and degenerates into multilingual token salad at both 1.4B and 25B — because a ~1% reward gives GRPO nothing to bootstrap from. GEPA prompt optimization instead surfaces whatever latent ability already exists (25B climbs 4.2%→8.3% on gsm8k+MATH-500; the 1.4B base stays 0%→0%, since there is nothing to elicit), underscoring that capability must precede either prompting or RL.

The recipe only starts to work after fixing two silent confounds Held traced. First, the recipe's default rollout temperature of 1.0 is a dead zone for a weak base: per-rollout pass-rate collapses above ~0.5, and at 1.0 it sits at the ~1% no-signal floor where nearly every GRPO group is all-wrong and yields zero advantage; dropping to temp 0.5 roughly triples the signal while keeping enough sampling diversity. Second, <|im_end|> is not an EOS token for the Llama-3 tokenizer, so rollout generation never stops there and the model rambles into hallucinated <|im_start|> turns that the verifier then grades as garbage. With few-shot prompting, temp 0.5, and the stop-token fix, GRPO climbs from ~4% to ~29% train reward over 20 epochs and MATH-500 shows real transfer (5.2%→~17%), though it overfits past ~step 70 and AIME-24 stays pinned at 0%. The lesson: the recipe works mechanically once format, temperature, and stop tokens are right, but can't manufacture competence — the real levers are a stronger base and/or verifier-augmented training. Marin's triage bot pinpointed the one concrete code hook, get_stop_tokens(model_type) in rl_experiment_utils.py, which keys stop strings purely on model_type — exactly the mismatch that silently bites a Qwen3-arch model carrying a Llama-style tokenizer. The tuned checkpoint is up as WillHeld/Delphi-25B-SimpleRL-Math.

Alongside the experiment, RL-data curation itself got organized. Benjamin Feuer split the OT-Next data effort into midtraining and RL tracks, naming open-thoughts/TaskTrove as the canonical RL-data source and pointing new contributors at #6191 (agentic trace datasets benchmarked with Qwen 3.5 under the Terminus-2 harness); he also flagged that the recent MiniMax tech report leans on knowledge domains essentially unrepresented in TaskTrove, framing the gap-filling agenda. Bhavishya proposed a concrete first step: applying Test-Time Curricula for Targeted RL — the SIFT method, which embeds a target set and a training set and then selects the training points collectively closest to the target — as a mini proof-of-concept on an open terminal-bench dataset before extending to Snorkel's data. This picks up directly from last week's close-out of the Delphi reasoning-RL scaling study #6279, shifting the epic from finished ablations toward sourcing the data the next round of RL runs will need.

Coming off last week's open of the July inference epic #6867, the GPU sibling issue #6042 closed when Isaac Hodes logged that eight-way expert-parallel (EP8) serving of a real GrugMoE checkpoint had passed on CoreWeave/H100, moving the remaining full-size and reinforcement-learning-grade work onto sub-issues #6869 and #6870. The TPU serving stack, meanwhile, spent the week consolidating its fork pins: #7025 advanced the vllm and tpu-inference fork pins to the landed marin-community main SHAs, and #7094 pinned the fork build so TPU builds skip the default Rust artifacts (vllm-rs and the Rust tool parser) while keeping VLLM_REQUIRE_RUST_FRONTEND=1 as an opt-in. Rohith Kuditipudi had flagged in code-talk that he had patches to overlay onto the two forks, and Russell Power opened #7097 to synchronize them so Grug only needs a single set of patches. The churn was not free — Rohith Kuditipudi reported in infra that spinning up many vLLM workers in parallel after the pin bump hit uv git-lock timeouts and a missing zephyr module on the workers.

A concrete TPU gap surfaced in #7085: ragged paged attention v3 (RPA v3) raises NotImplementedError: Unsupported tpu_version=4 on v4-8 workers, so Rohith Kuditipudi could not run inference there at all. He noted the v3 tuned-block-sizes table already carries a conservative v4 fallback, so the minimal fix is adding a case 4 to the default heuristic; Romain Yon relayed it into code-talk. Two regression harnesses landed to lock down correctness against the actual 67B checkpoint. #7116 vendored the June TPU 67B cooldown launcher plus an 8×H100 BF16 inference test that loads step-18000 from object storage and asserts the next token after “The United States Of” is “ America” on all eight devices, with a top-25 logprob check against a golden. #7117 adds the export side: it loads the same checkpoint, applies the pending QB router bias, and writes a directly vLLM-compatible BF16 Hugging Face artifact verified by a whole-tree SHA256 over its 39 shards — the full 134 GB export passed, though two reruns of the existing inference assertion hit an NCCL ncclAlltoAll error on the JIT path. Larry Dial confirmed the cooled-down checkpoint the tests target in #6811.

On GPUs, the serve path advanced on two fronts. #7111 asked for a way to make marin-serve --gpu actually boot vLLM, since the TPU vllm extra conflicts with gpu and stock PyPI vLLM's torch pin collided with Marin's torch 2.11 / CUDA 13 stack. Weaverbot, at Russell Power's direction, opened #7133, which sidesteps resolution entirely by provisioning CUDA vLLM per-job in a throwaway uv-tool env (uvx --from vllm[runai]==0.25.0 --torch-backend cu128) so vLLM's torch/CUDA tree never enters uv.lock; along the way it found the issue's torch==2.7.0 premise stale, since stock vLLM 0.25.0 now targets torch 2.11 / CUDA 13. It still needs a real H100 boot test. Separately, #7106 reported that marin-serve fails cryptically when run outside a workspace checkout — an empty bundle makes the container's uv sync die with “No pyproject.toml found” — and #7107 fixes it by locating the checkout root by walking up from both the current directory and the CLI's own install location. Russell Power pushed back on whether marin-serve could instead run vLLM from its packaged library; the agent explained the TPU serving stack is a source build from Marin's own vllm/tpu-inference forks rather than a package install, so that route would require the fork config to travel with the published wheel — a separate packaging change.

Will Held added inference-side plumbing for logit mixing in #7113: a RunningModel that serves a logit-mixed subset of the OpenAI completions API over two brokered vLLM systems (a teacher and a student), querying both per token and sampling from the combined top-candidate distribution while leaving the shared broker, worker, and proxy code untouched. The epic's stretch goal — inference fast enough on GPUs for RL rollouts, tracked in #6870 — stayed in view too: Romain Yon asked in reinforcement-learning whether the MarinSkyRL setup will need vLLM's native RL APIs to roll out the incoming GrugMoE model in the short term, which is exactly the “fast enough on GPUs” question the epic exists to answer.

The two write-ups this epic banked last week were formally closed this week, both as reference rather than work items. Russell Power closed #6958 — the end-to-end validation of agentic software-engineering evals on Iris v6e TPUs — and #6877, OpenThoughts-Agent's unified Harbor-eval entrypoint shared for reference, both as “not directly actionable,” and raised the open meta-question of where such documentation issues should live long-term (a wiki or discussion board). Before #6958 closed, Benjamin Feuer attached the side-by-side parity traces: the GLM-4.7-swesmith base model scored 0.240 (72/300 resolved) on swebench-verified-random-100 on a preemptible v6e against the H100 reference's 0.237 (71/300), with the only harness exception being legitimate agent step-budget timeouts and zero infrastructure errors, and both trace datasets published on Hugging Face. In the evals channel, Romain Yon confirmed the coordination loop had closed via OpenThoughts-Agent PR #33, and noted that re-implementing that eval functionality by replacing Marin evals sits in the backlog behind the grug work.

The actionable follow-up is new this week: Russell Power filed #7044 to add CI for Marin's harbor and evalchemy forks, which today can silently rot — harbor is a rev-pinned git dependency and evalchemy is cloned at runtime at a fixed commit, and nothing exercises either on a schedule or checks eval quality against cluster resources. The agent-drafted design splits the work along the lines the existing infrastructure already favors: a cheap per-commit smoke that installs both evaluators, resolves the evalchemy runtime clone at its pin, and dry-run-plans one evalchemy and one harbor eval config to catch API drift; plus a daily eval ferry cloned from the marin-canary-ferry pattern that runs a small fixed model on a small task sample (a couple of evalchemy math tasks and one harbor dataset) through iris_monitor.py wait, gated by a new validate_eval_metrics.py on completion, a throughput ceiling, and a loose accuracy floor — loose at first, tightened after calibration. Cadence, the exact model and benchmarks, per-run TPU budget, and the harbor agent/container backend are left as explicit decisions for a human before implementation. This is the concrete engineering that makes the epic's definition of done — Evalchemy and Harbor easily and reliably triggered from Marin on TPUs — stick rather than drift.

Eval-running friction surfaced on the lm-evals path that #6864 is meant to standardize. Rohith Kuditipudi filed #7004 after fresh GSM8K prompt generation and grading broke under the locked eval stack: the pinned lm-eval's bare gsm8k dataset path is rejected once the lockfile resolves datasets 4.x against huggingface-hub 1.21, which requires a namespaced repo id, so get_task_dict(["gsm8k"]) fails on an empty cache before any prompts or grades are produced. He flagged it to Romain Yon in code-talk — exactly the kind of dependency-pin rot the #7044 CI is designed to catch before it reaches a run.

The eval surface also grew outward. Benjamin Feuer opened a companion wish-list epic #7090 — a deliberately loose, living backlog of new evals Marin wants (more nuanced IFBench/IFEval, an agentless “baby-TB” few-turn terminal eval, arbitrary unseen-tool use, reliable structured generation, subagent lifecycle management, basic numeracy failure modes) — and recruited external contributors in the evals channel, pitching each item as work someone can drive largely independently. Separately, Jacob Silterra shared BioConfoundBench, an external biology eval built around misleading train/test splits where frontier models go 0/90 on the enzyme confound while the controls invert cleanly, offering it for a Marin trial. Both broaden the menu of what Marin will eventually run, on top of the TPU-triggering plumbing this epic is hardening.

With last week's v0 audit worklist filed, the July-hero datakit release turned this week toward the epic's definition-of-done items on new dataset inclusions and more code data. Three sources were registered, all still open in review. #7119 onboards three components of allenai/dolma3.5_pool — dolma4pdfs (OCR'd academic and web PDFs), dolma_code (reprocessed Stack v2 code), and dolma_code_prose (documents mixing prose and code) — each pulled through its own directory-scoped glob rather than the full ~10T-token pool, with mixture token counts estimated from a byte-weighted .jsonl.zst sample and cross-checked against the token_count field on a subset of shards. #7112 registers EssentialAI's eai-taxonomy-code-w-dclm, roughly 564B tokens over 273.8M documents of Common Crawl web text kept for code and CS reasoning by the EAI taxonomy classifier plus instruction density by the DCLM classifier — a thin wrapper over the shared normalize steps since its text/id fields already match the default schema. #7008 revives the source portion of the stale #5276 to register SWE-rebench V2 ConTree execution traces (roughly 182.6B Llama-3 tokens), deliberately leaving the ConTree tracing and tokenize pipeline out to avoid carrying that infrastructure, on the assumption it belongs in marin-experiments as a one-off.

A second thread hardened ingestion of the GitHub Actions logs (GHALogs) source, whose normalize step had been reading a manually pre-staged 142 GB Zenodo archive and raising FileNotFoundError when it was missing. Rafal Wojdyla's agent added #7023, a download step that streams the archive from Zenodo straight to the object store in 64 MB chunks, publishes atomically after verifying the exact byte length so a truncated stream never becomes the published archive, and is wired as a dependency of materialize so the chain auto-stages before reading. A real staging run then exposed why a single stream is fragile: it broke mid-body at ~8 GB with an IncompleteRead and restarted from byte zero, and Zenodo throttled the connection to ~1.1 MB/s — a projected ~36 hours for the full archive, since Zenodo throttles per connection, not per client. #7095 replaces the single stream with 32 contiguous byte-range shards run as a zephyr pipeline, each issuing bounded Range requests, resuming from its last yielded offset on a drop, and publishing its part atomically onto the bucket's time-to-live temp prefix so a failed run's parts get swept; the archive is then reassembled server-side, with concurrent shards sidestepping any single connection's unlucky throttle.

The third thread made the datakit web explorer — the ducky-backed store and pipeline viewer landed as #6641 — deployable and self-sufficient. #6999 makes it a pure ducky client so the dashboard needs only a ducky URL and no object-store credentials of its own: the store artifact, dedup record, and lineage cache reads that previously hit the store directly now go through the ducky service, the tensorstore-only per-bucket cache view (which DuckDB cannot read) is dropped in favor of the reconstructed store_bucket_samples, and lineage resolution is made scheme-aware, preferring the zero-egress R2 replica and targeting depth-3 source globs. That same PR fixes deployment against the Identity-Aware-Proxy-fronted marin controller, which the old credential-less client failed with Unauthorized; the fix resolved 114 of 114 sources end-to-end in about 13 seconds. #7002 then moves the Sources leaderboard computation into the app itself, running two progressive background passes at startup — a fast file-count glob to install size ordering (which drops bucket sampling from over 45s to ~2s), then exact per-source doc counts, quality distributions, contamination rates, and dedup drop rates — so the app serves immediately and stays fresh with no pre-baked summary file to go stale. A deployed instance was already circulating, with Benjamin Feuer sharing a link to it in data-mixing.

Outside the core team, the week's contributions clustered on GPU and data-pipeline plumbing. @timodonnell traced levanter's GPU attention silently falling back to an unfused reference path #7013 and wired the shipped FlashAttention-4 kernels in as the default #7015; Matt Wittmann threaded stateful fp8 ragged-dot ops through the grug MoE expert MLP #7079; and Will Moss kept the zephyr data pipeline maturing on CoreWeave object storage #6996, #7072, #7120. Ashwinee Panda re-audited the cross-framework xorl reproduction of the Delphi RL run under a same-ruler contract #6915, and Pranshu Chaturvedi caught a model-size bug in his tokenizer digit-variant sweep #6571 and queued a clean rerun.

The week opened with the first Marin community meeting, after which Percy Liang shared his slides. In the random channel, Kevin Yin weighed in on the Dynamic Tanh “Transformers without Normalization” result, judging it unlikely to buy improvements since tanh was no faster than RMSNorm in his own tests. And on the satellite side, Gonzalo Benegas reported that the MarinDNA Mendelian-traits leaderboard now covers supervised linear probing, where its best model widens its margin over Evo 2.

Three newcomers introduced themselves: Jose, a self-taught engineer working through Stanford's CS336 who found Marin via its open course resources; Anmol Kabra, a Cornell PhD student on synthetic data, RL fine-tuning, and AI-for-science interning at Snorkel AI; and Daljeet Virdi, co-founder of Diffuse Labs, which builds RL environments for frontier labs. Daljeet Virdi's environment work intersects the RL-data breakout now organizing around TaskTrove and the agentic-trace index #6191, and Anmol Kabra brings context on that same synthetic- and RL-data sourcing effort. Two researchers also joined without posting an intro: Richard Liaw (Ray, Ray Tune; Anyscale), whose distributed-orchestration work intersects the Iris cross-cluster federation that went live this week #7064, and Margaret Li (Branch-Train-Merge / Branch-Train-MiX), whose modular-expert-model work intersects the GrugMoE routing and DenseMixer SFT-router threads #7088.

The reading shared this week skewed toward cheaper proxies for expensive capabilities — Graham Neubig et al's PACE proxy for agentic-capability evaluation, SimCT for cross-tokenizer distillation, and test-time curricula for targeted RL — alongside a skeptical revisit of normalization-free transformers and Marin's own Delphi scaling-laws writeup.