Marin: Week of June 29th summary

With the June GrugMoE-on-vLLM-TPU epic #6041 closed on July 2, Romain Yon opened #6867 to carry inference through July: the definition of done is serving full-size GrugMoE checkpoints correctly on both TPUs and GPUs, with fast-enough GPU inference as the stretch. Three sub-issues split the path — #6868 targets correct vLLM inference on the largest GrugMoE checkpoint to date on TPUs, #6869 the same on GPUs, and #6870 throughput good enough for reinforcement-learning rollouts on GPUs, with the fast-enough bar still to be pinned down relative to JAX and a reference model. Several of the caveats named at the June epic's close — throughput and latency coverage, wider parallel serving — are exactly what the new epic takes up.

The TPU foundation the epic starts from landed this week in two stages. #6735, merged June 29 by Romain Yon, moved Marin and Levanter onto Transformers 5 / Hugging Face Hub 1.x (transformers==5.12.1, huggingface-hub==1.21.0), retiring the temporary Transformers-4 bridge #6734. That cleared the way for the fork repin #6733, merged July 1: it pins the Marin-owned vllm and tpu-inference forks to tpu-inference v0.23.0 and its tested vLLM last-known-good build, converges the lock onto a single jax 0.10.1 / libtpu 0.0.41 stack instead of splitting inference and training extras, keeps only the fork overlays still needed (GrugMoE support among them), and fixes a Levanter DataLoader failure exposed by JAX 0.10.1's new stack primitive. Validation on v6e-4 covered direct vllm.LLM.generate, the GrugMoE real-checkpoint end-to-end test, and a local-mode broker/proxy/worker HumanEval smoke. Two detours surfaced along the way: Will Moss hit a macOS-only resolver failure pitting vLLM's CPU metadata (numba==0.65.0) against tpu-inference's 0.62.1 and opened the stopgap override #6766; Romain Yon traced the root cause to vLLM's setup.py ignoring VLLM_TARGET_DEVICE on macOS metadata hosts and fixed it in the fork, which Will Moss confirmed. Separately, the fully brokered Iris HumanEval smoke was blocked by a controller 404 on EndpointService/ListEndpoints #6754 — a control-plane mismatch rather than a serving regression, resolved after Russell Power restarted the cluster with an update to the endpoint API on June 30. Process documentation hardened the refresh workflow itself: #6724 writes down the post-merge fork-promotion protocol, #6768 has refreshes keep shared workspace dependencies aligned by default, and #6827 separates upstream refreshes from fixed-base overlay pin edits, while #6756 updated the served-Qwen HumanEval script to the current Iris config-loader API and #6749 cleaned up the GPT-2 tokenizer test fixtures flagged in Russell Power's review of the migration.

On GPUs, correctness got its first end-to-end evidence. Draft PR #6891 built a harness that exports a real checkpoint from the moe_may_compute_opt_d512 run at step 10,980, serves it with tensor parallelism 1, data parallelism 8, and expert parallelism 8 — each GPU owning a slice of the experts — on an 8×H100 CoreWeave node, and compares 16 greedy completions against a JAX/Levanter reference sharded over the same expert-8 mesh. The full CoreWeave run passed under both the Triton and FlashAttention backends — all eight data-parallel ranks responding, routed-expert ownership covering all eight expert ranks, and vLLM matching the Levanter reference batch for batch — and a fresh PR-head validation run passed again on July 3, though two split-environment reruns hit an opaque WorkerProc initialization failed before GrugMoE startup that Romain Yon classified as GPU wheel/bootstrap drift rather than a correctness mismatch. The harness deliberately covers short greedy parity and expert coverage only; performance, sampling parity, and long-running stability wait on #6870.

Probing GPU attention backends then surfaced a real model-side bug. #6964 documents that Grug MoE's Exclusive Self-Attention (XSA) value correction assumed a single hard-coded query-head sharding, while GPU backends are free to return attention output either head-sharded or head-dim-sharded; #6965 fixes the correction to follow the actual attention-output partition spec and adds a focused lowering test pinning the XSA contract under grouped-query attention on the compact Grug mesh. The validation harness was restacked on top of that fix as #6966 — carrying the CoreWeave/H100 harness, the vLLM fork pin, and dual Triton/FlashAttention summary gating — and #6891 closed as superseded. The same GrugMoE-on-GPU mesh reasoning surfaced in #automate-research, where David Hall caught Codex leaking raw chain-of-thought as it puzzled through compact_grug_mesh layouts across node counts.

Romain Yon opened the milestone's eval epic #6863 this week with a one-line definition of done: both Evalchemy and Harbor can be easily triggered from Marin on TPUs. Its three sub-issues split the path there. #6864 standardizes eval running behind a single high-level entry point — ./run-eval --cluster=... --checkpoint=... --eval-config=... — covering both Levanter checkpoints and Hugging Face models on TPU, with results landing in Google Cloud Storage before any UI or database layer, plus a side quest to clean up legacy eval code. #6865 adds Harbor support with a concrete parity target — Qwen3-32B on TBench2 through Daytona sandboxes — and #6866 covers invoking evalchemy.

The epic's first week already banked its most important de-risking result. Benjamin Feuer filed #6958 documenting that agentic software-engineering evaluations — a terminus-2 agent solving SWE-bench-Verified tasks — now run end-to-end on Iris v6e TPUs: vLLM serves the model on TPU, the agent drives Daytona sandboxes, verifiers score the final repo state, and results auto-register to the eval database, just as on the reference SLURM/GPU workflow. The parity check backs it up: the GLM-4.7-swesmith base model scored 0.240 (72/300 resolved) on swebench-verified-random-100 on a preemptible v6e against the H100 reference's 0.237 (71/300), with zero infrastructure errors and full per-trial trace datasets published on Hugging Face for both runs — a result he announced in the evals Discord channel. The write-up also nails down the one TPU-specific trap: vLLM's GPU-only --swap-space flag is rejected by the tpu-inference api_server, crashing the serve and presenting as a silent health-check timeout, so GPU-only flags need stripping on the TPU serve path. That validated TPU backend is exactly the substrate #6865's Harbor work builds on, and he also shared #6877 for coordination — OpenThoughts-Agent's unified Harbor-eval entrypoint on its SLURM clusters, a working reference for the kind of single front door #6864 wants on Marin.

Supporting infrastructure moved on two fronts. On typed interfaces, Russell Power's Executor-replacement PR #6649, merged Tuesday, typed two eval outputs — LevanterEvalResult for the Levanter evaluator's results.json and CompiledEvalResult for the evalchemy compiler's rollups — and his follow-up #6781 proposes that the three remaining evaluators (the vLLM lm_evaluation_harness runner, evalchemy, and harbor), which today write divergent nested, timestamped layouts with no shared schema, each also emit a small marin-owned summary.json of per-task metrics plus provenance at the artifact root, so one typed EvalResult reader can serve all of them instead of fragile per-tool globbing. On serving plumbing, Benjamin Feuer's #6847 asked for a first-class way for off-cluster Daytona and Modal sandboxes to reach on-cluster vLLM without one unauthenticated paid pinggy tunnel per job; Russell Power took it up the same day, and the resulting agent-drafted PR #6857 merged Friday, giving each Iris endpoint a declared access mode — private, public, or bearer — with controller-minted scoped tokens that authorize only that endpoint's proxy path, and per-provider ingress that opens only the proxy route (an Identity-Aware-Proxy-exempt backend on GCP, a Traefik ingress on CoreWeave). Together those remove the last ad-hoc bridge in the agentic-eval loop that #6958 validated.

The GPU fleet tripled this week. Russell Power stood up a second CoreWeave cluster, marin-rn02a (RNO2A) — 64 nodes of 8 H100s each, 512 GPUs total, pinned warm rather than autoscaled since the capacity is prepaid — with Kueue gang admission, finelog, and controller state and checkpoints on the shared CoreWeave object-storage bucket reached through the node-local LOTA cache endpoint #6909. Bring-up was validated end to end with CPU/GPU smokes and single- and two-node training runs, though the multi-node leg first required a networking fix: the cluster config pinned NCCL_SOCKET_IFNAME to an exact-match interface name that the task image's NCCL (NVIDIA Collective Communications Library) build does not honor, so bootstrap found no socket and multi-node clique init died #6940; #6941 switched all CoreWeave clusters to the exclude-list form already proven in the GPU-smoke continuous-integration lane. Around the edges, Rafal Wojdyla put TTL lifecycle rules on the CoreWeave temp buckets and made bucket store types explicit config #6770, Romain Yon documented how to reach the object store from a laptop #6832, and the CPU scale-group specs were corrected to the real node hardware in #6793 and #6794. Because some GPU kernel work wants JAX's one-process-per-GPU launch layout, David Hall raised the launch-layout question on Discord, and after weighing gang-scheduling alternatives Russell Power shipped processes_per_task — N JAX processes inside one pod, preserving intra-node NVLink — in #6776, closing #6690.

RNO2A's first serious workload — a tokenizer bake-off soak fanning out to roughly 1,500 pods — promptly found every soft spot in the control plane, and the fixes landed the same week. Kueue's plain-Pod admission webhook had been intercepting pod creation in every namespace, which deadlocked fresh-node delivery on RNO2A (a new node's networking pod was rejected by a webhook it couldn't reach yet); #6894 scoped the webhook to the namespace Iris actually submits into. The controller itself first ran out of memory at its 16Gi limit, then — CPU-starved at 4 cores — failed its 1-second liveness probe and was killed mid-reconcile, orphaning in-flight sub-jobs #6944; #6943 moved its SQLite state off an NFS-backed volume onto node-local NVMe, and #6945 baked in 16 CPU / 64Gi Burstable defaults with a forgiving probe. A related failure — a user pod legally preempting the Kueue manager off the shared control node, taking the admission webhook down with it — got the whole control plane an iris-system PriorityClass above every user band #6929. A monitor to detect stalled GPU processes from their NCCL log signatures is proposed as follow-up in #6938.

The best debugging story of the week is #6950: the soak's two 128k-vocab arms reproducibly hung about 7 minutes into their first distributed collective and died on JAX's coordination timeout, every attempt. Suspicion fell on a bad InfiniBand leaf group, and Russell Power cordoned the five suspect nodes — but a relaunch on known-good hosts hung identically, and purpose-built minimal reproducers ruled out both the cross-leaf fabric and S3 starvation. The real culprit was software all along: a thread race in Levanter's tokenizer staging that intermittently forces a fatal Hugging Face Hub 404 for mirror-only tokenizer refs on a subset of the gang, whose survivors then park forever in the first collective #6954. With the staging body serialized behind a process-wide lock in #6955 — which also surfaces build_caches failures instead of letting them strand the gang — the retried arm cleared the death zone and became the first 128k arm ever to survive. The other host-side reliability arc was the checkpoint leak: #6774 diagnosed checkpoint saves leaving ~26 GB of permanent host-RAM residue per save until the cgroup killed training, #6785 gave each save a fresh tensorstore Context as a same-day mitigation, and after controlled A/B experiments pinned the dominant term on JAX staging every shard through pinned host memory the TPU runtime never returns, #6924 swapped saves to pageable staging — held dead flat across 140 saves in an overnight validation run — with the residual ~3 GB/h drift measured and closed as harmless allocator page retention in #6932. The hero-run side of that story is told in the 67B-A2B section; here it turned a monotonic ratchet into a bounded sawtooth for every large run on the fleet.

Russell Power also rebuilt Iris around federation. #6730 turned the controller into a meta-scheduler that routes each job to a single authoritative backend, and two design docs (#6718, #6814) set out the peer-federation model the rest of the week implemented: a federation module for observing peers in #6826, live handoff of whole jobs to a peer cluster in #6835, a single cluster-invariant job coordinate that eliminates remote-id rebasing in #6884, a durable relay of each cluster's logs into one shared global finelog behind mandatory authenticated ingress in #6871, and proxied exec and profile for federated tasks in #6846. Adjacent controller work landed too: #6857 gives Iris endpoints auth-gated public ingress via controller-minted scoped tokens, so off-cluster sandboxes can reach on-cluster vLLM without tunnels; #6853 let each backend author its full status rather than having the controller overlay worker health from its own roster; and an outside contributor added a matched pair of golden-replay scenarios isolating the preemption-budget gate — the branch where a preempted executing task either terminates or requeues — in #6800.

The week's other rewrite was the execution engine: #6649 replaced the eager, import-time Executor with lazy ArtifactSteps addressed by an explicit name@version — 276 files changed, a net five-thousand-line deletion — with follow-ups making cached outputs short-circuit their unbuilt ancestors #6791, moving training outputs into per-user directories built directly rather than baked in (#6790, #6809), and keeping pre-migration artifact records readable (#6807, #6890, #6892). The daily ferry canaries earned their keep against both rewrites, with each catch fixed in days: a trailing-slash MARIN_PREFIX produced doubled path separators that broke tokenize shard consolidation #6838 and then cache reads #6904 — patched in #6844, then fixed structurally by a StoragePath value type whose joins make a doubled separator unrepresentable #6922; a wedged W&B finish() left a healthy run with no metrics file #6906, fixed by writing the tracker replicate file before finalization #6923; a controller redeploy that baked --fresh into the pod command wiped a live canary mid-run #6808, now a one-shot wipe #6810; the new auth-gated edge briefly rejected every CI service-account submission as Unauthorized at the Identity-Aware Proxy layer #6934, fixed by falling back to the desktop client id when minting the edge token #6939; and a step-count floor calibrated for 4,096-token sequences failed every healthy 8,192-sequence run until #6960 recalibrated it. Steadier housekeeping ran alongside: Will Moss continued simplifying Zephyr toward workers that just report CPU and memory capacity and run whatever the coordinator hands them (#6789, #6831, #6751, closing #5855) and standardized pytest configuration across the repo's packages #6786, while David Hall added a train-state sharding-spec dump that preserves each Grug run's parameter and optimizer-state layout as a tracked artifact #6903.

The new supervised fine-tuning (SFT) data-curation investment epic #6714 picks up where the June synthetic-data work left off, with instruction-following lift as its metric of record — and it inherits a substantial closing deliverable. Benjamin Feuer posted the wind-down cohort update for the Qwen3.5-122B trace-generation campaign to the agentic trace dataset index #6191: 23 more datasets totalling 109,230 rows of Qwen3.5-122B teacher trajectories at 32k context, bringing the cohort to 49 datasets, each annotated with row counts and candid quality flags. The flags matter as much as the volume: codenet-python-v2 ships zero-byte stdin test files so every rollout hits an EOFError and scores reward 0 regardless of correctness, nemotron-cpp's verifier hands out a vacuous reward of 1 on empty test suites, and swe-rebench-patched-oracle came back with mean reward 0.000 because its SWE-agent tasks are simply too hard for the 122B teacher — leaving those slices useful only reward-stripped or as negative data. A companion data-quality result from Ahmed Ahmed — MATH-500 contamination in Nemotron-CC-Math — is covered in the pretraining-mix section.

Tooling around the synthetic-data-to-SFT path also firmed up. Benjamin Feuer added axolotl as a first-class SFT backend to OpenThoughts-Agent via a marin-community/axolotl fork in #6839 — motivated by training and serving on a single delphi.jinja chat template so the template-split-at-save-time failure class that once produced 0% SWE-bench scores cannot recur; the Delphi masking canary passed (reasoning spans trained, user and system turns masked), and the fork's template-integrity, MFU, and registry plugins were merged to the fork's main the same day. On the serving side, #6937 asks to finish the native /proxy ingress rollout so datagen sandboxes can reach on-cluster vLLM at iris.oa.dev with bearer tokens instead of pinggy tunnels — Russell Power plans to flip it on Monday, deliberately careful about opening up proxy tunnels and noting endpoint tokens currently live at most 24 hours — while #6877 documents the unified Harbor-eval entrypoint used to evaluate the resulting models, and an outside researcher asked in #6842 for Weights & Biases access to the OT-Agent-ColdSFT+RL-8B run to aid reproducibility.

Looking ahead, OpenThoughts-Next held its kickoff meeting on July 1, as announced in the news channel: the project will scope open data recipes for training agents, this time centered on mid-training and reinforcement learning of mixture-of-experts models, working with Marin models once they land, with the meeting repeating weekly and open to all. Meanwhile the alignment-function pipeline #3355, a spec-to-synthetic-preference-data-to-DPO (direct preference optimization) pipeline, was closed as stale.

The mixture work entered July already in production: the 10T-token mix assembled in June is the data under the 67B-A2B Mixture-of-Experts hero run tracked in #6044, and this week was about publishing the evidence behind it. Will Held posted a side-by-side speedup table across 111 evals in data-mixing — the new optimized mix versus the same data pool mixed proportionally versus the old Marin mix — answering Percy Liang's ask for a one-pass regeneration of the full eval set, with each cell clickable through to its underlying per-task scaling curve since fit quality varies by task; a follow-on RFC, #6802, asks where such analysis artifacts should durably live. Underneath the table, Calvin Xu closed out the two remaining scaling-anchor ladders: the proportional and UniMax-8 baselines over the 39 Dolma3/Dolmino buckets at four Delphi rungs in #6607, and the OLMix-for-Uncheatable-Eval baseline in #6608.

The table's headline question — curated versus proportional — got its recorded verdict in #6757, the epic's active experiment. Because each iso-FLOP speedup is an extrapolation through a four-point scaling fit, Will Held filtered the tasks on a 90 percent confidence interval from the fit residual, under which roughly 70 percent of per-task mix differences read as noise at this compute (3e17–3e19 FLOPs). What survives: both new mixes significantly beat the nemotron baseline on aggregate uncheatable bits-per-byte — a 1.78x iso-FLOP speedup for curated — and on code and math, both significantly regress on commonsense tasks like HellaSwag and PIQA, and the curated-versus-proportional head-to-head itself produced fewer significant cells than chance would predict. The recorded recommendation: take curated as a weak default, since its one candidate downstream edge — code — is in scope for a reasoning-, code-, and math-weighted intelligence target, but at these scales the mix choice is not a measurable lever, and a confident call needs at least one larger-scale point.

The open scientific question carried into July — does a phased curriculum actually beat the best single-phase mixture? — picked up a concrete test rig. Under #6609, Calvin Xu submitted the one-phase uncheatable-optimized scaling validation panel with its full scope declared up front: an OLMix one-phase uncheatable mixture and a DSP one-phase effective-exposure mixture, each trained at four Delphi scales spanning 3e18 to 1e21 FLOPs, with the native OLMoBaseEval Table-9 eval attached after every final export. The panel runs under a single Iris parent job, with the two smallest scales already training at submission time, and a companion tracker for the OLMix-curriculum ablation opened as #6801. A literature audit in #6931 asked whether nearby curriculum and specialized-pretraining papers already rule out single-phase mixtures being near-optimal in Marin's setting and concluded they do not: the audited comparisons often confound phase placement with aggregate target-domain exposure, so the proposed decisive test is a matched-exposure phase-order experiment — hold the aggregate mixture fixed, vary only where buckets appear in training, and check the effect against the measured 3e18 noise floor.

Hygiene for future math-heavy mixtures came from Ahmed Ahmed's validation-contamination investigation in #6742. He reported in midtraining that the puzzling Delphi math-midtraining prediction miss at 1e22 FLOPs traced to near-duplicate documents in the old Nemotron-CC-Math validation split — a clean actual-seen validation set cut the endpoint error from about 18.6 percent to 2.8 percent — and then ran a reproducible benchmark decontamination of the 45M-document corpus against math evals: 28.2 percent of MATH-500 test and 17 of 30 AIME24 problems have 10-gram overlap with corpus documents, despite Nemotron-CC-Math's stated deduplication against MATH-500 and GSM8K, while GSM8K itself is largely clean at around 2 percent. The takeaway for mixture work is to run decontamination in-house before trusting external corpora's claims. Scoping for the next mix also picked up a document-level signal: out of a data-curation discussion, Rafal Wojdyla filed #6750 to evaluate web-graph centrality — PageRank-style hub scores computed over the crawl hyperlink graph — as a data-selection feature, and Greg Lindahl of Common Crawl replied on the issue that Common Crawl publishes a web-graph dataset synchronized with its crawls, already uses harmonic centrality to rank hosts, and offered its conversion code; the related quality-classifier rework and the rest of the data-quality signal work sit with the diagnostics epic #6712.

The epic also picked up its first housekeeping win. Russell Power filed #6777 noting dataset definitions were scattered across six places in the experiments tree, and the agent-written #6797 — peer-reviewed as a proposal on the issue before implementation — merged on July 1, consolidating every dataset catalog into one flat experiments/datasets/ tree under a single convention: _dataset() for a single corpus, _datasets() for a keyed family, every handle's cache identity untouched so nothing re-tokenizes, and each module runnable on its own to print or opt-in build its plan.

The July milestone breaks data-selection diagnostics out as its own investment area in #6712 — strengthen the perplexity and diagnostic signals that predict downstream quality, with rank correlation between proxy and eval as the metric to improve — and it opens with its strongest result already in hand. On #6096, Will Held extended the SimpleRL-Zoo correlation study into a full exploitation/exploration decomposition. The exploitation half: a base model's bits per byte (BPB) on verified OpenMathReasoning chain-of-thought and tool-integrated-reasoning traces predicts post-reinforcement-learning math accuracy across the ten SimpleRL-Zoo base models (Pearson r = −0.89 on the chain-of-thought split, −0.92 on tool-integrated reasoning, −0.97 within the 7–8B cohort), and the signal is distribution-specific — real math traces predict far better than templated synthetic-reasoning bundles (−0.68 to −0.72), so the predictor tracks the target distribution rather than generic language-model quality. The exploration half asks whether a policy-spread term adds anything beyond correct-trace likelihood: token entropy turns out nearly redundant with BPB, but an embedding effective-rank metric — the participation ratio of the policy-weighted top-k output-embedding cloud, meant to capture semantic rather than syntactic branching — adds real signal, lifting the joint model to R² = 0.931 from 0.797 for BPB alone and cutting leave-one-out cross-validation error from 10.6 to 6.7 accuracy points, with a negative sign: at matched BPB, more semantic branching predicts worse post-RL performance.

The write-up leads with its own deflating caveat: RL adds a near-constant ~16 points to every model, so base and post-RL accuracy are 0.97 correlated — the predictors forecast a model's math-capability level, which RL preserves, not which base benefits most from RL, and predicting the gain itself fails at R² = 0.33 on this small, family-confounded sample of ten. Rafal Wojdyla pressed on exactly that point, and Will Held answered that it matches what Benjamin Feuer has been telling him: pre-RL pass@K is the main predictor of post-RL pass@K when everything else is held constant. Rafal followed up asking which evals come next and how to close the end-to-end loop back into pretraining, offering to help — which is the question this epic exists to answer.

On the document-quality side — the per-document counterpart to the mixture-level work under #6713 — Russell Power closed out #6739 with a deliverable and a diagnosis. The deliverable: a pooled fast-transformer quality classifier at 0.41M FLOPs per token that beats the fasttext baseline at matching the Sonnet quality oracle (AUC 0.875 and Spearman ρ 0.703 versus 0.846 and 0.641), comfortably under the 1M-FLOPs-per-token budget. The diagnosis: it will not go higher for free. Distilling three free teachers — a source-of-origin prior, Nvidia's Nemotron quality buckets, and FineWeb-Edu's per-document score — all landed below training from scratch on the 5.6k gold labels, so the ~0.87 plateau is a ceiling set by label quality, and breaking it means either paying for more oracle labels or, per his survey of how production filters are actually validated, judging classifiers by small-model downstream ablations rather than oracle agreement, since the oracle itself is edu-leaning. Russell Power reported the arc in data-mixing, where Rafal Wojdyla noted the deployed domain and quality classifiers are v0 pipeline placeholders with real headroom. A second candidate document-level signal, web-graph centrality #6750, is covered with the mixture work under #6713.

The recipe itself landed. Larry Dial's #6153, open since early June, merged the May Recipe onto main's MoE template: half-RoPE everywhere, partial key offset (PKO) on every fourth layer, split w_gate/w_up storage, routing renormalization to sum 2.5, 256 experts at top-4, no gradient clipping, 1% warmup, and MuonH replacing AdamH on the weight-matrix group — the optimizer last week's summary noted the TPU recipe was keeping — plus the learning-rate refit from #5951. The PR ships the epic's definition of done in miniature: a compute-optimal reference table at four isoFLOP budgets from the #6074 sweeps and a fitted scaling law (loss(C) = 1.6 + 88.90·C^−0.0941) — the same exponent as the v16 baseline but a ~2.12x equal-throughput compute-equivalent speedup at every budget. This epic's calls feed directly into the July hero-run architecture #6701 and the loss preregistration #6702, so the recipe-versus-baseline law above is the yardstick the hero run will be scored against.

The week's biggest open question was attention. Multi-head Latent Attention (MLA) looked good on quality at small scale — the best variant in #6522, with a learnable norm gain and XSA restored, reached ~1.14 effective speedup at d512 — but Larry Dial's FLOPs-per-token accounting across Trinity, Nemotron 3, DeepSeek-V4-Flash, and Gemma 4 configurations showed MLA on every layer costs +17–20% training FLOPs at 4k sequence length (about +5% if confined to a quarter of the layers) and balloons at long context, and he judged the team is unlikely to have bandwidth to optimize an MLA implementation on the GPU JAX stack before the planned September large-run kickoff. The same accounting is the current word on this epic's long-context scope — the extension plan in #6047 and the sequence-length experiments #6277 and #6232 ran nothing new this week, but the tables quantify the constraint they'll live under: an MLA stack would have to hold at 4k, or at most 8k, sequence length for much more of the run, while the grouped-query-attention hybrid keeps 32k training comparatively affordable. The working call is grouped-query attention (GQA), with #6889 opened to measure MLA's model-FLOPs-utilization at the d5260 target size before the decision is finalized. That decision produced a refreshed July Baseline #6882 — GQA 4:1, no long-window RoPE, no PKO — which finished all three cells just 0.004–0.010 above the May-Recipe law, close enough to referee the follow-ups. The standout on that baseline: alternating dense and MoE layers at matched active/total parameters #6885 is iso-loss but faster, with effective speedup growing 1.03 → 1.09 → 1.15 from d512 to d1024.

David Hall asked in the moe channel about revisiting latent MoE, and Larry Dial took it up in #6822: running the routed-expert path in a compressed latent halves the dispatch dimension for expert-parallel all-to-all traffic, Nemotron-style. Half-compression initially cost +0.04–0.055 macro loss at every scale, but the sweep traced the regression to an input-scale artifact — the latent down-projection under hyperball init leaves expert inputs at ~0.25 RMS, stuck in the low-signal SiLU regime — and a learnable RMSNorm on the latent (mirroring MLA's norm on the compressed KV, a connection Kaiyue Wen weighed in on) recovers it to loss-neutral on both the MLA and GQA stacks. The remaining payoff is purely the communication win at real expert-parallelism (EP) scale, which single-host runs cannot exercise; David Hall reckoned the ~0.96 measured speedup will more than pay for itself in comms.

The tokenizer question #6796 got a full arc in one week from Russell Power's agent (PR #6916), echoing the n-gram embedding trick Will Held flagged in LongCat 2.0. A FLOP-equivalent bits-per-byte (feBPB) rubric — quality as bits-per-byte, cost priced at the 250B-total/20B-active deployment target — scored a ~64k SuperBPE trained on the grug-moe mix plus a light hashed n-gram at −6.8% versus stock Llama-3 at proxy scale, crossing −10% under realistic serving-dominated lifetimes. A confirmation soak at 10B-total/500M-active — whose two 128k-vocab arms only joined after the tokenizer-staging thread race was run down in #6950, an infrastructure fix covered under the cluster epic — first shrank the win to ~1% and put the n-gram arm last, and then careful digging showed the soak itself was confounded three ways (superword merges trained on English-only text, an eval blind to the multilingual and math domains, serving fertility missing the code domain); a corrected re-run was training as the week closed. Elsewhere, David Hall's golem swept newcomer-facing TL;DR verdicts onto the older Good/Great 10T trackers #4023, #4024, #4033, #4035, #4036, #4037, and #4040, closing the loop on the spring optimizer and routing ablations the merged recipe now supersedes.

Epic #6710 is the July milestone's GPU-performance investment: the kernel and parallelism work behind commitment #6706 — getting B200 model FLOPs utilization (MFU) above the bar ahead of the planned B200 hero run #6689. The fp8 compute lane landed its foundation this week: Matt Wittmann merged #6660, giving haliax an Fp8DirectDotGeneralOp that feeds genuine E4M3 operands to the matmul instead of relying on XLA's pattern-matcher to recover fp8 from the legacy quantize-dequantize recipe, with H100 benchmarks of roughly 1.2× on the attention and shared-expert projections and 1.3× on the lm_head. The bigger story is the grouped path: dense dot_general covers only part of a Mixture of Experts (MoE) model, and the expert computation's ragged_dot — a grouped matrix multiply (GEMM) — had no fp8 equivalent, since XLA cannot rewrite a hand-written Pallas kernel. Experiment #6824 preregistered the hypothesis that an fp8 grouped GEMM with delayed scaling would beat the tuned bf16 Triton ragged_dot by at least 20% end-to-end on H100, and draft PR #6880 delivers Fp8RaggedDotOp well past that bar: a measured 1.41× forward-plus-backward speedup for the MoE expert MLP at the production operating point. Much of the margin came from a dedicated backward-pass optimization loop in #6930, which moved the acceptance numbers from 1.32×/1.19× to 1.41×/1.27× on the two expert GEMM shapes by retuning block configurations per leg, replacing the weight-gradient kernel's in-shared-memory boundary masking with pre-masked boundary blocks so the pipeline body is branch-free, and widening the cast-transpose tile. A fork-built jaxlib also validated that the intended mixed recipe — E5M2 gradients against E4M3 activations — costs nothing over uniform fp8; making that work on stock JAX waits on jax-ml/jax#38859. The loop's logbook ranks the biggest remaining lever as caching weight quantization across gradient-accumulation microbatches, measured at about +15% layer-level throughput.

With the kernels in place, #6911 ran a research spike on the last piece: carrying fp8 over the wire in the MoE dispatch and combine collectives, against a preregistered goal of at least 1.4× end-to-end on the full grug MoE layer at realistic parameters. The goal was met at 1.438× on two H100 nodes under expert parallelism (EP) 16 — bf16 at 33.47 ms/step versus fp8 at 23.28 — with a design lesson attached: fp8 belongs on permutation legs only. Quantizing the dispatch all_gather (E4M3 forward) and the combine-transpose all_gather (E5M2 backward) pays off, but decomposing the ring reduction into an fp8 all_to_all plus a local sum regressed cross-node, because NCCL's hierarchical reduce-scatter already reduces node-locally before crossing InfiniBand. Follow-up measurements pushed further: fusing the E4M3 quantization into the dispatch itself lifted the four-node EP32 ring result to 1.467×, and at that scale the ragged-all_to_all backend overtakes ring entirely, reaching 26.60 ms/step — 2.02× against the ring bf16 production configuration. The spike also root-caused that backend's puzzling single-node deficit to XLA's one-shot ragged all_to_all kernel, which an escape-hatch flag routes back through NCCL for a 96.3 to 26.5 ms/step recovery. Everything ships as opt-in branches — fp8-moe-mlp for the GEMM wiring, fp8-moe-mlp-comms for the wire format — and defaults stay bf16: demonstrating training stability with no loss regression is explicitly the gate before any of this turns on. The motivation is easy to picture: the same week, David Hall posted a mock expense report in the GPU channel itemizing his step time — "random all gather $3,600… Please help my tensorcores are starving" — and the all_gather line is exactly what the fp8 wire work shrinks.

The fused-kernel lane advanced in parallel. David Hall's Pallas Mosaic GPU (MGPU) fused MoE effort under #6597 pivoted from optimizing dispatch in isolation to a source-push design over the whole expert MLP — one custom gradient boundary that saves the W13 preactivation as its checkpoint — now tracked in #6848 with a production-boundary PR #6841 and prototype #6840: the stable target-shape median is 218 TFLOP/s per rank, about 13.8% faster than the serial ring-prologue baseline, and a one-node 20-step trainer smoke ran the path end to end at about 20% MFU and 554k tokens/s — right at the commitment's bar, on Hopper. Those probes also caught the CUDA stack loading the wrong cuDNN — a CUDA 12 torch dependency shadowing the CUDA 13 wheel — which #6947 fixes by restoring CUDA 13 precedence after toolchain staging. The Blackwell sibling #6933 had a busy week of its own: the intended fully-fused design was ruled out because the current JAX/Mosaic stack cannot lower peer-id memory references in Warpgroup mode, so the practical path became staged transport plus local Blackwell compute — and that staged path tuned its transport to about 1,596 W13-equivalent TFLOP/s per rank on B300, cleared its 60%-of-local-compute gates, survived the full 65k-tokens-per-rank target forward with zero dropped routes, and cut the public-path steady state from 39.97 s to 0.093 s by reusing the route plan across calls. A backward that initially exhausted memory at the target shape was unblocked by regrouping rows per destination-local expert with a 512-row tile, completing the first full target forward-plus-backward run — still on a JAX fallback backward, with a tuned Pallas backward as the named next step.

The rest of the epic's surface got steady attention. David Hall merged FlashAttention-4 sliding-window support for the B200 path in #6377, teaching the FA4/CuTe backends to handle sliding-window causal masks while keeping full-window masks on the existing causal path, and followed with #6910, which keeps FlashAttention-4 metadata tensors shard-local before entering the shard_map to avoid a global-metadata mismatch during lowering. #6720 added semantically meaningful jaxtyping axis annotations across the expert-parallelism MoE helpers. Rafal Wojdyla's standalone reproduction of the GPU MoE model under #6759 now runs the production grug MoE train step on synthetic data with no Marin infrastructure at all — a single-file launcher validated on an 8xH100 node, built for sharing with external engineers. And #6695, verifying the cross-entropy path in the May profile, remains open: the multi-node MGPU probes it depends on do not yet reach a train step where that measurement is useful.

This Investment epic carries the framework-selection question into July: evaluate candidate reinforcement-learning (RL) frameworks, close the parity gaps tracked in the feature checklist #6341, and settle which framework Marin standardizes post-training on. The working document is #6162, Benjamin Feuer's evolving comparison of candidates from veRL and slime to SkyRL and torchforge, which Jeff Hammerbacher cross-linked to the parity checklist this week. On the in-house side, Tai Vu's trainer refactor #4766 was brought under the epic: it moves the RL trainer off loss-specific REINFORCE-leave-one-out (RLOO) surfaces onto a compositional objective runtime with an objective-neutral trajectory and replay data plane, so new objectives and on-policy distillation can be added without rewiring the trainer. It preserves current RLOO behavior, fails fast on unsupported objective shapes, and is still in review.

The week's strongest datapoint for the comparison came from outside the repo: the cross-framework reproduction of the MarinSkyRL rlvr7500_w1 Delphi run in Together AI's xorl #6915, which matched and exceeded the reference math lifts on a fraction of the hardware; the full story, including the bit-aligned sampler and trainer numerics that kept the run truly on-policy, is covered in the Delphi RL section. Discussion in the Discord midtraining channel turned to what the divergence teaches, and Benjamin Feuer flagged K3-divergence tracking as a feature to weigh in SkyRL. Two adjacent threads wound down: Rafal Wojdyla closed #6095, the fakeray shim for running Ray-based frameworks on Iris/Fray, with Russell Power noting he may resurrect it when the MarinSkyRL evaluation deepens, and the multi-host weight-sync experiment #4287 was closed as not planned.

The Delphi reasoning-RL scaling study #6279 closed out this week. Benjamin Feuer posted the final two large-scale runs and summed up the arc on Discord: midtraining supplies the math capability, supervised fine-tuning elicits it, and forced-thinking RL adds further lift on top — together taking Delphi 1e22 from 1.6 to 53.4 on MATH500, 0 to 6.6 on AIME24, and 11.0 to 68.6 on GSM8K. The new runs settle two questions: in the RLVR-MATH-7500 sweep (reinforcement learning with verifiable rewards on a 7,500-problem math set), the length penalty wins — it adds +2.0 MATH500 and is the only leg with non-zero AIME24 — and dapo17k_w1, trained twice as long on the larger DAPO-17k set, became the first run in the study to beat its SFT AIME24 baseline at 6.6±0.4, evidence the 4k-context truncation wall is dataset- and training-length-sensitive rather than a hard ceiling. When Yiyuan Li pressed on the odd Qwen3 base-model ordering (8B scoring below 4B on MATH500), the raw pass@k artifacts went up on Hugging Face: the gap is a pass@1-only effect at temperature 0.7 that vanishes for k≥8 — and Yiyuan Li's own majority-voting reanalysis confirmed the expected scaling holds.

The close-out drew outside verification within days. In #6915, Ashwinee Panda reproduced the flagship rlvr7500_w1 run in Together's xorl framework — same checkpoint, data, and recipe, but on 16 H100s in 8 hours where the original used 128 A100s for roughly 41 — and exceeded the published deltas: +12.3 MATH500, +8.8 gsm8k-flex, with AIME24 no longer regressing. The write-up credits sampler↔trainer numerical alignment as the enabler: with bit-aligned kernels the run is fully on-policy and PPO clipping never fires across all 146 steps, whereas a pipelined variant of the same recipe collapsed at step 21, with the K3 KL estimator as the leading indicator. It is a strong cross-framework datapoint that the Group Relative Policy Optimization result is a property of the recipe, not the infrastructure.

The week's other resolution was months in the making. #6959 records the corrected agentic-RL comparison after Benjamin Feuer root-caused a summarization bug in the terminus-2 harness: when the agent's periodic transcript-summarization call timed out, the harness aborted the whole trial as a task failure, deflating scores unevenly — one base measurement lost 206 of 300 trials this way — which had made a single hero RL run look irreproducibly special. With the bug fixed and every affected model re-evaluated, the clean result is that RL reliably improves agentic software-engineering performance regardless of recipe or dataset: all 20 RL variants beat the 8B SFT base on SWE-bench-Verified (0.303–0.367 versus roughly 0.25), while being statistically indistinguishable from one another. Behaviorally, RL teaches the model to close the loop — edit the source, re-run the tests, and submit only once they pass — where the SFT base stalls in repository inspection.

This epic tracks landing the 67B-A2B “Grug” 10T-token production run, which was preregistered and launched last week and keeps its day-to-day logbook on #6044. The run spent its first full week training on the TPU v4-2048 and passed its planned first gear change. Stage one ran at the 33.5M-token batch through step 15,000 (~513B tokens); the original W&B run ended in a crash after ~81 hours — consistent with Larry Dial's note in the infra channel that hero-run attempts routinely die to hardware and need manual restarts, and that Iris's two-week branch-freshness threshold forces a resync of ~96 main PRs mid-run each time — and on July 1 he relaunched from the step-15k checkpoint as resume15k_v2 with the batch doubled to 67.1M tokens and the replica axis halved to 8. The resumed run has sustained ~2.5M tokens/s since, lifting MFU from 13.5% during ramp-in to 18.6%, and by week's end had processed roughly 1.47T of its 10.07T tokens — about 15% — at train loss 1.44. Larry Dial announced the run in the moe channel with a public W&B report, estimating about 50 days to completion.

Against the preregistered stage-1 target of 2.269 paloma macro loss at 8T tokens (evaluated at sequence length 8,192 over 1,024 sequences), the run currently reads 2.429 — a mid-run number with the learning rate still near peak, which Larry Dial cautioned is not yet the one to compare against the target, likening it to reading a pot's water level while it is still boiling: a cooldown drops the loss further. The early signs are favorable. At 300B tokens (3% in), loss had already reached 2.55, roughly where the 1e22 dense 10B Delphi model finished, per his lineage recap of Delphi 1e22/1e23 through the 129B-A29B 1T run, and David Hall's reaction to the side-by-side Paloma curves against prior runs — “wow that paloma loss” — set off a thread on how much is the new data mix versus architecture, with Will Held noting the new mix had been slightly worse on Paloma in his experiments, crediting the difference to the architecture work.

The week's main incident for the run was on the host, not the accelerators: checkpoint saves were leaking host RAM until the container cgroup killed training, and the Grug job was among those bitten — lasting about 20 saves per attempt, per the diagnosis in #6774. Russell Power merged a first fix in #6785 on July 1 expressly so the hero run could pick it up, and the dominant leak was closed in #6924; the full diagnosis-to-fix arc is covered in the cluster epic's section.

Planning also started for what happens to the model mid-flight. This epic's first sub-issue, #6811, opened this week, targets an intermediate cooldown from roughly the 1T-token mark — graded on pass@256 across a small eval set — so mid-training, SFT, and RL can exercise the inference and post-training stack well before the full 10T checkpoint lands. Related plumbing in #6752 proposes decoupling Hugging Face checkpoint export from the training job into its own artifact step, since the current inline export cannot succeed on multi-host TPU gangs and fails jobs after training itself has finished — exactly the export path a cooldown checkpoint will need.

The June data-pipeline epic closed this week, and its follow-on — the datakit release for the July hero run — took shape around what the first end-to-end store revealed. On July 1, Rafal Wojdyla closed #6036 against its two acceptance criteria: CoreWeave's native object store is now the blessed storage for pretraining data — settling benchmark #6671, which had found CoreWeave beating Cloudflare R2 on reads, writes, listing, and tail latency — and the datakit pipeline now produces its output natively on that store at s3://marin-us-east-02a/marin/datakit/store_8ac06c74 rather than being mirrored in from R2. The work continues under #6037, whose definition of done spans deciding new dataset inclusions, adding a Common Crawl proof-of-concept crawl, adding more code data, fixing the known child issues, and evaluating the new mix via #6054 before producing it. Rafal Wojdyla scoped it on the epic this week: the data feeds the mid-July hero run, and the work covers fixing known issues, exploring the data, and plugging into the end-to-end feedback loop tracked in #6096.

The June closure let the throwaway scaffolding come down: in #6688, Russell Power deleted the R2-to-cwobject nemotron cache mirror (2.25 TB) and temp checkpoints from cwobject-data test runs (226 GB) off the cluster-local store, leaving only the mirror and monkeypatch code retirement gated on the upstream virtual-hosted-S3 change tracked in #6686. A brief ambiguity over two copies of the store — a finalized top-level one and an in-progress marin/-prefixed rebuild — was resolved when Rafal Wojdyla confirmed the marin/ path as canonical. Making native CoreWeave data real took two more object-store fixes. Zephyr had been reading parquet shards by handing a raw s3:// string to pyarrow, whose native client uses path-style addressing that cwobject rejects with HTTP 400; #6952 routes those reads through fsspec's virtual-host-aware filesystem — the same helper the JSONL reader already used — and merged after verifying against a CoreWeave-hosted Wikipedia shard. On the write side, #6887 documents a harder problem: Levanter's tensorstore native-S3 jagged-array writes hang forever with no timeout on the heaviest arrays, which stalled the clustered-store shuffle build at 638 of 644 caches on CoreWeave. The store also got quality-of-life work: Romain Yon documented how to reach the marin-us-east-02a bucket from a laptop in #6832, and #6770 configured time-to-live lifecycle rules on the CoreWeave buckets' temp paths and moved bucket store types into config.

Separately, Rafal Wojdyla's agent hardened the pipeline's cache identity for cross-region reproducibility: #6895 makes every reference-pipeline step's hash complete and region-independent, folding in previously un-hashed dedup, centroid, and store parameters and replacing region-specific gs:// paths with caller-supplied version tags. That directly addresses the non-determinism logged in #6798, where two builds of the same store diverged; follow-ups #6896 pins the tokenizer bytes, #6897 version-tags the decontam eval corpus, and #6898 checksums the download modules. In smaller cleanup, #6769 deleted the dead npm registry-metadata download module — download-only code whose sole consumers, the perplexity-gap eval experiments, were removed earlier — closing #6746.

The epic's eleven open sub-issues are the worklist that came out of auditing the full clustered store via ducky SQL, and the recurring theme is that the store's domain and quality axes are largely uninformative. On quality, #6849 finds the fastText classifier sorts documents by domain and modality rather than intrinsic quality, #6860 finds it returns near-constant scores within many sources, and #6859 finds it judges long documents on a truncated 4 KB lead. On domain, #6850 shows the luxical embedding collapsing all code into a single cluster and #6855 shows the k=40 clustering severely imbalanced with language and modality catch-alls at a 64,533× largest-to-smallest ratio. Dedup is the most consequential: #6851 and #6854 show fuzzy dedup over-merging templated synthetic data and wiping whole sources — a median of roughly 76% of documents removed per source, and 100% of massive_function_calling — while #6858 flags the store over-sharded down to roughly one element per shard in small buckets, and #6852 notes decontamination's deliberately precision-favoring matching flags very few documents, at the cost of recall on embedded or short-line contamination. Rafal Wojdyla framed the classifiers as intentionally v0 in data-mixing — “the goal was to have e2e pipeline. We can definitely do better!” — and #6856 requests a web visualizer to explore each store's stage lineage. Two long-stale tokenization-cleanup items, #4588 and #4476, were closed in the same triage.

The week's most visible side project was ducky, an always-on Iris service for ad-hoc DuckDB SQL over the project's parquet data in object storage. Rafal Wojdyla landed the core service in #6762 — paste SQL into a dashboard, it runs in embedded DuckDB on a full v6e TPU host, and a capped preview is backed by the full result spilled to a time-limited Google Cloud Storage path — closing out the design issue #6760. Follow-ups through the week rounded it into a real tool: pre-baked catalog views over the finelog and datakit datasets plus parquet-footer caching and a dashboard picker #6875, per-query logging to finelog #6886, a preemption-resilient programmatic client that authenticates through the Identity-Aware Proxy with a service account #6893, and an automatic deploy workflow on merge to main #6899, #6900. A companion issue #6761 sketches smallquery, a distributed BigQuery-style SQL engine on preemptible TPU VMs, positioning ducky as the single-host stepping stone. In the same spirit of shared infrastructure, #6802 and its in-progress implementation #6816 propose durable public hosting for one-off analysis HTML pages, replacing the personal sites where such dossiers currently live.

Testing and developer tooling also saw steady work. Will Moss pushed the unified unit-test workflow forward on three fronts: a cleanup pass standardizing per-package pytest invocation and deleting dead configuration #6764, a merged step resolving pytest namespace collisions across packages #6786 with a small conftest import fix behind it #6834, and an import-driven test-selection workflow that builds a reverse dependency graph so only the tests affected by a diff run #6833; along the way it surfaced that lib/rigging's fifteen test files are not run by any workflow at all #6925. Russell Power fixed a collision where concurrent agent worktrees on one host overwrote each other's lint-review logs #6917, #6919, stopped agents from appending self-credit footers to PR bodies #6780, and sketched two workflow ideas: an experiment-issue template that an agent reaps nightly into documentation and blog teasers #6775, and dataset modules that are both importable and directly runnable as lazy builds #6782. The nightly nightshift passes continued their behavior-preserving cleanups across the subprojects #6747, #6772, #6806, #6837, #6905.

A run of this week's threads came from outside the core team. The fp8 arc in the GPU section is Matt Wittmann's work end to end — the merged haliax direct-quantization op #6660, the H100 fp8 ragged-dot kernel #6880, and the over-the-wire spike #6911. Elsewhere, Matthew Jacobs added a matched pair of controller golden-replay scenarios isolating Iris's preemption-budget gate in #6800, Tai Vu pushed the RL trainer refactor onto an objective runtime and neutral data plane in #4766, and Rok Mihevc weighed in on the smallquery design thread #6761, suggesting DataFusion and PyArrow before building a SQL engine from scratch.

The reinforcement-learning channel carried the week's most substantive exchange. Benjamin Feuer closed out the Delphi reasoning-RL study #6279 — midtraining, SFT, and RL together lift Delphi 1e22 from 1.6 to 53.4 on MATH500 and 11.0 to 68.6 on GSM8K — and in a second thread traced months of hard-to-reproduce agentic-RL comparisons to a summarization bug in the terminus-2 harness, with the corrected comparison written up in #6959. The close-out drew outside verification within days: Ashwinee Panda reproduced the flagship rlvr7500_w1 run in Together's xorl framework on 16 H100s against the original's 128 A100s, with same-harness math deltas at least matching the original, filed as #6915; Yiyuan Li pressed on the Qwen3 base-model MATH500 ordering in the same study, and the raw pass@k artifacts went up on Hugging Face in response.

Two announcements set the community calendar: Percy Liang kicked off a roughly biweekly Marin community meeting over Zoom — roadmap, open research problems, experimental results — starting Tuesday, July 7, and David Hall published a manifesto on why Marin builds frontier models in the open. The week's news reading circled how training data gets made and measured — null-expert MoE architectures, one-model web extraction-and-curation pipelines that Michael Ryan is now benchmarking against his own, and whether smooth scaling curves decompose into discrete capability quanta.

Introductions brought Franziska Weindel, a PhD student at TU Munich working on agentic discovery for problems in information and coding theory — ground that intersects the automate-research effort — and Rajat, an ML engineer at Chime interested in adaptive AI systems that continuously learn and improve. Among the two dozen who joined without posting an introduction is the handle of Sumanth Hegde of Anyscale, a core contributor to SkyRL — the RL library whose Marin fork, MarinSkyRL, ran this week's large-scale reasoning-RL experiments.