Marin: Week of August 24th summary

The ragged all-to-all expert-parallel backend, reworked in #8549 by Matt Wittmann, merged this week and is now set to become the hero run's default transport via #8684. An A/B test at hero shape on one NVL72 rack, both arms restored from the live hero's step-6000 checkpoint, measured 22.87% MFU for ragged versus 22.71% for pooled-wave, with a 150-fold reduction in expert-assignment drops (0.018% versus 2.67%) and 12 GiB lower peak device memory (137.9 versus 149.9 GiB). The switch pins a patched PJRT wheel from the new marin XLA fork, and checkpoint migration from the old pooled-wave format is handled automatically. Separately, Matt Wittmann fixed edge bounds and capacity bugs in the ring expert-parallel (EP) backend's ragged-dot kernel in #8610, closing the wrong-output and NaN-gradient failures tracked in #8578.

Three further optimizations, stacked in draft #8753, push hero EP64 MFU from 22.84% to 23.50%. A layer-carry offload frees enough device memory (peak HBM drops from 138.0 to 116.5 GiB) for the XLA latency-hiding scheduler to overlap ragged transport with compute, adding 0.44 percentage points (pp). A packed QuACK gate/up interleave raises effective bandwidth from 1.9 to 7.13 TB/s, contributing another 0.22 pp. A wider FA4 forward tile tuned for the d6144 B200 shape adds 0.28 pp, measured at the step-24000 restore point. Combined, the three changes yield roughly 4% more tokens per second. A parallel diagnostic branch from Rafal Wojdyla in #8733 measured 22.90% p50 MFU (244,738 tok/s) with one expert wave, fused shared experts, and small-parameter Fully Sharded Data Parallel (FSDP), a 5.6% throughput gain over its control.

David Hall added automatic DualPipeV and ZeroBubble pipeline schedules for Grug MoE in #8739. The Snowball 67B-A2B configuration reaches 603,327 tok/s and 19.46% median analytic MFU on 64 H100s, versus 543,490 tok/s and 17.53% for its matched 1F1B control. Command buffers remain disabled because JaxPP's device-ragged pipelines still deadlock during NCCL multi-communicator initialization #7655. On the profiling side, Hall's #8691 recovers JAX named-scope paths from GPU XPlane events, expanding each pipeline stage's profile summary from 3–4 hierarchical regions to 89–135, covering MoE routing, expert execution, and attention. Will Held upgraded the CUDA 13 attention stack to flash-attn-4 4.0.0b28, CUTLASS DSL 4.6.2, and QuACK 0.6.4 in #8741, pulling in the upstream SM90 sparse backward synchronization fix that caused intermittent NaN gradients on H100. Romain Yon fixed Megablox ragged-dot gradient tiling on TPU in #8719 and enabled model-sharded reference attention in #8709.

An early context-parallelism probe #8754 trained the hero shape at sequence length 262,144 on one NVL72 rack with EP16 and four-way context parallelism: 8.78% MFU with 18% expert-assignment drops still rising, confirming that the capacity factors tuned at sequence 4,096 do not carry to long context. Additional infrastructure work included bounding the TensorStore decoded-chunk cache from 125 GB to 1 GB (#8642, Russell Power), sharding expert weights over an auxiliary mesh axis to reduce per-device memory #8674, and upgrading to JAX 0.11.1 after fixing its dot_general sharding-rule break in Grug attention #8715.

Will Held stood up a standalone H100 scaling ladder #8673 and confirmed that Hopper loss tracks the GB200 ladder within noise. The initial launcher covers d512 and d768 rungs at 791 tokens per active parameter with EP8 placement and the Triton Sonic backend for dropless evaluation; d384 was dropped because its shallow depth put it off the log-linear trend. A follow-up PR #8683 extends the H100 ladder to d1024 and d1536, and the d2048 rung launched on 192 H100s as EP8 times 24 replicas at global batch 1,536 #8679. Cross-validating the scaling recipe on Hopper hardware strengthens the five-rung loss predictions that anchor the hero run.

James Thewlis identified a bug in the MuonH optimizer where the Frobenius norm for 4D expert-weight tensors was computed over all axes including the expert dimension, rather than the intended per-matrix last-two dimensions #8621. John Larry Dial reran the d768 scaling rung with the per-expert norm fix and measured 3.018 versus 3.015 Paloma macro loss for the original config — within noise. All ladder tuning and the live hero run used the original norm, so the run continues unchanged. A PR with the corrected norm axes is open #8749.

On the context-extension front, Will Held launched controlled experiments varying the long-context data skew during the 262K context extension. The 2x treatment #8734 completed on v4-2048, and a 4x treatment #8790 is now running — both branch from the same step-156,000 checkpoint and preserve the qk=1.75 configuration while oversampling documents longer than 64K tokens. Separately, Kaiyue Wen closed the d512 learning-rate sweep #7856 with a constant-LR extension covering five token budgets from 30x to 600x active parameters. Linear decay consistently beat constant LR by 0.15–0.25 Paloma points, with the gap narrowing at longer budgets; the fitted optimal constant LR follows a power law of LR* = 8.21 × 10⁻³ × (tokens/1B)⁻⁰·⁵⁶.

The infrastructure-as-code stack completed a long-planned consolidation. #8458 made the single marin Pulumi stack the sole owner of all GCP IAM grants, and #8650 replaced additive member resources with one authoritative binding per target, role, and condition — a live audit of 57 targets, 197 bindings, and 736 members found zero managed-binding drift. #8462 retired the legacy permissions stack, #8700 moved Identity-Aware Proxy access to per-service bindings, and #8649 added a read-only skill that traces Pulumi diffs back to the pull requests that caused them. Rafal Wojdyla, Russell Power, Romain Yon, and Will Moss drove the rollout; a shared marin-deploy CLI #8537 now routes all service rollouts — Echo, Grafana, Loom, and others — through one command, with Iris and Finelog rollouts following in #8652.

Iris's scheduler gained a new SYSTEM tier above PRODUCTION #8495, so hero and infrastructure tasks preempt everything else after worker loss instead of competing with unrelated admin jobs. A Kueue Topology-Aware Scheduling (TAS) bug left interactive-priority pods gated for 23 minutes while batch work held the cluster #8781; Russell Power pinned Kueue 0.18.3, enabled same-cycle TAS recomputation, and derived binding quotas from cluster configuration so preemption receives quota pressure before victim selection #8782. A coscheduled SkyRL job on CoreWeave exposed a separate gang-recovery gap: when the Ray head was preempted, worker tasks that had exited cleanly were marked terminally succeeded, blocking the gang from reforming #8774. The fix returns every coscheduled task to pending on retriable failure or preemption. On TPU, a v4-2048 slice registered only 255 of 256 workers because one host's GCP metadata response was delayed; Will Held restarted the lagging worker to unblock the job #8743, and #8745 now aborts the probe when metadata is missing so the failure surfaces immediately.

Finelog and Levanter telemetry saw structural changes to contain the hero run's write volume. #8571 separated training metrics from service telemetry — Levanter produced 66.2% of retained rows in the production snapshot, and a 10.08 GiB replay projected a 68.6% compressed-byte reduction from process-zero-only publication, lower cadence, and narrower metric selection. #8678 stopped nonprimary JAX processes from publishing step, progress, and heartbeat rows, eliminating the per-process multiplier. #8707 partitioned levanter.metrics online by exact run_id, so queries against one run prune all others at read time. On the reliability side, a Rust panic in the cw-us-east-02a Finelog server poisoned the namespace mutex and left the process alive but non-functional; #8773 now aborts the server on any panic, and #8786 bounds each disk-backed namespace to 200 MiB of Arrow buffers so a persistence stall cannot exhaust host memory.

Canary and monitoring infrastructure kept pace. The TPU ferry moved to v6e as its default variant after v5p capacity became unavailable #8682, and the CoreWeave loss ceiling was recalibrated from 8.0 to 10.0 to match the new 9.9M-parameter canary model #8698. #8758 gave the canary triage agent file and temporary-file tools it had been denied, ending permission-gated incomplete results. On the observability side, #8714 scoped each node agent's DCGM exporter discovery to its own node — the previous cluster-wide list walked 386 pods per collection pass — and #8712 moved node-metric scraping from the apiserver's nodes/proxy subresource to the local kubelet's loopback endpoint, removing a cross-cluster hop from every sample. The Grafana capacity dashboard now shows each active job's Kubernetes priority and links to its canonical Iris route #8643.

The Zephyr shuffle engine received a substantial set of merged improvements this week. Will Moss unified the memory model into a shared budget with calibration-based knobs for read and write fractions, collapsing the in-memory-versus-external-sort decision into a single fan-in value #8204. A companion change replaced the scatter sidecar helpers with a typed _Sidecar that owns msgpack read/write, fixing the serialization format tracked in #6005 #8195. On top of that, schema caching landed for scatter Parquet chunks: reducers now read Arrow schemas from the mapper sidecar instead of fetching one object-storage footer per input, and the bounded multi-pass merge was consolidated into a single path #8677. The ZephyrContext module split #8181 rounded out the shuffle cleanup by breaking the monolithic execution.py into separate coordinator and worker modules.

Reliability work addressed a canary failure in the datakit smoke ferry: Yunpeng Xia added retry logic for memory-store operations lost when an actor restarts mid-run #8752, fixing the verify_fuzzy_dups step that had been dying on NOT_FOUND after preemption #8615. Separately, Jah-yee opened a fix for ThreadedBatchWriter.close() to propagate writer failures instead of blocking indefinitely #8665, and Akshay Pal opened a fix for empty reducer shard reads #8772.

Rafal Wojdyla continued building out the fuzzy-dedup verification pipeline with a three-PR stack: solve complete fuzzy-duplicate clusters using directional word n-gram containment #8723, materialize cluster text with a versioned manifest and MinHash partitioning for oversized components #8724, and verify materialized clusters in bounded distributed map tasks #8725. On the data-catalog side, Will Held documented immutable reconstruction revisions for three pre-staged Datakit sources—Focus Crawl, Nemotron Code v1, and v2—pinning the exact extractor, resolver, and Software Heritage graph configuration needed to rebuild each artifact #8759. Looking ahead, issue #8732 tracks replacing Polars with DataFusion inside Zephyr's shuffle internals; early benchmarks on 13 SQL-migrated pipelines are reported in the tracking PR #8651.

Romain Yon merged #8720, bringing full-size Hero GrugMoE serving to Marin on H100 GPUs. The PR pins Marin to an immutable vLLM release built from vllm#51 and enables the qualified DP8/EP8 path with batch-invariant execution and the built-in sampler. This closes #8644, which Will Held filed to track schema-2 checkpoint loading for the hero architecture -- including LatentMoE projections, shared experts, depthwise causal SConv, and the interleaved fused half-RoPE convention. Qualification passed 1,524 Marin tests and 33 focused vLLM overlay tests on H100. Romain announced on Discord that the implementation has been tested on the largest ladder checkpoint and shared a starter configuration.

Alongside the serving work, Romain overhauled the inference observability stack. #8629 replaced the previous first-1,024-sample policy for vLLM Prometheus forwarding -- which dropped 23–61% of samples in observed serves -- with explicit family-based selection of twelve named metric families. #8573 separated request-level time per output token (TPOT) from inter-token latency, fixing a dashboard that had been conflating the two distributions. #8614 repaired the Finelog query planner for the vLLM overview by explicitly naming output columns across union branches, and #8727 added collector health panels so operators can distinguish serving degradation from telemetry gaps. #8438 fixed a false-green vLLM GPU nightly that masked a failed Iris build behind tee's exit status.

On the TPU side, Rohith Kuditipudi closed #7085, which tracked ragged paged attention v3 (RPA v3) failures on TPU v4 workers that blocked cross-region evaluation jobs.

Ahmad Qamar merged two fixes that unblocked SkyRL runs launched from the Marin graph. In #8509, a missing shell quote on retention-override paths containing ttl=14d caused Hydra to reject the second = sign, killing every graph-dispatched RL run at node allocation. In #8510, the SkyRL entrypoint registry switched from module paths to named keys, so the old skyrl_train.entrypoints.main_base was silently rejected; the fix maps through the new standard key and surfaces the last lines of stderr in dispatch failures for faster debugging.

Separately, moujf opened #8722 to validate SkyRL role-plan consistency at init time: it catches n_samples_per_prompt < 2 (where every Group Relative Policy Optimization (GRPO) advantage would be zero, so the run silently learns nothing) and batch sizes not divisible by the sample count. Two Fray actor-framework fixes also landed in review: Jah-yee fixed LocalActorGroup.wait_ready consuming all actor handles and breaking subsequent discovery #8668, and Yunpeng Xia corrected operation recovery so that serialized application errors are no longer masked as ActorUnavailableError #8767.

Benjamin Feuer published the final report for the Snowball 67B-A2B non-agentic math RL campaign and closed #7786. The recommended recipe (experiment E6) improved AIME24 from 17.67% to 27%, MATH-500 from 64% to 78%, and OlympiadBench from 12.67% to 20%. A notable negative result: AdamW outperformed the Muon-H optimizer used in Snowball pre-training, learning faster and reaching a higher peak training reward. Expert-parallel (EP) training at EP=8 on Grug models showed gradient-norm instability tracked in MarinSkyRL #335. Feuer also noted that training reward does not reliably correlate with downstream eval scores, making comprehensive metric tracking and automatic judges important for future runs.

Russell Power completed a two-round curriculum-RL experiment (#8765, #8769) measuring whether dynamic prompt-sampling strategies improve SkyRL GRPO training efficiency at a fixed compute budget. In round one, five arms trained Qwen3-0.6B on a six-bin graded math pool; grade-adaptive sampling matched baseline quality at roughly 53% of naive's token spend, confirming that curricula buy token efficiency rather than a higher endpoint. Round two added DAPO (Dynamic Allocation Policy Optimization) filtering and procedural-task bins across seven arms. The best performer was thompson-dapo, which reached 0.434 on MATH-500 and 0.456 on GSM8K, exceeding every other arm on both benchmarks.

In related work, Feuer closed a Qwen3-235B-A22B smoke test #8653 confirming that the SkyRL/Iris training path holds at 235B parameters on a third-party MoE—six optimizer steps on 96 H100s with no OOM or rank drop—though the run produced no learning signal because 55% of samples hit the 8,192-token generation cap. Marianna Nezhurina opened #8481 to investigate whether short-horizon agentic RL on small models teaches transferable skills (tool use, context management) or only task-specific ones.

Mark Muchane registered the five Code Alchemy training subsets as independent Datakit sources in #8687, hydrating 93 million rows from the pinned release and tokenizing them with the Marin tokenizer across 512 workers. The five subsets—code-dev, code-dialogue, code-enhance, code-qa, and code-trace—total 917 billion tokens, with code-dialogue alone contributing 510B. A separate issue #8670 tracks adding the Swallow Code dataset from Tokyo Tech as another code source. On the reproducibility side, Will Held opened #8764 with a guide that traces Datakit catalog names through provider IDs, pinned revisions, and normalization steps so outside researchers can rebuild the full pretraining source pool from the original providers, and pinned the exact extraction commit for the Common Crawl focus crawl comparison in #6570.

Michael Ryan posted a weekly sync on the spec-driven Common Crawl curation effort #2351, reporting that the final distilled pipeline is ready. The three-stage cascade—a fastText language filter, a 90M-parameter pooled classifier, and a 10M-parameter Ettin classifier—covers 91–93% of FineWeb-Edu, DCLM, and the previous Marin high-quality pool while retaining far more tokens than earlier pipelines. Data mixing experiments on OLMo base evals show the new and old specs performing close together, but the new spec yields more tokens and therefore more headroom for further mixing gains. The target timeline is an ICLR submission by end of September and a 30T+ token dataset ready by October. In Discord’s #data-mixing channel, David Hall and Rafal Wojdyla discussed releasing detailed metadata so researchers can reconstruct training data from original providers without rehosting restricted content, and Held noted that several sources (such as Nemotron) forbid redistribution outright.

Separately, Ahmed Ahmed closed out the mid-training recipe identification effort #4547 with a published dossier of candidate mixes, and Held flagged a version bump to Cutlass, QuACK, and Flash Attention in the #code-review channel to fix frequent NaNs that appeared during data mixing runs on H100s.

David Hall added generative MRCR (Multi-turn Retrieval with Conversational Reasoning) scoring to Marin's served evaluation path in #8706, letting Evalchemy run pinned OpenAI MRCR tasks against Grug checkpoints with nonce-gated SequenceMatcher scoring across multiple context lengths and needle counts. Four 67B checkpoint exports completed canonical generative MRCR on a deterministic 299-example, 18-cell selection #8702. A separate oracle-normalized paired measurement at 262K tokens #8701 confirmed that the context-extension checkpoints retained most of the available needle benefit, with the deployable qk=1.75 checkpoint performing cleanly.

Hall also prototyped RiffleEval #8655, a new long-context evaluation that interleaves chunks from same-genre documents in a riffle-shuffle pattern and measures autoregressive bits-per-byte. The paired BPB matrix across 42 checkpoint-length cells is complete, and an anonymous-marker variant (removing stream IDs) showed that stripping per-document identity markers measurably degrades prediction, indicating the model tracks document boundaries through the markers rather than content alone. On the data-quality side, Benjamin Feuer sealed all 103 TaskTrove v3.42 cells in a five-model panel evaluating every source for data-quality issues #8602, and Mark Muchane began scoping a move from single quality scores to multi-axis data labeling #8453, drawing on recent work such as RegMix and MATES.

Community contributors continued designing new evaluation frameworks. Ayush Sunil Munot and Mrinal Kumar discussed a think-budget-calibration eval #8357 that measures whether a model allocates reasoning tokens in proportion to item difficulty, resolving questions about how to obtain oracle budget labels without circularity. On Discord, Mrinal Kumar noted that reusable benchmark pieces belong in Evalchemy while Marin handles experiment-level orchestration such as checkpoint sweeps and metric aggregation.

A series of checkpoint-performance improvements landed for the hero run. #8599 stages unsliced GPU checkpoint shards through disposable pageable host memory so JAX does not retain transfer buffers on donated training arrays. #8626 extends that path to replica-split slices, and #8647 overlaps host staging with asynchronous TensorStore opens so the two costs run in parallel. #8663 publishes per-phase checkpoint timing, staged bytes, and peak RSS to Finelog so the gains are measurable in production. #8662 removes a redundant write-mode TensorStore open that doubled the per-array cost in JaggedArrayStore.

The Grafana training dashboard gained several hero-run panels. #8619 from Rafal Wojdyla adds active-execution time, active share, and an Attempts table linking each Iris execution. #8620 fixes the percent-encoded Iris links and cleans up the crowded status strip. #8623 from Larry Dial adds a progress-efficiency tile showing the fraction of an ideal run's throughput. #8660 repoints the infra dashboard loss, Paloma, and MFU panels at the 535B-A23B W&B report. #8575 from Mark Muchane routes Grafana hero alerts through a single operator-behavior registry.

The agentic-lint pipeline was overhauled. #8740 from Russell Power replaces the monolithic Claude PR review job with two independent Loom sessions (correctness and lint-catalog), restricted to trusted same-repository pushes. #8746 from Will Held adds a bounded maintainability pass that selects up to four high-impact changes and assigns a reviewer to each. #8735 migrates code-health telemetry from W&B to append-only Finelog namespaces, and #8736 gives the lint-review CI job the Finelog credentials it had been missing. #8641 isolates Codex review agents in ephemeral read-only sandboxes and strips parent session credentials.

Echo's federated-search service saw three improvements. #8717 bundles the dashboard's JavaScript, CSS, and font subsets into a single HTML response, eliminating five parallel asset fetches. #8692 reports per-stage query timing through the Server-Timing header and reduces rerank latency. #8777 lets echo get accept numeric file grading keys from search results instead of requiring repository-qualified source IDs.

Matt Wittmann upgraded JAX to 0.11.1 across Marin, Levanter, Iris, and Fray in #8716, supplying the ragged collective flags needed for expert-parallel dispatch. The upgrade required adapting Grug attention to the new dot_general sharding rules #8715. #8472 adds merged cache catalogs to Levanter, replacing per-component ledger reads with a single versioned catalog for data mixtures with hundreds of components. #8617 from Rafal Wojdyla adds a zero-argument production trigger for the current d6144 hero run.

Russell Power consolidated the Iris and Finelog agent skills into two compact entry points #8636, made coordinating GitHub issues optional for multi-session research #8635, and opened a new epic for auto-improving agentic lint rules from PR review feedback #8738. Finelog work continued with an issue for native object-storage offloading #8737 and a fix for phantom catalog rows that retried archive uploads forever #8784.

On the scheduling side, Iris bugs surfaced and were tracked: a v4-2048 slice registering only 255 of 256 workers #8743, a preempted Ray head whose workers stayed terminally succeeded #8774, and interactive-priority pods failing to preempt batch under Kueue TAS lack-of-fit #8781. An external contributor flagged that the MuonH optimizer's Frobenius norm includes the expert dimension for 4D inputs #8621. Mark Muchane opened dataset requests for Swallow Code #8670 and Code Alchemy #8671. Dmitry Duev dropped the deprecated RunDisabled type alias from the W&B integration #8693. The nightly cleanup bot removed dead code paths across Levanter and the GCP bootstrap stack in four separate PRs.

Six external contributors opened PRs this week. Yunpeng Xia landed #8752 (retry lost memory-store operations in Zephyr after actor restart) and has two more open across Fray and Grug. Jah-yee opened three PRs touching Fray actor handles, Iris port preservation, and Zephyr writer failure propagation—all distributed-systems hardening. chiang21fcb submitted a fix to compute MuonH hyperball norms over matrix dims only #8749, 牟剣峰 added SkyRL role-plan validation #8722, AkshayPal fixed empty reducer shard shuffle reads #8772, and Dmitry Duev cleaned up a deprecated wandb type alias (#8693, merged).

On Discord, Benjamin Feuer posted the Snowball 67B RL report in #reinforcement-learning—Snowball trains stably under RL, with AIME24 improving from 17.67% to 27% and MATH-500 from 64% to 78% (full report linked to #7786). In #code-review, Matt Wittmann shared #8684, making ragged all-to-all the default expert-parallel backend for the hero run, which pins a patched PJRT wheel from Marin’s new XLA fork. Percy Liang announced a hero-run panel for the Sept 1 community meeting, where the team behind the run will take questions.

Forty-five people introduced themselves and roughly 290 more joined silently via the welcome room. Among those who introduced themselves: Mattia (FAIR, Edinburgh PhD—experience with pre-training up to 3B and RL post-training integration at 50B scale) brings context on the RL post-training pipeline that Snowball is exercising. Linyi Li, an assistant professor at Simon Fraser who previously worked on code-LLM pre-training at ByteDance Seed, intersects with ongoing data-selection and mid-training work. Other backgrounds span Together AI inference (Sirish), Prime Intellect RL (Sebastian/snimu), CMU efficient-architecture PhD (Zikun Li), Cornell linguistics PhD (Qi), and INSAIT RL-for-LLMs PhD (Vatsal).

The week’s shared research centered on hyperparameter transfer: in #scaling-laws, Will Held pointed to a survey connecting muP and related scaling heuristics across depth, batch size, and token horizon.