Marin: Week of August 17th summary

The 535B-A23B EP64 (64-way expert parallelism) MoE hero run launched on August 19 across 704 B200 GPUs (11 NVL72 racks on CoreWeave), training on 18 trillion tokens of the Harrier data mix. Larry Dial posted the full model specification—d6144, 384 experts top-8, pooled-wave all-to-all transport—and a five-rung scaling-ladder analysis in #8435: four clean rungs and a d2048 rung that reached 81% yield a power-law prediction of approximately 2.04 dropless Paloma macro-loss at the hero’s 2.70e24 training FLOPs. Will Held finalized the Harrier mixture on a fuzzy-deduplicated datakit store #8427 with monotonic PI (proportional importance) weights #8452, and added uncheatable evals to every ladder rung #8425. Percy Liang announced the sprint in Discord; a dedicated #hero-run-2026 channel opened for public tracking.

The first days hammered the checkpoint pipeline. Host OOM kills recurred roughly every three hours during saves #8506. Russell Power walked through malloc_trim #8540, jemalloc #8550, staging-budget raises from 4 to 16 GiB per process #8514, and write-replica fanout from 64 to 1024 #8486, while Mark Muchane routed checkpoints to cluster-local storage #8559 and hero paths were locked against deletion #8504. A separate restore stall blocked 703 GPUs for four and a half hours when one rank’s TensorStore read hung #8534. Matt Wittmann root-caused it to an unbounded post-restore barrier and landed a timed replacement #8538; Russell Power added a TensorStore low-speed abort #8584. Replica-aware restore now reads each unique checkpoint shard once and distributes via JAX reduction, cutting benchmark S3 traffic by 57.7% #8589. Rafal Wojdyla built a training-run status dashboard #8479, wired three hero-health alert rules through Slack #8568, and pulled the full loss curve from W&B after Finelog evicted early steps #8566.

Two MoE transport alternatives reached parity with the production pooled-wave backend. Matt Wittmann’s ragged EP64 rewrite, tested in a checkpoint-restored A/B from the live hero’s step-6000 state, delivered 22.46 vs 22.52% MFU with 0.015% drops against 2.67% and a 10 GiB lower runtime device peak #8549 #8317. Mark Muchane’s Mixture-of-Kittens implementation matched pooled-wave at EP64 hero shape—237,613 vs 237,310 tokens/s—with zero drops #8108. On the kernel side, David Hall fixed silent FP8 quality loss from gradient accumulation dropping the amax history to the last microbatch #8360 #8365. Larry Dial merged the combined hero-kernel stack: bf16 backward GEMMs in fused cross-entropy (a 2.33x speedup), fused ShortConv, and hoisted router all-reduces #8385. Will Held’s Pallas short-convolution experiments passed Gate 1 at both d512 (1.242x combined speedup) and d768 (1.131x) #8377.

The full Snowball 67B-A2B cold-start SFT checkpoint chain is now public on Hugging Face. Benjamin Feuer published the base cooldown at step 105149, the Chat stage at step 257, Thinking at step 630, and both Agentic branches (OpenCode and Nemotron Terminal) at steps 1000 and 1888, covering every durable checkpoint the training runs retained. A community request for intermediate post-training checkpoints #8341 was resolved by the release. On the source-run side, Larry Dial reported that the 67B-A2B at 10T tokens has reached Marin’s lowest Paloma macro loss ever, noting the comparison is not strict since this phase runs at 65k context length. The loss slope accelerated after entering mix2 and again on longer context, with 2T tokens now on mix2 and 1T of that on longer context. A W&B report tracks the full run.

Ahmad Qamar brought the Snowball E6 RLVR (reinforcement learning with verifiable rewards) rerun with MuonH to an end-to-end completion from the Marin graph. MuonH reached a learning signal where the earlier arm E10-B had been blocked on OOM around step 5. The run used a reduced geometry (5 nodes, batch size 32, 6 steps on CoreWeave) due to capacity constraints, so reward and throughput are not directly comparable to the original #7786. Four fixes were required: #8509 quoted Hydra retention overrides so the RL stage no longer died at launch, #8510 surfaced launcher stderr in error messages, #8565 added ExternalModel and ExternalDataSource so SkyRLSpec can reference published checkpoints and datasets, and MarinSkyRL#423 fixed job-status recording. Benjamin Feuer flagged a learning-rate footgun in prior attempts: Muon-H matrices and the language-model head used the configured 1e-5 rate while embeddings, router weights, and norms fell through to Adam at an implicit 6e-4, which MarinSkyRL PR #405 corrected.

Separately, Qamar opened a coordinated PR set for RL observability and identity: #8562 and #8551 add a dedicated Grafana dashboard for RL runs (the existing one filters only Levanter pretraining), and #8561 and #8555 derive run IDs from the artifact’s storage address instead of a version basename, so that an SFT and a pretraining checkpoint at the same version no longer collide. On the data-selection front, Feuer published three experiment results informing what to train on: a 48-cell harness/context/summarization ablation finding that harness preference is model- and dataset-dependent, a cross-cluster HPO (hyperparameter optimization) study showing a wide tolerance band for RL hyperparameters but that HPO matters for long-run stability, and a five-model reward-delta analysis identifying 24+ TaskTrove sources with monotonic, substantive gaps across Qwen3 Coder, Qwen3.5 122B, and GLM 5.2. A culminating five-model data quality panel on TaskTrove v3.42 also found that model capacity is not predictive of agentic benchmark performance.

Iris received a wave of reliability and operability work. Russell Power and agents landed Kubernetes disruption evidence retention (#8601, #8604), a fix for kubelet resource metric decoding after the Kubernetes client v36 upgrade broke Prometheus parsing (#8582, regression from #8381), and grace-period handling for transient unknown pod phases so the hero run stops burning failure quota on ephemeral Kubernetes hiccups #8587. #8463 fixed a bug where Iris’s callable runner swallowed fatal exceptions, letting the clean-exit hook mark failed jobs as succeeded. The workload client and CLI were normalized: #8364 and #8399 expose immutable Job/Task/Attempt handles with a unified verb vocabulary (job cancel, job complete, task preempt), replacing ad hoc commands. #8424 preserves terminal causes like OOMKilled through job summaries. Ryan Williams re-landed the client-freshness gate (#8522, #8523), anchoring the version floor to the controller’s own build so quiet weeks no longer reject clients running identical code. Romain Yon upgraded the Kubernetes client to v36.0.3 #8381 and excluded GitHub Actions credentials from workspace bundles #8518.

Observability improved across Grafana, Finelog, and fleet telemetry. Russell Power cut Finelog query latency by orders of magnitude: structured telemetry queries dropped from 8.5 seconds to 200 milliseconds through index bypass and timestamp-bound planning #8379, shared-prefix IN queries now prune with a half-open range #8516, and Cluster Capacity dashboard medians fell from 838 to 244 milliseconds #8526. A new per-GPU SM (streaming multiprocessor) utilization raster replaced the fleet histogram #8513, and a cluster capacity packing view now shows live pod placement against node capacity #8423. Rafal Wojdyla exposed Levanter training status through Iris #8512 with ad hoc checkpoint support (#8544 in the hero epic), and the log viewer gained paging, time-bound controls, and bulk context expansion #8493. Finelog Kubernetes deploys moved to Pulumi #7690, contributed by Will Moss. The Evalchemy and Harbor nightly dashboards stopped flagging fast successful runs as unhealthy (#8521, #8532).

Infrastructure-as-code and storage management saw substantial changes. Russell Power moved shared GCS, CoreWeave, and R2 bucket management into a dedicated Pulumi stack #8343, replacing the old configure_buckets.py script. CoreWeave bucket policies were scoped to the Open Athena organization while denying object deletion under hero checkpoint prefixes #8588. The distributed storage scanner was generalized to cover GCS, CoreWeave, and R2 backends #8382, and fsutil rm now streams deletes while listing rather than materializing the full object set first #8355. The synthetic infrastructure probe was retired after all 38 runs since July were cancelled at timeout #8473, and the flaky GCP pull-request smoke test was removed after a 33% failure rate from TPU capacity issues #8408. Canary runs switched to a fixed 10M-parameter model with one-day outputs (#8600, #8545) and region-local placement #8605.

The data pipeline gained several fixes. Rafal Wojdyla fixed Zephyr’s object-store client leak: each shard had been creating a fresh S3 client, parking connections in TIME_WAIT until port exhaustion killed the process #8406. Zephyr per-stage counter totals were fixed #8353, CoreWeave spill reads were corrected for LOTA path-style rewriting #8482, and Zephyr’s InputFileSpec.format field, previously write-only, now actually controls the file reader #8530. FineStore gained bounded-part storage for large objects #8466. Zephyr A/B benchmarks defaulted to GCP (#8342, #8451).

Developer tooling improved across the board. Romain Yon indexed six community repositories in Echo #8448, with per-repository scoping so cross-repo search does not dilute results. EvalDash now serves from a PostgreSQL catalog with a 3.5-second startup instead of 3.5 minutes #8346. Loom sessions now submit Iris jobs as user loom instead of app #8488, and Slack sessions default to medium reasoning effort #8433. Agent skill context was trimmed by 15% #8494. Benjamin Feuer fixed Evalchemy to exclude infrastructure-error completions from task scores #8337. David Hall published the Agent MoE experiment digest covering 80 tracker experiments #7623 and fixed FP8 state accumulation across microbatches (#8360, #8365). The Whisper model’s cross-attention initialization bug, which gave cross-attention projections bit-identical weights to self-attention, was also fixed #8446.

Will Moss (Airbnb) opened #8497 adding A/B test benchmark sizing guidance for the Datakit pipeline, and a new contributor Ian Morgan submitted #8591 bounding CI system-dependency setup so slow mirrors cannot consume test budgets.

In #news, Colin Raffel, Will Held, and Jenia Jitsev debated a paper on small-scale experiment reliability for scaling laws — Will Held contrasted it with Marin’s Tensor Programs-style heuristics, while Colin Raffel noted the paper’s finding that hyperparameter sensitivity decreases at scale. In #questions, Jenia Jitsev raised the question of Marin model adoption in vLLM and HuggingFace; Percy Liang outlined a phased plan, and Romain Yon explained why Marin currently uses its own Levanter architecture for experimental freedom. In #moe, catto proposed an initialization trick to guarantee perfect load balancing on the first batch without the QB solve; Larry Dial said they would test it for future runs.

Eight people introduced themselves. Nobin Sarwar (PhD, UMBC; scientific reasoning and LLM safety), Carlos Aspillaga (CENIA, Chile; low-resource languages), Jay Zhou (PhD, USC), Abel (CS graduate, Universitas Indonesia), Evan Quintana (research intern, Sandia National Labs; AI security), and a research engineer named Alex focused on kernels. Leonard, co-founder and CTO of Living Models, joined alongside Jean du Terrail to build foundation models for living systems — a direct connection to the MarinFold protein-structure work, where Gonzalo Benegas welcomed the overlap. Lena Lincke, starting a PhD at TU Munich with a classical RL background, is focused on LLM post-training — the area where the team’s next hero run lands after pretraining.

Shared research centered on MoE scaling: a hyperparameter transfer study for a 155B-A17B MoE at 10T tokens, and a PrimeIntellect thread on the current limits of agentic research.