Marin: Week of August 10th summary

The expert-parallel 64-rank (EP64) one-shot crash that had blocked ragged all-to-all on GB200 racks was root-caused by Matt Wittmann to a hard-coded 32-peer bound in XLA’s barrier kernel (kMaxPeers in MultiGpuBarrierKernel): a 64-rank clique wrote past the signal region in symmetric memory, producing CUDA_ERROR_ILLEGAL_ADDRESS at the first training step. The same bug explained the #8313 NCCL 2.30.7 regression. A one-constant rebuild wheel fixed it, and the issue was filed upstream as openxla/xla#47283. With the patched wheel, the hero shape (E384, top-8, expert width 3072) completed its first EP64 rack run at 20.20% MFU #8077. Nineteen configuration arms explored the launch-knob space #8317; QuACK GEMM cluster-shape tuning pushed the stock ragged path to 21.82% MFU / 241k tokens/s. In parallel, the experimental Marin EP fused backend #8311 validated multi-process transport end-to-end on the NCCL device API and measured EP64 at 17.75 s/step with the fused puts costing 1.16 s/step versus the ragged path’s 2.30 s. Three upstream XLA blockers were filed with validated fixes (openxla/xla#47406). On the doubled-expert front, Rafal Wojdyla selected the three-stripe destination layout for the 8-of-384 MoE and completed a 200-step stability run on one GB200 rack with no OOM or divergence #7279.

Several numerics and runtime fixes hardened the hero template. Wittmann found that the fused cross-entropy forward had been accumulating bfloat16 logits while the backward recomputed them in float32 — a silent mismatch on every step of both Grug MoE heroes; the float32 accumulation fix landed in #8217. Checkpoint writes were split across all ranks #8140 instead of bottlenecking on process 0, which had been staging ~10 GiB of replicated state alone on the EP hero mesh; the split, bounded at 16 GiB per process, was validated on the 546B EP model. XLA command buffers were re-enabled at the default capture set with collectives kept eager #8277. The EP hero now runs one JAX process per GPU #8209, bypassing the auto-PGLE profiling-session collisions that had been killing gangs under the process-per-node layout. Larry Dial refined the hero shape #8132: routed expert width aligned to hidden_dim (6144), sliding window extended to 2048 tokens, global attention every fourth layer. A 640-GPU hardware smoke test ran 16 hours and 125 billion tokens on the 535B-parameter model without incident.

The data pipeline is converging on the hero-run artifact. The Harrier embedding backfill completed for all 292 DataKit sources, including fuzzy-duplicate documents the canonical run had excluded #8162. Decontamination was validated against the pinned Artificial Analysis Intelligence Index v4.1.1 corpus (4,165 records across nine benchmarks) #8168, and the required corpus was registered in #8189. A fuzzy-verification audit #8155 across 500+ document comparisons confirmed the current dedup rules are precise; verified duplicates were registered for final store construction #8319. Will Held registered the Harrier domain-cluster assignments for all 292 sources #8285, and hero data stages are now resolvable by source name #8242. A new burndown issue #8233 tracks the remaining launch gates across architecture, runtime, and data.

Benjamin Feuer completed the first end-to-end cold-start supervised fine-tuning (SFT) pipeline for a Marin 67B-A2B checkpoint, running three ordered stages — Chat, Thinking, and Agentic — on the Snowball 5.7T-token cooldown #8225. Every completed stage was exported to a public laion Hugging Face repository and evaluated on the standard benchmark cohort. The best checkpoints reached 89% on GSM8K (0-shot), 78% on MATH500, and 38% on AIME24. On agentic evals using the OpenThoughts-TBLite dev set, the 5.7T Nemotron Terminal Stage 3 checkpoint earned 20.4% mean reward (56/300 full-reward trials), compared with 1.3% for the OpenCode path. A full writeup documents the pipeline as a reusable blueprint for future Marin cooldowns. Two stacked PRs — #8171 (fixed-EOT data path) and #8172 (launch stage restoration) — preserve the recovered artifacts and SFT launch configurations.

The earlier Grug agentic SFT non-termination investigation #7743 closed out. Will Held re-evaluated the checkpoint after rendering supervised turns with the tokenizer’s native <|eot_id|> terminator: 285 of 300 trials now reach a normal turn boundary, confirming the end-of-turn fix resolves non-termination. However, the fixed-EOT checkpoint solved only 11/300 tasks (3.7%) versus 57/300 (19.0%) for the Nemotron Terminal control, indicating that the data mixture — not just the terminator — drives agentic capability. Benjamin Feuer traced S3 provenance across three distinct training lineages and recovered the original July source artifact, making the full recipe reproducible.

On the reinforcement learning side, Benjamin Feuer reported that after patching numerics and training-stability issues in MarinSkyRL, long-horizon RL runs are substantially more stable. He is now experimenting with 400-step / 10-epoch runs instead of the previous 80-step / 2-epoch arms, with early reward trends looking promising. A reproducibility package for the TaskTrove X10 and X2 arms was published #7785. Separately, Russell Power opened a storage audit #8290 identifying roughly 114 TB of cleanup candidates under RL and agentic job paths; #8296 merged to route SkyRL checkpoint, trace, and session state to lifecycle-managed temporary storage, keeping only canonical terminal exports on durable paths.

Iris’s Kubernetes scheduling saw several improvements this week. Co-scheduled GPU workloads now receive explicit admission priority ordering within each band #8014, and same-band production preemption was disabled so CPU-only production tasks can no longer be evicted by GPU gangs #8307. Node agents now collect per-task CPU and memory stats directly from kubelet resource metrics, working around the missing Metrics API on certain CoreWeave clusters #8214. Stale task cache directories are reclaimed on Kubernetes nodes #8286, and federation I/O stalls are now isolated from the scheduler loop #8259. On the observability side, Russell Power added CoreWeave-style Grafana node dashboards with per-GPU utilization, temperature, NVLink throughput, and fault tracking #8064, while Rafal Wojdyla added CoreWeave S3 storage quota alerts #8238 and storage usage reports #8264.

Will Held landed the Harrier embedding pipeline #7998, which embeds canonical documents from the global fuzzy-dedup artifact using microsoft/harrier-oss-v1-0.6b across all 292 normalized sources. On the Zephyr side, Russell Power capped scatter chunks at 512 Parquet row groups to keep reducer scans bounded #8267, and Will Moss made chunk-file schema reads concurrent for faster wall-clock schema unification #7996. Rafal Wojdyla split oversized documents into chunk_index rows to prevent Parquet write failures on documents exceeding the signed 32-bit page-size limit #8100, and added Parquet previews to fsutil so that cat, head, and the browser render Parquet files with schema and row-group metadata instead of a binary placeholder #8192. FineStore gained transactional manifests and cache adapters #8210, publishing archives through a conditional HEAD pointer with atomic multi-table transactions.

The eval stack saw meaningful scoring improvements: every eval score now carries a confidence interval that accounts for items a run did not grade #8122, preventing runs that skipped items from outranking those that graded them. The evaldash began its FineStore v1 migration with a legacy read view that keeps older eval images working during the rollout #8324. Benjamin Feuer preserved benchmark generation limits during eval retries #8315. On the inference side, the GPU vLLM pin advanced to the cu130 release #8220 and the TPU stack refreshed to tpu-inference v0.26.0 #8207. HuggingFace model caches are now pinned to immutable commits rather than mutable branch or tag references #8159, and eval serve host memory is automatically sized from the checkpoint #8139.

Matt Wittmann optimized Haliax layer initialization by building ArrayStacked layers in one vmapped trace instead of tracing per layer, reducing HLO (High-Level Operations) size proportional to model depth #8206. Tim O’Donnell made Levanter’s RoPE (Rotary Position Embedding) export backward-compatible by emitting top-level rope_theta alongside rope_parameters, so Transformers 4 no longer silently falls back to theta 10000 #7801. Portable fast-tokenizer metadata export was added to normalize generic tokenizer class names across Transformers 5 #8163. JAX distributed shutdown is now handled on process exit for every Levanter entrypoint #8295. SkyRL state management was tightened to route checkpoints, raw Harbor traces, and retained trajectories to temporary storage with a rolling resume depth capped at five #8296. An open contribution adds a depthwise causal 1D convolution kernel to Levanter with a hand-written Pallas TPU implementation #8331.

Dependency updates were automated via a dedicated GitHub App #8176 that runs updates without operator review when identity, scope, and CI prove safety. A weekly fork-ferry migration coordinator #8245 now launches per-fork sessions for TPU vLLM, Evalchemy, and Harbor. CI gained affected-test selection for Levanter accelerator suites #8170, with a fallback to full suites when selection cannot determine scope #8177. Russell Power batched XLA autotune cache files into compressed archive shards for concurrent write safety #8178. Rafal Wojdyla’s xprof tool now opens hosted profiles directly in Trace Viewer #8158. Echo, the internal search agent, gained a conversation tab and search-result feedback collection #8291.

Matt Wittmann drove expert-parallel MoE forward on GB200 with seven merged PRs, including XLA command buffers for the hero EP run #8277, one JAX process per GPU #8209, a vmapped ArrayStacked build that cut compile times by 75% #8206, and a fused cross-entropy logit-dtype correction #8217 that had been silently accumulating forward logits in bfloat16. He also filed issues tracking NCCL 2.30.7 regressions at EP64 and PGLE/CUPTI contention on GB200. Will Moss continued Zephyr data-pipeline work with concurrent chunk-file schema reads #7996. Mayank opened a depthwise causal convolution kernel tested on TPU v6e #8331 and reported linear attention is next. Tim O'Donnell merged rope export in transformers-4 format for MarinFold’s HF pipeline #7801.

In #code-review, Matt Wittmann surfaced a subtle correctness bug: the fused cross-entropy loss kernel accumulated forward logits in bfloat16 while backward used float32, and the fix was already running in hero templates. In #architecture, Kaiyue Wen proposed reusing midpoint activations for K/V to accelerate prefill #8196, building on the YOCO paper; Larry Dial noted the gain depends on where the layer cut falls. Franziska Weindel laid out a detailed experiment plan for OT-Agent SFT in #openthoughts-next, covering teacher/student interactions, task filtering, and trajectory-filtering alternatives including TermiGen.

Nine people introduced themselves this week, spanning genomic language models (Jean du Terrail, Living Models), robotics VLAs (Denis, Sereact), nonprofit LLM training (Shenyang Huang, LawZero), biological foundation models (Jason Krone, Biohub), computational biology (Jonas, NIH), open model auditing (Eliel, DePaul), and distributed training at Hugging Face (Amy). Richard Liaw, co-author of Ray, joined to follow the project. Jason Krone’s focus on biological foundation models intersects with MarinFold’s protein-structure pretraining, and Amy’s experience maintaining the transformers library brings context on the HF model-export pipeline.

The research links shared this week clustered around scaling-law refinements for small-scale predictions, data-mixing strategies favoring continuous blending over staged phase changes, and agentic trajectory curation for SFT data.