Week of March 2nd summary for marin-community/marin

#2836 Infrastructure: MoE Training Support

Summary: Train a 50B MoE model on GPU hardware reliably — from data preparation through sustained multi-node training with automatic fault recovery. This epic tracks all the infrastructure, data pipeline, and training work needed to get there by March 31.

Building on last week's reliability push, Iris saw its biggest week yet. @rjpower landed controller snapshot/checkpoint #3167 so restarts no longer orphan running jobs, a reservation system for pre-provisioning worker capacity #3123, #3223, and flexible device variant scheduling #3254 that lets jobs specify multiple acceptable TPU types for cross-region placement. The autoscaler was reworked with token-bucket rate limiting for scale-down #3212 and reduced lock contention under high task counts #3356. A complete logging overhaul replaced GCS-based log reads with heartbeat-forwarded logs stored in SQLite on the controller #3244, #3301, #3283, #3296, #3325, fixing dropped logs on task completion and eliminating file descriptor exhaustion. MirrorFileSystem #3258 provides transparent cross-region file access while CrossRegionGuardedFS #3162 blocks large cross-region reads. On the training side, @dlwh continued building on last week's Grug refactor with improved variant contract checks #3169, visual diff tooling for reviewing template-heavy Grug code #3127, a new modular_opt variant #3293, and MoE ring expert-parallel optimizations #3377, #3398 that closed the EP benchmark milestone #2710. Fused cross-entropy was stabilized: one production forward path #3125, miss-only autotune sweeps #3251, a backend-dispatched GMM API with GPU fallback #3256, and v4 vmem fallback stabilization #3354. @yonromai fixed Pallas GPU CE tracing on non-GB10 #3148, added NVIDIA weight tile limits #3160, and enabled S3 compilation caching #3195.

#3096 Pre-training: MoE Scaling Laws

Following last week's initial MoE experiments on v4 and v5p, @ClassicLarry ran an extensive set of scaling law experiments — replicating expert-count sweeps from the TGL paper #3182 and progressing to full isoflop sweeps #2167. Key findings: DeepSeek-style aux-loss-free load balancing outperformed traditional LBL across configurations, and at high sparsity ratios (2:128) a 4x LBL coefficient boost showed only marginal benefit. The sweep converged on a baseline architecture — 64 routed experts, K=2, with AF balancing (bias_rate=0.01) plus 0.001 aux loss — now tracked as moe_iteration_01. @yonromai added a MoE canary ferry for daily TPU regression testing #3342 alongside canary diagnostics improvements: data loader stall monitoring #3346, always-on profiling with persistent artifacts #3299, MFU gating on trailing p50 #3279, and CW canary OOM fixes #3217. @dlwh set block shuffle as the default for new Grug runs #3371, dispatched Grug through Fray jobs to fix multinode training #3269, and auto-detect multinode TPUs to set replicas and coscheduling #3233.

#3100 Data Sources for Pre-training

Summary: We will need 20T of high-quality (including / in particular code) tokens for our large MoE runs in Q2/Q3; this is the work in March that we will do to enable that.

After last week's tokenization debugging, @ravwojdyla completed Nemotron-CC tokenization at scale — processing nearly 2 trillion tokens across all 7 quality tiers at ~150M tokens/sec across 512 workers #2829. The Luxical embedding experiment #3191 kicked off to evaluate frozen embeddings as general-purpose quality/topic classifiers, with Luxical's creator @lukemerrick offering usage guidance #3049. @Helw150 added Nemotron V2 data #3317. Zephyr saw Vortex upgraded to support GCS #3268, group_by enhancements including secondary sort and generator reducers #3250, #3247, and download reliability fixes #3324, #3142.

#3192 Synthetic Data

Progress on the SFT front after last week's 0% resolve rate: @AlienKevin achieved 5/43 on the Rust subset of SWE-bench Multilingual by switching to Qwen2.5-Coder-32B-Instruct as the student model and using the exact SWE-smith hyperparameters #2956. A TRL sanity check on Modal confirmed the earlier Qwen3-8B repetition issues were not a Marin-specific bug. On OpenThoughts4, @moojink completed follow-up experiments with the larger Qwen3-235B-A22B teacher #2262 — surprisingly, it showed only modest advantage over 32B for Llama3.1-8B-Instruct students, suggesting diminishing returns from teacher scale.

Other Changes

@gonzalobenegas added LLR-based variant effect prediction evaluation for DNA models #3144 and an EDA notebook on perplexity vs downstream task performance #3333. Documentation alignment by @dlwh-golem across TPU cluster setup #3415, contributing hooks #3307, MkDocs commands #3271, and README paths #3235. CI improvements for PR checks on all target branches #3150.

Top 15 runs (by FLOPs) this week (completed, running, crashed)

The largest single run this week was Will Held's 10B-parameter AdamH scaling ladder at 1e22 FLOP budget on v4-256 (128 chips), reaching 47.5% MFU and processing 116B tokens over 59 hours before crashing — the first Marin run at this scale. MoE scaling law experiments intensified: ClassicLarry ran two complete TGL Phase 1 expert-count sweeps (2 to 256 experts, 8.3B tokens each) on v4-8 plus an EKN nano scaling sweep testing K and LBL coefficient interactions across ~20 configurations. The 256-expert configs consistently achieved the best loss (3.060-3.186). Moo Jin Kim completed a large OpenThoughts4 SFT run (Qwen3-1.7B with Qwen3-32B teacher, 100k steps on v4-128 at 34.7% MFU). David Hall began 32B-A4B MoE bring-up on v5p-64, running several short profiling attempts at 11-20% MFU that crashed but established baseline performance numbers. Parallel-attn-mlp experiments on v6e hardware investigated persistent NaN issues under restart.