Week of March 9th summary for marin-community/marin

#2836 Infrastructure: MoE Training Support

Summary: Train a 50B MoE model on GPU hardware reliably — from data preparation through sustained multi-node training with automatic fault recovery. This epic tracks all the infrastructure, data pipeline, and training work needed to get there by March 31.

Iris underwent a deep architectural overhaul this week. @rjpower made SQLite the canonical state store #3408, replacing the prior in-memory state with a proper schema, migration system, and normalized tables for scaling groups and slices #3514. The controller now checkpoints its SQLite state to GCS for post-mortem analysis #3497. Auth was overhauled from per-RPC DB lookups to HMAC-SHA256 JWTs #3630, #3537. The dashboard was rewritten from Preact+HTM to Vue 3 with TypeScript, Rsbuild, and Tailwind v4 — 26 components covering all existing tabs plus a new task detail page #3511. Multi-VM CoreWeave support landed with JAX coordinator bootstrap for distributed training across VMs #3638, alongside namespace-qualified RBAC so multiple Iris instances on the same cluster no longer interfere #3703. Job lifecycle got new preemption and existing-job policies #3685, proper slice reaping on worker failure #3425, and ghost-slice prevention during scale-down #3571. The autoscaler saw deadlock fixes and rate-limit logging #3580, #3531, #3616. On the data processing side, @ravwojdyla rewrote Zephyr's shuffle to use Parquet instead of per-chunk pickle blobs #3482, #3656, added dynamic batch sizing for writers #3498, and made exact dedup work at Nemotron scale via single-pass hash-group-write #3442. Protobuf generation was moved from checked-in files to an auto-generating hatch build hook #3631. @dlwh batch-tiled the XLA fused cross-entropy path to handle long sequences without hitting the TPU int32 word-count limit #3533, tuned mixed-dtype block sizes #3452, and fixed GCS executor step-lock races under worker churn #3541. @rjpower also introduced nightshift — automated GitHub Actions workflows using Claude agents for overnight cleanup, dead code removal, and issue triage #3557, #3615.

#3096 Pre-training: MoE Scaling Laws

Building on last week's initial isoflop sweep, @ClassicLarry completed full results for the 15-config isoflop grid across three FLOP budgets (1e18, 3e18, 1e19) with five model widths each #2167. The sweep confirmed d768 as the optimal width at 1e18 and 3e18, with d1024 competitive at 3e18. Parallel EKN scaling experiments #3182 explored expert count and granularity — 128 experts with K=2 achieved the best BPB (1.0565) but at significantly lower MFU (14.9%) than the K=4/E=32 baseline. Aux-loss-free balancing consistently outperformed traditional load balancing loss. @Helw150 pushed the architecture to iter_03 with sigmoid routing (replacing softmax), independent per-expert gating, and an AdamH optimizer sweep on the updated recipe. @WhenWen ran AdamH comparisons showing ~0.009–0.01 BPB improvements over the v02 baseline at smaller dimensions, and tested Gated Norm from recent literature on top of AdamH. A new sub-issue tracks testing the architecture at 1e21 and 1e22 FLOP scales #3800 to validate routing stability at production scale. On the GPU side, @chloechiaw is evaluating Tokamax's Pallas Triton ragged_dot kernel as a JAX-native alternative to custom GPU MoE kernels #2828. @yonromai consolidated all canaries into a single Grug MoE entry point running through Iris #3587, with fused CE autotune fallback fixes #3605.

#3100 Data Sources for Pre-training

Summary: We will need 20T of high-quality (including / in particular code) tokens for our large MoE runs in Q2/Q3; this is the work in March that we will do to enable that.

Following last week's Nemotron-CC tokenization milestone, @ravwojdyla completed large-scale fuzzy dedup of Nemotron splits using the rewritten group-by pipeline #2829, validating that the Parquet-based Zephyr shuffle handles production-scale data without the file-count blowup that plagued the pickle approach. The exact dedup rewrite #3442 replaces the two-pass dup-map strategy with single-pass hash-group-annotate, outputting Vortex files directly. In a new embedding evaluation effort #3535, @ravwojdyla tested Luxical-One embeddings as a general-purpose representation for data curation. Quality filtering via linear probe scaled from Spearman 0.485 at N=125 to 0.75 at N=10K — promising but below the 0.8 go threshold. Luxical outperformed Arctic-L and BGE-large at 6× fewer dimensions. Topic clustering was a clear no-go across all models (best NMI 0.478). @gonzalobenegas continued DNA model work with separate uppercase/lowercase weight handling and a functional_pos experiment tracking functional vs nonfunctional log-likelihood across model sizes #3483.

#3192 Synthetic Data

On the OpenThoughts4 front, @moojink ran follow-up experiments with smaller models — training Qwen3-1.7B-Base on data from Qwen3-4B and Qwen3-32B teachers, plus a rejection sampling variant where Qwen3-4B generates and Qwen3-32B verifies #2262. These experiments test whether the modest advantage of larger teachers observed with Llama3.1-8B holds across student scales. @natolambert flagged the need to test multiple teachers as the key takeaway, with rejection sampling results still pending.

Other Changes

@dlwh added an HBM optimization guide #3595 and refactored the Pallas kernel skill with reference guides #3570. @dlwh-golem continued documentation alignment — contributing guides #3520, #3596, docs build guides #3551, #3607, eval entrypoint drift #3549, and agent workflow skills #3663, #3662.

Top 15 runs (by FLOPs) this week (completed, running, crashed)

MoE scaling dominated the week: a 15-config isoflop sweep at 1e18–1e19 FLOPs settled on d768 as optimal, alongside long-run architecture comparisons on preemptible v5p-64 and a 235B parameter trial queued for v5p-256. Agentic SFT reproduced NemotronTerminal baselines on Terminal-Bench.