Week of March 30th summary for marin-community/marin

@ClassicLarry launched the 1e22 MoE run #3800 — a 34.6B-total / 5.1B-active model on v4-512 with capacity factor 1.0, running at 488k tok/s and 23% MFU (W&B). The capacity factor was set to 1.0 based on ablation results #4016 showing cap=1.0 is 8.3% faster than 1.25 with negligible loss impact at 1e20 scale. The v7 isoflop sweep #4225 mapped LR schedule and decay interactions with AdamH across dimensions, finding that removing linear decay at small step counts (d2048 @ 1e18) improved BPB by 0.03 — suggesting decay fraction needs to scale with training length. @Helw150 resolved the QB overhead problem #3972: the async CPU overlap approach was a dead end (a misleading benchmark), but sharded microbatch QB eliminated the 1.2 MFU-point overhead entirely. He also updated the MoE baseline #4084 and validated moe-sharded-qb-gn-xsa as the best iter-04 configuration with a Vizier hyperparameter sweep running on us-east5-a #2167. On the Delphi dense scaling ladder #1337, the 1e23 run on v4-1024 continued at 30% MFU with Paloma macro BPB 0.79, and the 1e22 seed42 run finished at 47% MFU (macro BPB 0.84). @dlwh landed XLA-first Mamba-3 SISO and MIMO TPU kernels #3961 (+5,352 lines) with a sharding-safe ranked public API #4149, and @msclar merged the AdaMuon optimizer implementation #3300.

@rjpower drove a major Iris controller performance push: lock hold time in drain_dispatch_all dropped from 80ms to under 5ms #4222, a two-pass heartbeat batch delivered 4x faster provider loops #4210, lightweight job state polling reduced controller load #4209, the ORM query builder was replaced with raw SQL #4181, and log fetching was unified under FetchLogs with LIKE patterns #4202 (-598 lines). Checkpoint management got zstd compression and old-checkpoint pruning #4143. @ravwojdyla shipped the actor proxy service for external access to cluster actors #4126, refactored Zephyr chunking for improved shuffle scalability #3839 (+1,357/-539 lines), and allowed specifying coordinator resources #4095. User-defined counters (MapReduce-style per-job stats) were added #4085 with records_in/records_out counters for readers and writers #4189 and per-worker counter queries #4164. @yonromai fixed a TensorStore handle leak in cache-copy (~14 MiB/shard) #4198, fixed stale coordinators killing new workers on retry #4199, and added Slack alerts and Claude triage to the canary ferry #4158 #4177. @dlwh landed the region-aware executor on Iris #3824 (+1,075 lines). The integration test suite was redesigned #4009 and optional auth mode landed for gradual adoption #3937.

@dlwh drafted a staged modeling experiment skill #4166 to make modeling experiments Grug-first with W&B view tagging and optional stage gates.

The largest active run is the Delphi 1e23 dense scaling ladder (W&B), a 25B parameter model on v4-1024 now at 608B tokens with Paloma macro BPB 0.79 and train loss 2.08 — @Helw150 noted the loss trend during cooldown is tracking between the pessimistic and optimistic forecasts #1337. The 1e22 MoE v7 run (W&B) launched mid-week on v4-512 at 23% MFU, estimated 7.7 days #3800. Two finished v7 isoflop runs at 1e20 validated capacity factor=1.0 as 8.3% faster with near-identical loss #4016. On the post-training side, @AlienKevin's 32K v2 OT-Agent SFT (W&B) reached 13% SWE-bench (matching released 14%) #3896, while the 131K v2 run on v5p-256 (W&B) revealed that batch-size-scaled gradient clipping is needed at 131K context #3897. Three Qwen3-14B resilience SFT runs for math and medical domains completed on v5p-16 at 56-57% MFU.