Marin: Week of May 11th summary

Decontamination moved from sketch to production this week. #5519 closed after @ravwojdyla smoke-tested the existing bloom-based decon against Dolmino's leaked GSM8K test split — n=13, threshold=0.5 recovered the documented leak at 100% recall with 0.03% spurious train-split hits (all verified ngram overlap, not bloom false positives) — then validated the new datakit-native implementation end-to-end on a 45M-record, 231-partition slice of nemotron_cc_math_v1 against MMLU, finishing in 2h 33m on Iris and flagging 2,605 contaminated docs spanning 557 distinct MMLU questions. The supporting PRs landed in a tight stack: #5650 made mark_duplicates_bloom actually honor NGramConfig.overlap_threshold, #5655 retired the TRAIN_TEST_OVERLAP mode and the experiments/train_test_overlap/ scaffolding it grew out of, and #5656 ported decon into marin.datakit.decon, emitting a co-partitioned Parquet attributes dataset with {id, partition_id, contaminated, max_overlap} per source and preserving input filenames 1:1 so the consolidate stage can sorted-merge-join without a shuffle. Scoping for the next decon corpus is underway against Artificial Analysis's Intelligence Index v4.0 — 8 of 16 evals (HLE text-only, AA-Omniscience, GPQA Diamond, MMLU-Pro, LiveCodeBench, SciCode, …) verified, with per-eval extractors instead of generic field concat. The plan, kicked off in #deduplication, is to decon at doc level rather than span level for V0 — @dlwh wants an agent thrown at everything in AA and lm-eval-harness, @Helw150 pointed at #2663 as the current eval-set lineage, and @dlwh's in-progress 100-200M-token ppl basket will fold in once it's defined.

The stretch-goal deliverables — quality scores and domain tags for every Datakit source — both landed end-to-end runs. #5812 classified all 104 sources with AllenAI's Dolma3 fasttext quality classifier, emitting one float high_score per doc co-partitioned with normalized/*, and #5811 ran the matching Dolma3 WebOrganizer topic classifier across the same 104 sources in a single 10h 45m job with zero failures or preemptions, writing ~970 GiB and a 24-class distribution per document. On the embeddings side, #5809 wraps up the all-sources Luxical-One embedding pass that feeds the domain-bucket clustering tracked in #5808. All four are co-partitioned attribute datasets in the same shape as decon, so downstream consolidate / data-mixing can map-side join across quality, topic, embedding and contamination signals without a shuffle.

Underneath the science, @ravwojdyla hardened the datakit sync and artifact layer that the whole pipeline now leans on. #5648 introduced cross-prefix raw-download sync between MARIN-shaped prefixes — typically GCS to R2, one StepSpec per source — and #5734 followed up with a --scope {normalized,raw,all} flag so the canonical normalize output rides the same path; #5695 opened validate_source_staging to R2 via a --prefix flag. Resume semantics tightened with #5654, which made .executor_status the canonical leaf-fully-synced marker and pre-flights every source so already-synced ones drop before any StepSpec is built; #5725, which makes only SUCCESS count as done — stale RUNNING/FAILED markers re-sync, and uploads serialize on the same step_lock the executor itself uses; #5717, which drops raw-download deps from the sync StepSpec so re-runs no longer re-materialize from S3/HF; and #5700, a post-finalize sweep of .tmp.<uuid> orphans left behind by interrupted atomic renames. #5733 reshaped materialize_ghalogs_to_parquet to list the zip's central directory locally and round-robin partition members across workers, ending the single-shard OOM on large archives. #5727 renamed Artifact.load -> Artifact.from_path with MARIN-prefix-aware relative resolution, #5732 added a .executor_status==SUCCESS fallback when .artifact is missing and tightened typed loads to pydantic-only, and #5756 caught the three tokenize call-sites the rename had missed. #5664 wires the seven Nemotron-CC v1 quality splits, ~6.05T tokens total, into all_sources() alongside the v2 families, and #5626 added a bare=True normalize mode that drops every non-canonical column at the record level — motivated by starcoderdata's 87 per-language meta schemas and proof-pile-2's nested meta dict. On storage hygiene, @RohithKuditipudi's #5688 made the parquet-cache rsyncs pass --delete-unmatched-destination-objects so stale segments from prior scanner runs drop out of dir_summary, and #5744 formalized the larger Datakit GCS cleanup as a sub-task of #3721.

Tokenize stayed under active rework around its own datakit split. #5438 finally landed the long-promised carve-up — tokenize_attributes(NormalizedData)→TokenizedData emits a per-doc {id, input_ids} parquet co-partitioned with the source, and build_levanter_store consumes one or more of those into the trainer-facing store. #5388 parallelized LmDataConfig.build_caches over its ~100 components with a 32-thread pool, cutting the ~40-minute GCS-metadata round-trip that had been dominating Datakit Testbed startup. @nevillelyh's #5430 reworked the consolidate-removal experiment first sketched in #4814 — reading thousands of ledger metadata files has real overhead, so the ledger keeps getting consolidated; only the data files don't. @eric-czech fixed #5790 in #5791: _exemplar_for now scans past empty leading shards instead of crashing on results[0] when a filtered write_parquet emits an empty-footer file. On the source-onboarding side, @Helw150's #5305 registers TeraflopAI/SEC-EDGAR (43.7B tokens, ~8M filings) with a DuckDB transform sitting between download and normalize to work around an upstream Arrow page-header-size bug. And in #2351, @XenonMolecule posted the week's progress on the small-model raw-web-to-tokens experiment: three escalating quality-bar extraction prompts — low, medium, and high, the last targeting roughly a 1-in-10 pass rate — a scaling-ladder plan that fits Muennighoff-style discounted-token scaling laws per quality tier, and an open question about whether distilling a small extractor model could close the ~44-year, 8M-WARC gap on the high-quality spec.

The week opened with fallout from the previous week's coscheduled-gang fixes. @rjpower's #5634 chased down a ListJobs Value out of range: 6442450944 crash filed as #5667 on the May 12 canary — the coscheduled sibling cascade path was writing preemption_count = max_retries_preemption + 1 with max_retries_preemption = INT32_MAX as a tombstone, and three sibling rows summing through SQLite's SUM hit 3 x 2^31 — by anchoring the job_id_prefix filter and clamping int32 fields at the proto boundary, then #5638 removed the need for the arithmetic entirely by introducing a dedicated TASK_STATE_COSCHED_FAILED terminal state for cascade-killed siblings. @yonromai's #5685 caught the downstream contract change in scripts/workflows/iris_monitor.py, where the canary still passed JobName objects to a list_jobs(prefix=str) that #5634 had retyped. #5572 — the gang-on-first-sibling-error misclassification bug — stayed open through the week; @ahmeda14960 posted a fresh recurrence on Monday in #infra after seeing the infinite-preemption loop again on a 1e22 run, and #5753 was opened after a JAX-distributed-RPC sibling self-exit was misclassified as TASK_STATE_FAILED instead of TASK_STATE_WORKER_FAILED; #5754 extends TPU_INIT_FAILURE_PATTERNS with the gRPC Socket closed / RAW: Raising signal / Fatal Python error: Aborted signatures so jobs with max_retries_failure=0 stop terminal-failing on the first preempt cascade.

The controller was sluggish or worse on three separate occasions this week, and the response is the dominant arc. @rjpower sized the uvicorn default executor to 64 in #5710 so a burst of LaunchJob handlers blocked in _wait_until_job_drained couldn't starve heartbeats sharing the same pool, then unified all four replacement paths behind one drain-aware helper in #5629 after Larry hit a FAILED_PRECONDITION: worker-bound attempts have not finalized on May 11 (posted in #infra); #5713 capped the LaunchJob client timeout at 180s and re-checked existence inside the INSERT transaction so the losing handler short-circuits to ALREADY_EXISTS rather than tripping the jobs.job_id PK and racking up a retry storm. #5697 shrank the controller read pool from 32/4 to 2/2 — SQLite WAL admits concurrent readers but each contends on the WAL-index header lock when establishing a snapshot, and queuing at the SA pool turned out to be cheaper than spinning inside SQLite. #5765 dropped the @on_loop decorator so every sync RPC runs through asyncio.to_thread, removing the path where get_job_state and set_task_status_text could starve other handlers on the ASGI loop, and #5768 moved task status updates onto the RPC loop. #5774 — still open at week's end — replaces the per-task-per-worker SELECT + 2 UPDATEs + endpoints DELETE in the heartbeat-apply loop with a planner/dispatcher split that flushes one executemany per table, the highest-impact entry on @ravwojdyla-agent's profile-driven controller-contention issue #5751. The Friday-evening Iris VM upgrade did not go as planned — workers gave up on the controller after ten minutes of downtime and jobs were restarted — and an unrelated overnight stall on May 14-15 had @ravwojdyla-agent restarting the controller and capturing the py-spy snapshot that became #5751.

The week's biggest single PR is @rjpower's #5651 — +8.1k / -11.6k LOC across 115 files — which replaces the hand-rolled stores.py / Projection / ProtoCache / EndpointStore / _attr_cache / TransactionCursor stack with a single SQLAlchemy Core path; tables, indexes, FKs, and TypeDecorators are now declared once in schema.py, reads and writes are module-level functions over db.Tx, and the controller core sheds ~3.5k lines. #5776 followed up by merging SchedulingTickContext into SchedulingContext, deleting the create_scheduling_context wrappers, and converting 19 SQL call sites to expanding bindparam so SQLAlchemy reuses compiled statements; #5775 reworked benchmark_controller.py so every benchmark exercises the controller through its real Connect/HTTP surface against an out-of-process harness, with a PRODUCTION_MIX_RPS preset derived from the live stats page (SetTaskStatusText dominates at 316 rps, every other RPC under 8). #5636 introduced a reconcile-only worker protocol — a single Reconcile RPC that unifies dispatch, polling, and status reporting, plus a pure-compute reconcile_workers(ReconcileInputs) -> list[plan] split — gated by IRIS_RECONCILE_RPC_ENABLED so legacy StartTasks+PollTasks remains a flag flip, and #5641 is the still-open Phase E that collapses the eight inline JobState recompute call sites behind a single compute_job_state pure function plus a SQL view so controller, dashboards, and CLI tooling can never disagree on a job's state. #5701 is a 1300-line design doc proposing a single TickDriver thread that builds a shared ControlTick snapshot per tick — a production profile shows the four current loops burn ~57% of one core on redundant DB reads — and explicitly cites #5470 as a within-tick consistency failure that one snapshot would have prevented.

On the autoscaler side, #5699 replaced the per-group exponential backoff with a sliding-window BackoffDetector that grades each scale group HEALTHY / SUSPECT / CHURNING / HOSTILE over a recent-outcomes window so isolated preemptions stay noise instead of producing a capacity-pit sawtooth on preemptible TPU zones; #5716 followed with an AIMD health score in [probe_floor, 1.0] that modulates the token bucket's refill_rate instead of batch size, fixing a floor=1 per-tick bug where CHURNING groups still launched every 10s; #5720 partitioned the boot scan of list_all_slices so dead slices are async-terminated and discarded checkpoint slices are deleted from the slices table instead of accumulating as ghost rows. #5738 completed the zero-trust proxy migration begun in #5336 by routing ProxyResolver at <controller>/proxy/<name-with-dots> and deleting ActorProxy wholesale, so actor calls go through the generic EndpointProxy with no special case. @Helw150's #5707 closed #5708 (filed by @eric-czech) so compute_user_spend excludes user-volunteered BATCH-priority tasks from the per-user budget. @RohithKuditipudi's #5709 wired a daily 14:00 UTC cross-region-egress GHA that uploads to GCS and posts a Discord summary, fixing #5703 where cross_region.py had been reading the stale pre-finelog state/logs/ path since April 28; his #5757 adds an lib/iris/docs/controller-health.md triage doc structured around the four-phase oncall flow proposed in #5752. @yonromai's #5623 / #5649 added a queue-aware TPU canary wait so the canary stops reporting false-red when the GHA wait step spends its budget on Iris queue delay; @ravwojdyla-agent's #5755 and @yonromai's #5781 added p50/p90/p95/p98 latency tracking and a rolling 5m p50/max chart for both Iris and finelog /health on the prod and dev status pages.

Finelog spent the week continuing the production-hardening arc. #5777 made WriteRows / PushLogs block until rows are persisted to an L0 parquet segment via a max_persisted_seq cursor, moved the server to ASGI handlers, dropped the flush interval to 5s, lifted the live namespace registry into Catalog, and bundled an append-path slim-down replacing RecordBatch repack with AlignedBatch. #5674 switched Compactor.plan to count-based promotion after hundreds of small L1 segments stopped promoting to L2 on namespaces with small L0 flushes, and capped the in-memory chunk buffer so a 7-8s L0->L1 merge plus 1-2s upload can't balloon RAM to ~190 MB under sustained writes. #5711 closed a ~7 GiB leak where compaction inputs whose fs.rm hadn't completed before a crash were re-adopted on boot as REMOTE catalog rows. #5646 took migrations out of the runner's wrapping transaction so 0006 can run multi-statement DDL+DML on DuckDB, and restored MATCH_SCOPE_UNSPECIFIED -> REGEX as a back-compat fallback so the #5606 rollout doesn't break older dashboard / iris controller builds. @ravwojdyla-agent's #5673 cached _SealedBuffer.nbytes / num_rows at construction after py-spy showed RamBuffers.ram_bytes() at ~98% self-time recomputing pa.Table.nbytes on every call (closes #5671), and his #5696 added a finelog gcs-query CLI that runs DuckDB directly against the GCS-archived parquet with a time_created pre-filter, side-stepping the live FetchLogs path which never sees evicted segments. #5680 added finelog query with SSH / kubectl port-forward plumbing extracted to a new rigging.tunnel module, and #5766 made the iris controller resilient to finelog being unreachable at startup by wrapping the three startup get_table sites in a background retry loop with bounded exponential backoff. On the Zephyr side, #5485 stopped zephyr coordinators from terminating worker jobs before workers self-exit on SHUTDOWN, fixing #5484; @ravwojdyla-agent's #5624 capped fs.put batch_size=8 in atomic_rename after compounding fsspec defaults pushed in-flight multipart buffers to 11 GiB peak and SIGKILL'd tokenize workers; @ravwojdyla's #5698 closes the long-pending #4345 by retrying fs.mv with bounded backoff on the narrow allowlist of transient S3/R2 errors; #5769 and #5679 rate-limited zephyr's per-shard chunk progress log after a single normalize task generated 275M log rows in one hour on finelog #5678.

@yonromai landed the first concrete step from last week's RFC: #5712 bumps the TPU vLLM stack to vllm-tpu==0.19.0, tpu-inference==0.19.0, and libtpu==0.0.39, with a Grug TPU smoke job covering model init, two bounded train steps, eval, and checkpoint save, plus the existing test_inference_ctx, test_rollout_worker, and test_served_lm_eval suites passing. The PR explicitly does not claim to fix Delphi serving — @RohithKuditipudi filed #5672 for the scoped-VMEM allocation failure in the ragged-paged-attention Pallas kernel that the old bkv_p-halving monkey patch no longer intercepts under the new stack, and reverting pins toward the older vllm-tpu==0.13.2.post6 / jax==0.8.0 stack didn't move the failing kernel shape. The Levanter half of the contract closed out: #4828 shipped via #5331 — /v1/completions with echo=true prompt-side token_logprobs and the lm-eval local-completions slicing path — with remaining distributed eval/inference work now tracked under this epic and #5400.

The forks-vs-patches debate from the prior week converged into a staged plan. In #inference, @yonromai proposed bumping upstream vllm-tpu and tpu-inference to 0.19.0, rebasing the Marin forks onto those releases, then swapping Marin to consume the forks, with the same cadence each upstream release. @ahmeda14960 reframed the maintenance cost: Marin isn't really maintaining vLLM but rather tpu-inference, which is smaller and slower-moving, and the vllm fork itself only exists because vllm[tpu] was stale and pulled in NVIDIA packages that conflicted with TPU pins — a requirements-file patch would suffice there. @RohithKuditipudi noted his changes are localized to tpu-inference — a new RPC collective — and that he doesn't personally need a vllm fork. #5712 is the first leg of that plan.

A parallel design conversation kicked off in #inference when @Michael-Ryan asked who else wanted a dedicated distributed inference service and what their desiderata were. @yonromai described the Iris service he's been building — a local actor exposing an OpenAI HTTP endpoint that lm-eval talks to, with workloads queued and picked up by preemptible TPU workers each running a local vLLM — noting the prior RFC had rotted while he was on CoreWeave work and needed a refresh. By end of week he had circulated a cleaned-up Zephyr-backed multi-region inference draft for the same shape, this time aimed at the WARC-download workload @Michael-Ryan brought up. The boundary is the same one #5400 drew: evaluators see only an OpenAI API root and a model name; the service owns vLLM lifecycle, preemption, and routing.

Two smaller pieces moved on the eval-side plumbing. @dlwh's #5723 added an exclusive integer-label contract for per-token LM evaluation in Levanter and a LabeledEvaluator that aggregates losses and BPB over named label groups, separating training loss weights from evaluation annotations and giving future trace-style evals a generic span-label target; the follow-up #5724 ports the agent trace scoring path onto that framework, replacing the parallel mask evaluator with a trace_labeled_eval runner. @RohithKuditipudi opened #5632 to close #5631, adding a shared marin.evaluation.utils helper that fsspec-copies tokenizer files out of GCS-hosted HF checkpoints into a temporary local directory, so downstream eval code can load tokenizers without making Transformers or Levanter resolve gs:// paths themselves — a small but real friction point for running evals against checkpoints staged on GCS.

This week the gap suite moved from adding slice families to reading them. @dlwh uploaded the synthetic numeric-and-structured-format dataset called out in #5614 to Hugging Face as marin-community/synth-numeric-format-ppl with ten validation configs covering numeric copy/increment, compare/sort, range/checksum/base, tabular TSV/CSV, IP/port rows, SVG numeric blobs, JSON numeric arrays, mmCIF coordinate tables, format-preserving transforms, and chunk-boundary stress. The first 32B v16 run showed every controlled numeric-format config Marin-favored, but span heatmaps revealed the win was a prompt-format artifact: the User:/Assistant: framing made Qwen blow up on the first scored byte while teacher-forced continuations were neutral. @dlwh immediately spun out synth-numeric-format-prompt-ablation-ppl with three base-model prompt variants — newline, arrow, and equals — across the same ten subsets, and the aggregate gap collapsed to roughly -0.015, -0.005, and -0.005 bpb respectively.

The five surface-form probe families staged in #5618 all landed and ran through the v18 all-available diagnostic. #5643 added supervised target-only perplexity scoring with target spans preserved through score artifacts so single-model summaries and gap comparisons compute loss only over target bytes. Each probe was then archived as a closed experiment issue plus a long-lived research branch: #5659 for delimiter/whitespace/table formats (Marin -0.040 bpb aggregate, but Qwen ahead on CSV and pipe rows), #5660 for identifier/encoding grammars (Marin -0.203 bpb, treated as synthetic surface-form evidence rather than real encoding competence), #5661 for machine records and configs (Qwen +0.064 bpb, strongest on Dockerfile, Actions logs, nginx), #5662 for patch/diff surfaces (Marin -0.069 bpb), and #5663 for science/markup/tree-closure records (Qwen +0.089 bpb, with the largest gaps on SDF closure, entity escaping, and GenBank-style records). These branches are intentionally not wired into the regular eval suite.

With four weeks of slice work behind him, @dlwh opened #5686 to choose which of the newly landed slices should become lightweight training-run sentinels alongside the existing Paloma and Uncheatable portfolio, under a clearer name like format_readiness or surface_form_sentinels. The candidate pool is structured_text (ToTTo, WikiTableQuestions, GitTables, optionally WebDataCommons), synthetic_delimiter_format_ppl, the neutral arrow/equals variants of synthetic_numeric_format_prompt_ablation_ppl, and the Stack v2 programming-ecosystem coverage from #5254; the full gap dashboard and the role-framed numeric suite are explicitly excluded. The Stack v2 cut itself was declared a v0 at 105 of 136 slices after the materialization job stalled, and the 32B gap run on that v0 finished mid-week with the dashboard now pointing at the v21 artifact. The standalone tabular CSV slice #5372 was closed as superseded by the broader structured-text work. #5643 also tightened the basket coordination question from #5005: @dlwh told the dedup thread that for the next few weeks Uncheatable, Paloma, and his own ~100-200M-token basket are the PPL sets to keep pluggable.

@dlwh posted a refreshed perplexity gap dashboard with the new datasets and synthetic heatmaps, calling out that Marin is very bad at JSON and that Qwen clearly understands arithmetic progressions Marin does not. A follow-up suggestion proposed seeding from OEIS and filed #5770 to track it. A detailed Marin-vs-Qwen3 tokenizer comparison followed: similar 128k-vs-150k vocab sizes but weird multi-digit tokens like 190/191/192 that cause years to split as [19X, Y] in Marin, ~5k more whitespace-prefixed tokens than Qwen3, and 40 tokens beginning with grouped . punctuation against Qwen3's two. @dlwh confirmed the Marin tokenizer is essentially Llama 3 with two extra tokens and a new chat template, and asked to mirror the analysis onto #5079 and #4971.

The production-swarm spec firmed up this week. @Helw150 wrote up the latest sizing for #5364 after a Friday whiteboard with @Calvin-Xu: ~1,000 runs at a single scale, no scaling-law term in the mixing form, gated on the 1e23 preregistration run #4697 finishing on May 20. With ~380 hours of reserved-instance time at 20% MFU, that budgets ~6e22 FLOPs total, or 6e19 FLOPs per run — the current grug isoFLOP-correct shape is d=1280 (2.3B total, 408M active) on 26.2B tokens, following whatever Grug MoE recipe is on main on launch day. @Calvin-Xu noted the per-run figure came out roughly 10x smaller than they had originally whiteboarded once MFU was put back into the math. The launch target on #5365 is May 20 with a June 5 handoff to @ClassicLarry for two weeks of final arch tuning before the June 19 hero run; @ClassicLarry flagged that the optimizer is likely to keep moving under that schedule.

On the metric question for #5362, @Helw150 reported on a couple of weeks of factor-analysis over Calvin's benchmark suite — a stripped-down IRT — and confirmed that a low-dimensional aggregate does explain many benchmarks at once. The naive aggregate, however, over-rewards the factors that are easiest for the data mix to move (math and coding), at the expense of more uniform improvements; HellaSwag in particular looks tied to data quality in ways that may not be what they actually want to chase. The plan for the rest of the week was to fold @dlwh's perplexity work from #5367 into the mixing flow and see whether that flattens the over-optimization. Meanwhile @Calvin-Xu moved the 300M-scale preemptible swarm forward against the upcoming spec, fitting a 39-domain extension of the UtiliMax-style saturation+penalty form (good fit, since the existing 158-parameter slidermix run is effective-exposure DSP); @Helw150 noted the new Scaling Laws for Mixture Pretraining Under Data Constraints paper moved into repetition-aware modeling that Marin had been planning to be first into.

Source coverage kept expanding underneath the swarm. #5741 registered KantaHayashiAI/ClimbLab-Ja — ~300B Japanese tokens with Nemotron-ClimbLab-style semantic clusters and per-document quality scores passed through normalize — closing #5740, and #5743 brought in the four ODC-BY locuslab safety-pretraining datasets (moral_education, safeweb, refuseweb, fineweb_annotated) for #5742. @Helw150 opened #5745 for the non-GPT-5 portions of AgentTrove, asking for a positive type filter on original_teacher to avoid any OpenAI-ToS exposure, and @hammer proposed #5770 to train on OEIS integer sequences after @dlwh's observation that Qwen handles arithmetic progressions Marin sometimes does not. @taivu1998 filed #4996 to register FineProofs-SFT in both raw-messages and proof-only views for long-context reasoning runs. On the tokenization side of #3556, @ravwojdyla reported that all Nemotron v1 and other ferry datasets are now normalized on R2 via CW and that 5 of 7 Nemotron splits are tokenized, with the remaining two in progress on post-#5430 main. @ahmeda14960 posted a self-correction on #4547: every Delphi midtraining result previously reported under the 1e20 label was trained from a deprecated v5 isoflop checkpoint rather than the v6 bucket winner, so prior 1e20→1e22 transfer claims are weakened while the 1e21→1e22 within-family comparisons still hold; future Delphi bases will come from experiments/delphi_models.py rather than GCS-path grepping.

The preregistered 1e23 MoE run at #4697 kept moving but had to be relaunched mid-week: @Helw150 posted the new wandb URL pointing at moe_1e23_d5120_bs2048_ep8_ragged_48l_resume93092_clip15_20260514_031402, a resume from step 93092 with gradient-clip 1.5, and @ClassicLarry consolidated external tracking on the standing Marin 1e23 MoE wandb report so individual run links stop going stale. The 2.25 paloma macro target from the #4447 isoflop fit is still the live bet.

The MuonH gate-2 from @Kaiyue-Wen's #5596 finished with a mixed verdict: per-scale wall-clock speedups of 1.33x/1.26x/1.22x/1.14x at d512/d768/d1024/d1280, but the pinned-L_inf=1.6 four-point fit gave MuonH α=0.0906 versus baseline 0.0941, predicting a crossover near 1.65e21 and a +0.011 gap against baseline at 1e23. The no-warmup follow-up #5619 / PR #5620 finished both gate-2 points (d1024 paloma 3.123, d1280 2.971) and flipped the projection back to a small positive win at 1e23; Kaiyue-Wen noted in #moe that the four-point fit's own 7e-3 residual error swamps the predicted 1e23 gap, so the crossover should not be read literally. The MuonH LR retune #5621 / PR #5622 launched Phase 1 baseline-warmup and no-warmup sweeps in parallel at d512, then opened Phase 3 at d1024 over muonh× = adam× ∈ {0.5, 1, 2, 4} with warmup=0.0 throughout. The closely related #5598 NorMuonH matrix swap passed gate-1 cleanly (d512 3.7512, d768 marginally better than MuonH), keeping NorMuon and Aurora in the alternative-Muon shortlist.

The architecture line consolidated around a new canonical baseline. @ClassicLarry spent the week stacking the April pieces (256 experts, PKO every 4th, partial RoPE, last-layer PKO, no router/logit z-loss, embed routed to AdamH) into may_arch, then ran the consolidated GatedNorm routing comparison in #5750: across the four candidates — asymmetric default, gn→adam, gn→adamh, gn→muonh — the symmetric NS+hyperball wrap won at every scale, and a final no-hyperball control showed that dropping the Frobenius wrap regresses by +0.0086 / +0.0054 / +0.0014 at d512/d768/d1024 — the NS direction and the hyperball both matter. The follow-on #5763 tested may_arch + GN→MuonH + 1% warmup + no grad-clip at three scales and posted a strong d512 win (−0.0170 macro, ~7.8% wall) with d768/d1024 tied within noise; it was adopted as the canonical 1pct-noclip baseline on the grounds that the d512 lead is reproducible and the recipe is simpler. A GatedNorm LR-sensitivity sweep #5746 confirmed that scaling GN LR down by 0.7x helps (gnwup-0.7x the cleanest improvement at d768/d1024), and the 48-cell per-group LR sweep #5773 landed embed-LR=1.2x as the most promotable shift.

The downstream 1pct-noclip stack produced one borderline result. Col-norm equalization on K/V — a fix for the non-square shape that GQA produces — in #5767 beat the d768 anchor by Δ=−0.0035, and the d1024 promotion run #5778 beat its anchor by Δ=−0.0022, but Larry closed the issue noting that AuroraH was too slow and the gain was too small to merge. The MuonH paired-experts attempt in #5613 hit a JAX explicit-mesh ShardingTypeError on the _pair_3d_leading reshape and is mid-fix with split storage for w_gate/w_up, and the new KL SOAP H optimizer #5728 / PR #5729 failed gate-1 decisively at both scales (Δ +0.335 / +0.264 vs AdamH, ~0.24x speedup) — a clean negative result that closes the SOAP-H direction.

Larry set the cadence in #moe: end of May the team wants a recipe for the June run, ideally with some long-context handling, and he's confident the result will be MuonH plus tweaks. Three new agent-launched experiments queued against 1pct-noclip at week's end: sigmoid combine-weight renormalization to a fixed sum X ∈ {1, 2, 2.5, 3, 4} in #5797, a per-expert MuonH LR sqrt-sparsity split (shrink-expert at 0.125x vs boost-nonexpert at 8x) in #5799, and a kitchen-sink K=5 / half-shared / routing-renorm X=2.5 / split-storage stack in #5806. @ClassicLarry also shared the April MoE experimental-results summary ahead of the optimizer freeze.

@dlwh opened #5749 in draft on May 15, routing Grug's dynamic packed segment attention through Transformer Engine THD instead of dense JAX masks at 8192 sequence length. The split-reference path the PR replaces measured about 6.42 ms fwd+bwd on GH200; the initial cuDNN reshape fast path landed at 1.252 ms fwd+bwd against Megatron-style PyTorch TE fused THD at 1.142 ms on the loader-like B=1, S=8192, Hq=32, Hkv=8, D=128 BF16 shape. After narrowing to dynamic loader-style contiguous document runs — PackedTokenDataset / ChatDataset emit global doc IDs with -1 tail padding, not fixed block-aligned chunks — @dlwh rebuilt the path on SequenceDescriptor.from_segment_ids_and_pos after a controlled probe showed Marin's earlier offsets were [0, 17, 46, 77, 88] when TE expects segment starts padded with -1 like [0, 17, 46, 77, -1]. The corrected B=2 result is 0.572 ms forward / 2.305 ms fwd+bwd versus PyTorch TE at 0.434 / 2.114 ms — about 9% slower than Megatron's path and inside the prior ~5% target zone for B=1 but not yet for B=2. Tokamax Triton range-masks were numerically correct in the same envelope but ran 12.638 ms fwd+bwd, and Kvax blockwise document masks landed at 6.148 ms; both are non-starters for this shape.

The SonicMoE thread at #5328 continued the GH200 backward decomposition for the local Grug MoE block on the T=8192,H=2048,I=3072,E=8,K=2 BF16 shape. Best Grug forward was within ~2% of upstream Sonic/QuACK, but full-block fwd_bwd still sat at 4.035 ms vs 3.428 ms (+17.7%) and honest jax.checkpoint(prevent_cse=True) remat at 5.844 ms vs 4.798 ms (+21.8%). A first whole-MoE jax.custom_vjp attempt regressed badly to 12.804 ms fwd_bwd; wrapping only the activation -> down ragged_dot -> gather_sum sub-op with QuACK CuTe-DSL down-backward custom calls reached 3.998 ms fwd_bwd. The bigger speed lever turned out to be W13 GLU layout rather than dispatch overhead: storing W13 in Sonic's interleaved [gate0, up0, gate1, up1, ...] form avoided ~0.315 ms of runtime concat -> interleaved conversion, dropping fwd_bwd to 3.761 ms and remat to 5.036 ms. The CUTLASS DSL path was source-form-fragile — a ruff-suggested tuple-unpacking rewrite was enough to break cute.make_tensor lowering — so @dlwh made the QuACK boundary a JAX custom VJP that saves dispatch_output and falls back through XLA, reaching 4.4534 ms fwd/bwd/update, about 7.3% faster than the current production non-QuACK path. The new backends sonic_xla_interleaved_w13, sonic_pallas_interleaved_w13, and sonic_xla_interleaved_w13_quack_down are wired into experiments/grug/moe/model.py as opt-in choices via GrugModelConfig.moe_local_implementation; the QuACK backend is lazy-imported only on selection.

On the dispatch/combine sub-kernel at #5609, the isolated SonicMoE-shape token gather/sum path is now the cleanest remaining offender: matched wall-clock at the T=8192,K=2,H=2048,E=8 BF16 call-site has upstream Sonic at ~0.063 ms weighted and Grug/Pallas Triton at ~0.116 ms. A best faithful Pallas variant with BLOCK_H=2048, BLOCK_K=4, inline FMA emits 9.1 KB PTX / 312 lines / 7 ld.global / 1 st.global / 32 fma and still measures ~0.116 ms versus Sonic at ~0.083 ms wall / ~0.071 ms CUDA event, so @dlwh declared the source-level tile-shape hillclimb over for this kernel — the next useful artifact is SASS/nvdisasm to look at scheduling and register pressure rather than another tile sweep. A jax-triton overhead probe on jax-triton==0.3.1 showed a ~0.061-0.083 ms per-launch JAX wrapper floor that amortizes across dependent custom-call chains inside one jax.jit; embedded inside one compiled executable, a raw Sonic Triton gather/combine custom VJP measured 4.5626 ms fwd/bwd/update on the production shape against current sonic_xla_interleaved_w13 at 4.7740 ms.

On #4311, @hammer asked for a rough target gap between the existing stacks and Levanter/Grug; @dlwh answered that 10% of Megatron would be acceptable and is likely reachable. @dlwh also landed #5617 on May 13, which converts *.xplane.pb protobuf captures through xprof aggregate tables and normalizes step timing, kernel hot ops, collective breakdowns, and optimization candidates into a single profile_summary.v1 shape — the profiling pipeline the Grug-vs-Megatron sub-op breakdowns above run through. The GPU extras for marin[gpu] and marin-levanter[gpu] now ship the QuACK/Sonic stack: apache-tvm-ffi>=0.1.6, cuda-bindings>=13.2.0, nvidia-cutlass-dsl[cu13]>=4.5.0, quack-kernels>=0.4.0, torch-c-dlpack-ext, and a pinned sonic-moe @ Dao-AILab/sonic-moe@cfbd65f..., all Linux-only.

With the manual two-host H100 canary holding from last week, the active surface this week was kernel-level: closing the remaining gap between Grug's local MoE block and the SonicMoE/QuACK reference, and getting GPU attention off reference XLA. @dlwh opened #5627, which wires Grug attention to a Tokamax flex/Pallas backend so segment IDs work without dense mask materialization. On 1x GH200 at CoreWeave RNO2A, BF16, B=8 S=1024 Hq=16 Hkv=4 D=128 with four packed segments and a causal sliding window of 256, forward attention went from 0.920ms on Grug reference XLA to 0.366ms on the Tokamax flex path. The forward backward gap, however, was initially blocked on NotImplementedError: vjp not implemented from Tokamax; commit d4deabe9e on the same branch then routed range-representable Grug masks through Tokamax FlashAttention Triton forward and backward, leaving the older flex plus reference-VJP path selectable via prefer_flash=False. The dedicated tracker for that fast backward, #5658, sets an acceptance bar of at least 1.5x forward+backward speedup over reference XLA with no full-mask materialization.

On the local expert-MLP block, @dlwh opened #5718, a production-facing local backend named sonic that invokes SonicMoE's raw Triton token gather/combine through jax-triton with a custom VJP while keeping grouped GEMMs in the existing JAX ragged_dot path and the concat W13 layout. It removes the earlier sonic_xla and down_gather reference backends and wires jax-triton and triton into the GPU extras. The umbrella issue #5328 records the current state on the primary GH200 shape (tokens=8192, hidden=2048, intermediate=3072, experts=8, topk=2, BF16): forward at 1.285ms JAX/Grug vs 1.260ms real Sonic (+2.0%), no-remat fwd_bwd at 4.035ms vs 3.428ms (+17.7%), and honest checkpoint/remat at 5.844ms with prevent_cse=True vs 4.798ms (+21.8%). The remaining gap is in backward scheduling and materialization, not the isolated forward combine kernel. The narrower dispatch/combine tracker #5609 places Sonic Triton roughly 1.5-1.7x faster than the current JAX/Pallas gather/sum at repeat-1 wall-clock, after working around the fact that jax.lax.optimization_barrier does not lower in Pallas GPU and the naive repeat loop was being optimized away.

The B200 line moved from open question to characterized failure. #5675 caught a 2x B200x8 Grug MoE dense-FP8 long run silently wedging inside py_xla_execute shortly after its step-5000 checkpoint completed to R2, while Iris and Kubernetes still reported the pods as healthy. Rank 1 had logged CUDA error: Failed to destroy CUDA graph: CUDA_ERROR_INVALID_VALUE, but the message originates from the OpenXLA CudaCommandBuffer destructor, which only does LOG(ERROR) on failure and does not propagate a status, so Grug's try/except BaseException around the train loop never saw it. A matched BF16 run reproduced the same hang at step ~165, confirming the failure is not FP8-specific. Resuming with XLA_FLAGS=--xla_gpu_enable_command_buffer= empty crossed the next checkpoint boundary cleanly, uploaded a profile, and a long BF16 run with the same flag reached 27.1k/38.1k steps at about 1.1s/it without recurrence; @yonromai confirmed that mitigation is good enough to unblock the B200 pool while a tighter upstream repro remains elusive. As the operational hardening, #5677 added a ForwardProgressWatchdog callback that records a heartbeat on every on_step, runs a background daemon thread, and on timeout dumps Python stacks and SIGABRTs the process so the next silent native wedge exits nonzero with state rather than holding nodes indefinitely.

Cross-cutting with the FP8 thread on #5358, #5589 from @dlwh added a shared Grug linear helper with configurable dot_general state and wired base, modular_opt, and MoE dense projections through it, with an fp8_dense config that swaps in FP8 overwrite state while keeping raw array weights stable; @ClassicLarry pushed back on landing it on the base path, preferring a separate Grug subfolder until FP8's role in real runs is clearer. Net for the week: GPU attention has a fast forward and a now-callable fast backward path, the local MoE block is within 2% of Sonic on forward and inside 20% on remat backward, and the B200 hang has a working mitigation flag plus a watchdog to catch the next one.

A cross-cutting design landed in #5595, proposing to consolidate the seven separate *-unit.yaml GitHub Actions workflows into a single marin-unit orchestrator. The design routes through infra/select_tests.py to compute a per-package test matrix from the diff and dispatches legs via a new [tool.marin.tests] table declared in each package's pyproject.toml, with the end state being a unified marin-unit.yaml plus marin-integration.yaml covering the Python library packages.

On the housekeeping side, several long-running infrastructure tickets were closed as stale this week: #2749 (migrating the iris GCP platform layer from the gcloud CLI to the native Python client libraries for faster, lower-CPU controller operations), #2930 (live monitoring and stall detection for training jobs to catch gradual degradation that Levanter's NaN/Inf check misses), and #2865 (a proposal to add OpenAI Codex as a GitHub Action alongside the existing Claude Code workflows for side-by-side agent comparison).

External GitHub activity centered on optimizer and SWE work. @varunneal opened #5625, a Levanter port of the Dion optimizer with full sharded multi_transform routing and 11 CPU unit tests, untested on TPU but staged for stable-run validation. @AlienKevin continued driving the SWE-ZERO midtrain track in #4898 and #5611, the latter mirroring the 32K Marin-8B SFT runs at 8K truncation. @teetone filed #5687 reporting stale paths in the distributed GCS storage scanner, closed within the week. @MarinaZhang added herself to the Iris researcher budget in #5748.

On Discord, @willheld announced the Delphi blog and HF collection covering 88 base models from 3e18 to 1e23 FLOPs, with follow-up questions in #moe about whether the scaling-law work extends to MoEs. @Michael Ryan opened the distributed-inference design conversation, proposing a shared service to replace his WARC-region-specific setup. In #data-curation, @Moo Jin Kim and @JeniaJitsev coordinated on OpenThoughts4 generation strategy with Benjamin Feuer (LAION), converging on MV-16k as the training target.

Ten people joined this week, mostly researchers and engineers at adjacent shops: Varun (stochasm) and Charles Goddard, both pretraining at Arcee AI; Vladimir Salnikov from Danish Foundation Models working on Scandinavian pre-training and instruction data; Trishanu from MSR India on large-scale retrieval; Alexey, a LAION postdoc on referring video object segmentation; Gonzalo Fuentes, Tech Lead at Cenia on LatamGPT; and Jaisidh Singh, a Tübingen/MPI grad student writing a thesis on scaling laws for linear-dense hybrid attention LLMs. Singh's thesis topic intersects directly with the hybrid-attention threads circulating in #speedrun around the Olmo Hybrid post, and Goddard — author of MergeKit — lands as the team is increasingly thinking about model-merging and distillation pipelines for the June run. Silent joiners through the welcome room included Nishad Singhi and Oussama Elachqar, both with public ML-research footprints, but neither posted introductions.

News-channel sharing this week was dominated by data-curation drops — eight links covering LAION's OpenThoughts3/4 derivatives, MixtureVitae, Dolma3, and Nemotron-CC-v2 — alongside two scaling-laws papers and the Olmo Hybrid post-mortem, mirroring the project's current bottleneck on data mixture for the June model.