Marin: Week of May 4th summary

The week's defining incident was a coscheduled-gang placement collision that @ahmeda14960 opened as #5470 on May 6: two coscheduled v5p multi-host jobs dispatched within seconds of each other for the same (variant, region, band) tuple landed on identical physical worker hosts, then raced on the JAX coordinator port 8476 and racked up 700-1000 preemption cycles before iris gave up. Russell's first fix #5490 closed a specific window during _requeue_coscheduled_siblings where committed TPU capacity was decommitted in the DB transaction while the StopTasks RPCs for still-running sibling processes were sent after commit — the scheduling thread ran in between and assigned a second gang to workers that still had processes from the first. That didn't fully clear the problem; the recurrence on a v5p-128 on May 8 prompted #5572 for two related coscheduled-failure bugs and Ahmed's #5475 / #5472 drafts to revalidate assignment commits against live worker capacity and reject overlapping coscheduled TPU gangs before they share hosts. Russell's deeper rewrite landed Friday night via #5563, which replaces the controller's mutable per-worker capacity counters with a view derived from task_attempts — a worker holds R resources iff there exist task_attempts rows with worker_id IS NOT NULL AND finished_at_ms IS NULL summing to R — and #5550 moved all kill obligations into a KillRegistry that controller paths flush after commit, with scheduler exclusion now lasting until heartbeat confirms terminal state. Russell summarized it in the thread: the fix should resolve the two-jobs-on-the-same-worker collision, and Ahmed confirmed training jobs going again later that night.

The other dominant arc was the stats-service cutover finishing what #5212 and #5370 started two weeks ago. @rjpower's #5559 moved worker last-heartbeat / healthy / committed-* state out of the SQLite workers table into in-memory WorkerHealthTracker / WorkerCommitTracker, eliminating a per-heartbeat writer transaction that was bloating the WAL and starving dashboard reads; #5454 shared one read snapshot across the four queries in ListJobs / GetSchedulerState and bumped the per-conn page cache to 64 MB, #5546 replaced the per-request DB fan-out with a generic SnapshotView helper rebuilt every 2s, and #5412 turned the scheduler's inner fan-out from O(pending * workers) into O(jobs * workers) via per-job dedup and cached resource scalars — 312ms to 13.7ms on a 10000-task / 500-worker stress. Together they fixed the 212-second ListJobs Russell flagged in #infra earlier in the week, and #5417 / #5453 finished pruning the dead worker_task_history / resource_usage paths that the finelog migration had left behind. #5583 then moved the controller's 10-minute periodic CPU profile loop and attached profiles.sqlite3 out to the iris.profile finelog namespace — the third major dataset (after iris.task and iris.worker) to leave the controller — with design doc #5573. #5279 also finally merged, moving the executor's DAG walk out of the launch entrypoint into the training worker so a job preempted in us-central1 and rescheduled in us-east5 re-tokenizes locally instead of paying cross-region egress; it briefly broke legacy executor clients that still read JobInfo.worker_region, fixed by @yonromai's #5543 and #5544.

finelog itself spent the week hardening under production load. #5456 replaced the flat tmp_/logs_ compaction with a leveled scheme (L_n → L_{n+1} promotions, 64 MiB / 256 MiB level targets, files renamed to seg_L<n>_<min_seq>.parquet via migrations 0003 and 0004) and decoupled remote copies into a separate CopyWorker driven by the catalog. Migration 0003 then failed on production catalogs — DuckDB rejected CREATE INDEX after the in-txn UPDATE, then rejected DROP COLUMN once the index existed — and Compactor.plan OOMed on a backlog of 1001 L1 segments by trying to merge the whole contiguous run; #5476 split index creation into migration 0005, capped each compaction job at a target-sized prefix, opened a fresh DuckDB conn per merge with temp_directory pinned, and raised the container nofile ulimit. #5540 then deleted CopyWorker entirely in favor of a per-namespace _sync_step inside each DiskLogNamespace's bg loop, modeling segments as location ∈ {LOCAL, REMOTE, BOTH} with crash-safe adoption. #5441 added a segments catalog so the dashboard reads namespace stats without per-namespace count(*), and #5466 capped GCE finelog containers at nproc - 0.5 CPU and memtotal - 512 MiB after a runaway threatened to lock @ravwojdyla-agent out of the host. #5606 closed #5392 — literal job keys containing scientific-notation experiment names like 9e+20 were silently dropped because REGEX_META_RE reinterpreted them as patterns — by adding an explicit MatchScope enum (EXACT / PREFIX / REGEX) to FetchLogsRequest, and #5422 routed LogClient.fetch_logs through LogService.FetchLogs directly instead of translating to SQL on every poll, plus a safe_deploy.py that captures the running image digest before bootstrap and auto-rolls-back on health failure.

On the GPU side, @yonromai's #5428 completed the JAX 0.10 / CUDA 13 migration that was forced by last week's NCCL-on-CoreWeave-H100 crash: the CUDA 13 reproduction had pinned an H100x8 profiler crash to JAX 0.9.2, and JAX 0.10 passed the same raw profiler repro, the stock Levanter profiler window, and the full H100x8 CoreWeave canary, so Marin and Levanter GPU extras now run CUDA 13 / JAX 0.10 while CPU, TPU, and vLLM stay on 0.9.2. #5425 adds an opt-in cuda13 extra for B200 jobs that floors Linux cuBLAS above JAX's warning threshold. #5420 bootstrapped two new CoreWeave clusters under Iris — RNO2A (GH200, arm64) and USW09B (B200, amd64) — and switched storage from R2 to CoreWeave Object Storage at https://cwobject.com, with experiments/dedup/fineweb_10bt_exact.py running end-to-end on RNO2A. #5461 restored NCCL_SOCKET_IFNAME in defaults.task_env for the three CW configs after the bootstrap inadvertently dropped it, and #5479 fixed the CoreWeave canary's controller-routing race by waiting for deployment/iris-controller to settle to exactly one ready pod before exporting IRIS_CONTROLLER_URL. #5542 made K8sService.port_forward self-heal through pod rolls and stream drops by spawning a watchdog that respawns kubectl on SPDY death, and recast the CW smoke as in-cluster iris jobs so the canary path isn't gated on a local tunnel. The 8-run CW canary still came in at 4-of-8 for the week against 5-of-7 on TPU; datakit ran 0-of-2, with #5499 disabling the broken datakit-sources-staged lane so only datakit-smoke gates the scheduled Slack alert.

Russell's other thread was supply-chain. The week opened with him noting on infra that the GHCR repo's intermittent unavailability was the original cause of the May 6 placement-collision incident, and that this was happening more frequently as worker count grew and would need its own mirrors. #5506 renamed dupekit to marin-dupekit and published it to PyPI via OIDC trusted publishing on every merge, removing the github.com find-links entry that consumers had been pinned against; #5547 unified the dupekit release workflow on the same nightly+stable cadence as the libs-wheels publish, with the nightly version sorting above the current stable via a -dev.<YYYYMMDDhhmm> suffix and [tool.uv] prerelease = allow. #5570 published the rest of the workspace's marin-* wheels, #5565 added marin-finelog alongside its sisters after @gonzalobenegas hit #5564 trying to lock fresh marin-iris from his bolinas-dna repo, and #5503 / #5504 stood up four Artifact Registry remote-Python repos and wired the AR PyPI mirror env-vars into worker task setup so iris installs survive pypi.org / github.com flakes. The CI plumbing got a parallel cleanup: #5354 moved per-workflow logic into scripts/workflows/ and SHA-pinned non-trusted actions, taking the audit from 38 failures to 0 and closing #5067; #5595 proposed consolidating seven *-unit.yaml workflows into a single marin-unit orchestrator driven by per-package [tool.marin.tests] TOML tables; #5439 rebalanced the datakit CI ferries into three doc-length regimes all sized inside the GH-hosted 6h runner cap, with #5494 capturing per-step wall times and failure-bucket counts from iris job summary --json into a 90-day artifact after each scheduled ferry; and #5497 stopped the Claude review workflow from running on draft PRs. @wmoss's #5442 design doc and #5481 implementation move Zephyr's scatter internals (shuffle, external_sort, spill) from Python objects + cloudpickle to Arrow, after profiling showed Python-processing jobs spending 50-70% of their time in GC. @ravwojdyla's #5348 landed the long-pending Zephyr performance playbook, including a 10x-fineweb_edu variance baseline.

@yonromai spent the week converging on the shape of the Iris inference service. #5326 deleted the vLLM Docker sidecar entirely — the DockerVllmServerBackend class, the --mode/--docker-image smoke-test flags, the Docker image constants, and the planning doc at .agents/projects/vllm-docker.md — leaving native vllm serve as the only supported path now that Iris workers don't mount /var/run/docker.sock. A fail-fast check for MARIN_VLLM_MODE=docker stays in so stale job configs error early, and Harbor launcher validation rejects stale coordinator-side env vars before creating a remote step. Validation was a full v5p-8 smoke on Iris that served a request in 121s.

The bigger shift is the design RFC in #5400: a vLLM-only Iris inference service where evaluators keep talking to the existing RunningModel / OpenAI-compatible HTTP abstraction while Iris owns vLLM lifecycle, readiness, request brokering, cleanup, and diagnostics. Standard-model validation is the MVP; prompt-logprob scoring is parked as follow-up compatibility work. The companion routing MVP #5351 exercises the contract end-to-end with typed opaque OpenAI HTTP envelopes, an in-memory broker actor covering submit/lease/complete/fail with lease expiry, idempotent duplicate submit, and first-terminal-result-wins, plus an OpenAI-compatible proxy for /v1/completions and /v1/chat/completions and a worker actor that leases broker work and forwards to a running engine URL. The eval runner sees only an OpenAI API root and a model name, not whether the backend is Levanter, vLLM, or a deterministic stub — exactly the boundary needed before turning the manual milestone (mmlu_sl_verb_5shot and humaneval_5shot on the 1e22 MoE on v5p-8) into something pre-emption resilient. @dlwh's #5488 overrode lm-eval's metadata to drop the unfixed sqlitedict deserialization advisory — safe because Marin doesn't touch lm-eval's request-caching path.

The week's vLLM debugging surfaced exactly the kind of correctness gap the service needs to absorb. @RohithKuditipudi flagged in #inference that vLLM TPU logprobs are off by ~0.5 nats / ~5% relative vs HF and Levanter even at fp32, and anecdotally diverge more at bf16; HF on CPU and Levanter on TPU agree more closely with each other than either does with vLLM TPU. @dlwh attributed it to the loose attention tolerances of the vLLM kernel, and @ahmeda14960 conjectured a ragged-paged-attention version skew between vLLM and Levanter before walking it back. @yonromai opened a separate thread asking why a full fork of vllm and tpu-inference was preferred over patching them inside Marin, with @ahmeda14960's reply noting patching only ever worked for local checkouts.

On the eval-harness surface, @willheld reminded the #evals channel that the only way to add custom utils.py scoring logic today is to PR upstream lm-eval or point Marin at a fork, with @timodonnell running 8192-token rollouts through vllm offline for the protein-eval work that motivated the question. Three older eval-infra tickets closed out alongside the design push: #4269 (single way of running jobs — off Ray completely), #4088 (migrate classification inference to Fray & Iris), #2500 (set up agentic eval, with Harbor as the chosen framework), and #2614 (Terminal-Bench-Science). #5401 remains open as the live consumer: @calvinxu's 300M agentic-coding BPB eval is running the full 262-checkpoint sweep on v5p-8 in us-east5-a, exactly the workload the new service needs to keep alive through worker preemption.

The perplexity-gap suite expanded sharply this week as @dlwh landed four new slice families and queued six more for review. #5129 added byte-preserving structured-text PPL slices for ToTTo, WikiTableQuestions, GitTables, and WebDataCommons; #5128 added opt-in formal-methods and hardware-RTL slices behind a shared archive downloader; #5196 stood up a manifest-backed raw lm-eval bridge that stages MMLU auxiliary_train and GSM8K train; and #5189 registered SVG-Stack, TextOCR, and OCR-VQA behind a reusable raw-web-markup PPL helper. Still in review are paired-robustness slices for PAWS/FLORES at #5120, a switch to the uploaded HF synthetic-reasoning validation corpus at #5133, and the tokenizer-axis long-tail diagnostics scaffold at #5085.

The 32B all-available gap dashboard run finished, but only after skipping several slices that broke during staging. @dlwh filed five follow-ups to restore them: #5532 for structured_text/totto, #5533 for wikitablequestions, #5534 for gittables, #5535 for lm_eval/mmlu_auxiliary_train, and #5536 for wiring the #5052 synthetic_reasoning_ppl provider into the all-available run. Claude returned fixes for three of them within the week: #5539 pins TOTTO to the refs/convert/parquet revision and teaches the matcher the auto-convert layout, #5538 drops the hard subject requirement so MMLU auxiliary_train rows materialize with a subject-agnostic fallback, and #5537 matches prefixed split parquet shards like gittables' corpus_train.parquet.

Two longstanding tracking issues closed this week. #5247 — identifying a useful set of evals — wrapped with a 300M raw-metric matrix of 242 signal rows over 1073 metrics plus paired fixed- and variable-subset noise baselines, and a conclusion from the conversation with Will that correlation against accuracy is misleading because accuracy is itself noise; the operational rule is to use the smooth proxy with the highest SNR. #5254 — expanding programming-ecosystem coverage in the gap suite — closed after @dlwh signed off on slicing held-out eval sets per language out of Stack v2 (≈1M tokens for the big languages, 100K for small, skip languages under 2M).

Discussion in the tokenizer channel opened a new axis on the same problem. @dlwh flagged that whitespace handling and missing coverage of long runs of ideographic space (U+3000) keep surfacing in his gap work, and proposed running several models and their tokenizers through the whitespace-sensitive slices to get distributional reads on document length and distinct whitespace tokens. @dokhyon volunteered to extend the #5085 scaffold to cover the descriptive metrics and additional tokenizers.

The May milestone kicks off with the data pipeline epic #5360 establishing baseline throughput numbers for the two off-the-shelf quality classifiers the team plans to score every Datakit doc with. @ravwojdyla benchmarked the allenai/dolma3-fasttext-quality-classifier and ibm-granite/GneissWeb.Edu_classifier on a 250k-doc sample from Nemotron CC #5406, landing at roughly 1.7k and 2.1k docs/s single-core; the extrapolation says all 95 normalized Datakit sources score in ~16 hours on a 256-core cluster, and a follow-up estimate for the 15B-doc full corpus puts both classifiers at around 4-6 hours on 1000 CPUs. A parallel bench of the Luxical-One bag-of-words embedder #5410 came in at 421 docs/s at cpu=16 — about 14x below Datology's blog number — with scaling stalling past 8 cores on Iris's TPU-host VMs; @ravwojdyla reproduced a closer 2700 docs/s on a MacBook Air via the native Luxical API, suggesting the gap is a mix of sentence-transformers wrapper overhead and the e2-highmem hardware floor.

Underneath that, the Datakit pipeline got the partition-aware plumbing it needs to route co-partitioned attribute datasets without filesystem enumeration. @ravwojdyla stamped a partition_id column on every normalize row and lifted num_partitions onto NormalizedData in #5437, then stacked #5523 on top so fuzzy dedup routes records back to their source partition via the column instead of reconstructing paths; MinHashAttrData widens to {id, partition_id, buckets} and bumps to v2 with the v1 cache still readable. #5438 splits processing/tokenize into a datakit-style attribute stage (parquet of {id, input_ids} per doc) and a separate build_levanter_store stage that consumes one or more tokenized sources, while #5397 rewrites compute_minhash_attrs on top of zephyr's standard from_list → flat_map → write_parquet with skip_existing=True, so a partially-completed minhash step now resumes per shard instead of recomputing from scratch. Fuzzy dedup also picked up a regression suite at two layers — parametric LSH tests at (b=26, r=11) plus real-world data_integration scenarios against pinned HF fixtures — in #5436.

Source-side work continued to chip away at the long tail of normalize quirks: @ravwojdyla fixed worker OOMs on the largest HPLT tier-5/6 shards by switching _download_and_filter_shard to a true streaming yield in #5403, carved cp/biodiversity out of common-pile and stitched per-page scans into per-item documents in #5408 (with a follow-up #5451 after the reducer had to be a generator for zephyr to flatten correctly), renamed the HF download knob and bumped its default from 8 to 64 workers in #5489, and added a --downloads-only flag plus iris-region-driven MARIN_PREFIX to the renamed trigger_sources.py in #5492. New corpora landed too: @Helw150 added Amazon's MASSIVE multilingual tool-use dataset (11.39B tokens) in #5339, @dlwh wired sample-capped public diagnostic log sourcing for GHALogs/LogChunks/LogHub in #5121 (part of #5094, where the team's running inventory of public diagnostic-log corpora lives), and @Helw150 staged a SWE-Rebench v2 ConTree-traced pipeline for Code-World-Modeling in #5276 — with an early projection of ~183-235B tokens at 23% of shards posted in #data-curation. On the tokenize side, @nevillelyh reworked the consolidate-removal experiment from #4814 into #5430 after finding that 1000s of ledger metadata reads carry real overhead — the ledger gets consolidated, the data files don't.

The week's most concrete signal on dedup parameters came from @ravwojdyla's first full-Nemotron-v1 fuzzy run, posted in #deduplication: 512 workers, a 12-hour connected-components bootstrap followed by ~40-minute iterations (10 total), and roughly 18% fuzzy duplication at the chosen parameters — plus a separate exact-dup pass showing ~4% exact duplicates across 10.3B Nemotron records. The Datakit testbed #5200 baseline arm #5308 reached its final 5622/5622 training step with loss settling in the ~2.7-2.9 band, while the fuzzy-dedup arm #5309 and the 50%-duplication negative-control arm #5310 are running with early signal showing up on the datakit W&B project. @ravwojdyla also opened #5519 to resurrect a V0 decontamination pipeline — the plan, sketched in #deduplication, is to follow the industry standard (ngram overlap rather than bloom filters, per @ahmeda14960's earlier guidance), use Dolmino's accidentally-leaked GSM8K as a known-recall fixture, and ship validation datasets before running on the full corpus.

The preregistered 1e23 MoE run from #4697 (sized off the #4447 isoflop fit at 48 layers / d5120 / 131B total / 16B active / ~1.02T tokens, with a paloma macro target of 2.25 against a pinned L_inf=1.6 asymptote) crossed 58% completion mid-week, with @dlwh posting the curve in #moe on 2026-05-08 and noting that the loss shape after the initial phase is largely the LR schedule talking — @Kaiyue-Wen took the bet that adding a slow sine to the schedule would reproduce the wobbles. @ClassicLarry also shared the month-over-month aggregate-progress figure for the recipe at 3e18 FLOPs as part of an internal presentation. The 2.25 target remains an open bet.

The headline ablation result was @ClassicLarry's 256-routed-experts run #5387, which promoted decisively: e256 beats the e64 baseline by 1.170x effective speedup at d768 (3.3851 vs 3.4339) and 1.252x at d1024 (3.1138 vs 3.1605), with the quality gain outrunning the TPS penalty at scale; the d512 point was a wash (~14% TPS hit) and d1280 OOMed at step 0 on both v5p-8 and v5p-32. Most of the rest of the agent sweep landed as cleanly-documented negative results: GELU in GatedNorm #5302 failed d768 and closes out the silu-vs-relu/relu2/gelu question; GEGLU on the MLPs #5407 gave a 1-2% quality win that the GELU cost ate; deeper networks at compute-optimal budget #5423 underperformed the scaling prediction at d512/d768 and crashed multi-host at d1024; pick-5-from-64 with a half-sized shared expert #5424 was loss-flat at a ~9% TPS cost; the router-LR sweep #5478 found a tiny d512 win without z-loss and a d768 win with z-loss but turned negative at d1024, so default router LR stays; and the sequence-length sweep #5493 kept 4k as the d512 best but showed 8k beating 4k at d768, which Larry reads as a signal that larger sizes will want longer context.

The optimizer line moved more than the architecture line. @ClassicLarry's nano-walks investigation #5585 rebuilt every architectural feature of grug/moe one knob at a time on top of modded-nanogpt, and confirmed that Muon's 30% nano advantage carries all the way to the d768 compute-optimal recipe on in-distribution paloma — but that Muon trails AdamH by a wide margin on OOD code, with the C++ per-token-loss plot striking enough that Larry's working hypothesis became that the role of architecture is to minimize train loss while the role of data is to make sure minimizing train loss gets us what we want, prompting a StarCoder-up-weight slider experiment with @willheld in #moe. Larry also kicked off GrugMuon — pick-2-from-32 with hidden-dim-width experts and Muon on expert MLPs — as a 30-run sweep in #5517.

Meanwhile @Kaiyue-Wen's MuonH matrix-optimizer-swap ablation in #5596 / PR #5597 swapped AdamH for MuonH on every matrix except the lm-head (which stays on AdamH) and kept Adam on the baseline Adam group. Gate 1 passed cleanly (1.33x at d512, 1.26x at d768) and Gate 2 passed per-scale at d1024/d1280, but the pinned-L_inf=1.6 scaling-law fit on all four points gave MuonH α=0.0906 versus baseline 0.0941 — a shallower slope that projects to 2.2626 at 1e23, +0.0106 worse than the 2.252 baseline. The stacked follow-up #5600 / PR #5601 raised router_z_loss_coef from 1e-3 to 4e-3 under the corrected mask and improved cleanly on the MuonH baseline at both d512 (3.7499) and d768 (3.3881), and by the end of the week Kaiyue-Wen was reporting in #moe a uniform 20–30% step-wise speedup over AdamH with no token-throughput regression — including on C++ — by holding the lm-head on AdamH while routing everything else through MuonH. A paired-experts MuonH variant (32 NS instances per layer instead of 64, square w_down at the Grug defaults) was queued as #5613, and Larry's deep-dive on the original router z-loss #5214 nailed down why the term works at all — it pulls all 64 logits toward x=-4 so sigmoid(-4)=0.02 gives every expert a 2% default weight, and the routed contribution gets compensated by stronger neuron firing rather than logit growth.

On the infra side that the scaling recipe sits on, #5350 merged the Triton ragged_dot backward swap and bought the Grug MoE H100x8 step a 3.1x steady-state speedup (0.530s → 0.171s) plus a ~7.5x drop in JAX 0.9.x compile temp memory (54 GiB → 7.2 GiB), unblocking the CoreWeave GPU canary. @yonromai's CoreWeave Grug-MoE accelerator validation #5458 closed as a findings report after H100x8/GH200/B200 warm-node smokes all completed 20/20, splitting the remaining work into the multi-host canary tracker #5452, the cold-start JAX rendezvous timeout #5480 (CW direct H100x8 x 2 fails before any NCCL signal because replica 0 enters initialize_jax 300 s before replica 1 is ready — draft fix in #5482 deferred), and the profiler bucket #5509 (closed and split into #5527 / #5528 / #5529, with PR #5524 the focused fix for the deterministic Levanter profiler double-stop). On the TPU side, @ahmeda14960 escalated #5470: the Iris coscheduler deterministically places concurrently-dispatched gangs on identical worker hosts, triggering 707-cycle preemption thrash on the same JAX coordinator port and burning an estimated ~1100 v5p host-hours across four 36-hour incidents on v5p-64/128/256; @rjpower has a real scheduler fix in flight at #5563, and his #5279 moving the executor inside the training job (removing Iris region inheritance) merged on 2026-05-07. @rjpower also opened #5549 to make Levanter checkpoints region-aware via mirror:// + resolve_tree, so a trial preempted in one region resumes the prior step in another instead of restarting from zero.

@yonromai landed #5350 on May 4, replacing the Haliax ragged_dot custom-VJP backward contractions with explicit Pallas-Triton kernels for both dlhs and drhs. The H100x8 Grug MoE train step dropped from a 0.530s median to 0.171s (a 3.10x steady-state speedup), and XLA compile temp memory fell from ~54 GiB to ~7.2 GiB. That memory drop closed out #5394, the regression where the JAX 0.9.2 stack had pushed the Grug MoE compile path within striking distance of the H100 80 GiB ceiling and was implicated in the CoreWeave GPU canary's RESOURCE_EXHAUSTED failures. @dlwh also merged #4867, fixing sender-side output offsets in Levanter's ragged all-to-all so returned slices land in the correct remote slots, with a ring-vs-ragged EP parity test.

The CUDA 13 / JAX 0.10 GPU stack landed as #5428, after #5425 was closed as superseded. @yonromai's validation in #5458 confirmed direct one-GPU smokes and short Grug MoE training/profiler runs passing on H100x8, GH200, and B200x8 against the new stack. Multinode H100 rendezvous was unblocked separately by #5482, which mapped Iris device-type/device-variant constraints into Kubernetes node selectors and added a pre-initialize barrier in initialize_jax() so task 0 stops burning the 300s init deadline waiting on a cold second H100 node. Two H100 canary issues filed mid-week, #5509 (segfault around profiler stop) and #5510 (Pallas fused cross-entropy autotune missing and falling back to XLA), have since been closed. The B200 multinode jax.profiler.stop_trace hang at #5528 is still open; @yonromai's draft #5576 adds a typed ProfileOptionsConfig and defaults multinode GPU canaries to device_tracer_level=0, which on a B200x8 x 2 run brought stop_trace from ~440s down to 3.1-3.4s.

On the systems-characterization front, @dlwh opened #5587 to track GH200 MoE measurements (single-host fit boundary, host offload, FP8, EP) and rolled up the first round of single-GH200 numbers against Megatron in #4311. After steelmanning the Megatron sidecar with NGC PyTorch, Transformer Engine, grouped GEMM, MoE permute/router fusions, and Apex FusedAdam, NGC Megatron-Core hit 23,622 tok/s on the 006a shape versus strict Grug at 18,388 tok/s — about a 28% single-GPU gap. The new epic #5608 coordinates closing that gap across three workstreams: #5609 replaces the MoE dispatch/combine scatter-gather kernels (currently ~46.6 ms/step in XLA fusions vs Megatron's ~12.9 ms TE+Apex permute/copy), #5610 adds a flash-like GPU attention path matching the splash-style signature for Grug masks, and #5328 ports SonicMoE-style GMM/MLP kernels.

The SonicMoE thread in #5328 produced a small but real end-to-end win this week. A full-W13 custom VJP via generic ragged_dot_general was a clear negative result (10.13 ms vs 2.78 ms autodiff). Narrowing to a Pallas direct-store kernel for just the expert-padded unpack_bwd hotspot hit ~0.39 ms versus 1.00 ms for XLA scatter-add, and promoting it as a custom VJP around only that sub-op improved whole padded-W13 backward by about 20% in microbench. End-to-end on the 1x GH200 006a shape that translated to 2.8% throughput: step time from 0.4455s to 0.4332s and BF16 MFU from 22.02% to 22.64%. @yonromai flagged on Discord (code-review) that #5350 was being merged specifically to get the GPU canary green, and later pitched a dashboard-style skill for tracking stable perf metrics against main's Grug MoE head over time.

The week opened with the largest single GPU-side win of the milestone so far: #5350 from @yonromai moved Haliax's ragged_dot backward off jax.lax.ragged_dot_general and onto explicit Pallas-Triton kernels for both dlhs and drhs. On H100x8 the Grug MoE train step dropped from 0.530s to 0.171s (a 3.10x steady-state speedup), and the JAX 0.9.x compile-temp regression tracked in #5394 went from roughly 54 GiB down to 7.2 GiB. The full CoreWeave GPU canary went green at 99 steps with final loss 6.5599. The follow-on stack change in #5428 then switched Marin and Levanter GPU extras to JAX 0.10 on CUDA 13, with CPU/TPU/vLLM held on 0.9.2 — chosen because JAX 0.10 was the version that survived the H100 profiler repro that crashed 0.9.2.

With the single-host path healthy, the focus turned to actually getting two hosts to rendezvous. #5429 enabled manual two-host H100 canary runs (scheduled runs still pay for one warm H100), and a representative dispatch scaled iris-ci-h100-8x from one to two nodes, ran the real Grug MoE canary across two distinct H100 hosts for 100/100 steps with final loss 6.6870, then scaled back. #5461 restored NCCL_SOCKET_IFNAME on all three CoreWeave clusters (enp157s0np0 for H100, eth0 for GH200, enp44s0np0 for B200), reviving a fix originally landed in #3806 and lost when the canary infrastructure was rebuilt. @yonromai also ran the first cross-accelerator validation batch in #5458, showing that the JAX 0.10 / CUDA 13 stack runs single-node Grug smokes on H100x8, GH200x1, and B200x8 — scheduling and plumbing evidence, not yet a benchmark.

Two-host cold start remains the live wound. #5480 documents a deterministic RegisterTask DEADLINE_EXCEEDED when replica 0 hits initialize_jax() before replica 1 has finished syncing dependencies; the draft fix in #5482 would have added Iris device-type/variant nodeSelector mapping, a pre-initialize readiness barrier, and a 1800s initialization timeout, but @yonromai closed it as deferred after concluding the current Grug perf work can proceed under a manual warm-node precondition; #5480 stays open as the tracked follow-up. #5452 records the parallel reality that no clean end-to-end two-host Grug canary has yet run on main. Around that, the canary lane shed a string of smaller failures: #5469 (controller routing, fixed by #5479), #5590 (fresh controller deleting the runner pod it inherited mid-rollout), the profiler segfault in #5509, and the Pallas fused cross-entropy fallback in #5510.

Looking past H100, #5528 caught CoreWeave B200x8 x 2 warm-node runs emitting usable metrics (~370k tokens/s, ~794k gflops/s) before hanging in the first jax.profiler.stop_trace() for ~10 minutes, distinct from the double-stop case being patched in #5524. And @dlwh opened #5587 — a GH200 systems-characterization plan using the two RNO2A hosts to attribute multihost communication, host-offload cost, FP8 stability, and MoE permute/GMM/unpermute share — feeding the new umbrella epic #5608 for closing the Grug-vs-Megatron GPU profile gaps. Net for the week: the within-host MoE step is now 3x faster, the manual two-host path has produced a clean canary, but multi-host on main still depends on warm nodes and a hand-driven controller.

A handful of changes landed outside the May milestone epics. @Helw150's long-pending #3092 finally merged, addressing the Qwen export bug reported against the marin-32b-base Hugging Face checkpoint — a four-line change that had been open since February.

External GitHub activity was light but pointed. @AlienKevin kept the SWE-ZERO line moving — #5588 to reproduce mini-coder-1.7b, #5611 to midtrain Qwen3-1.7B-Base on SWE-ZERO trajectories truncated to 8K, plus the parallel quality-validation and 140B-scaling tickets at #4898 and #4719 — and posted Terminal-Bench eval traces for Marin-8B and 32B SFTs in #evals, with Benjamin Feuer pressing on whether base-model NLP numbers extrapolate to the Instruct checkpoints. @leloykun, Franz Cesista, who has written extensively on Shampoo and Muon variants, opened #2695 to test Prism and Shampoo-Prism on top of the MuonH baseline that @Kaiyue-Wen landed this week.

The most substantive Discord moment was the MuonH thread in #moe, where Kaiyue-Wen posted the working configuration — swap every AdamH for MuonH except the lm head — yielding a uniform 20-30% step-wise speedup including on C++, tying back to #5596. @ClassicLarry's nano-walks deepdive in #moe nailed down where Muon falls short on OOD code in the full grug recipe. Separately, @ahmeda14960's #infra escalation on identical-host preemption thrash pulled @rjpower into the coscheduler fix at #5279 and the in-flight #5550, and @gonzalobenegas filed the agent-authored library-use issue at #5564 for the bolinas-dna experiment.

Nine introductions landed. Toekneechin works on Gemini posttraining RL for coding; Hafez Ghaemi is a Mila PhD student on SSL and world models; Weijun Li is a PhD student on security/privacy in ML/NLP; Nicholas Guttenberg is a semi-retired ML research consultant with Japan and ML-for-ecology collaborations; Yanhong Wu is a research scientist at Meta; Jordan (Frosty40) is a game dev moving into ML research; Natu Lauchande works in ML/AI engineering; Ridgway researches LLM and agent evaluation combined with psychology; Alon Lahav is a UCSD masters student in ML systems research. Toekneechin followed up the same day with a replay-buffer question for multistep RL environments in #reinforcement-learning; Ridgway's eval-and-psychology framing intersects the perplexity-gap suite at #5367. Among silent joiners in welcome-room, Malte Ostendorff (Occiglot multilingual LLM work at DFKI) and Xavier Gonzalez (Stanford, parallel-scan and state-space work) arrived without posting — their backgrounds bracket the data-pipeline and optimizer arcs currently active.

News-channel reading clustered on optimizer, tokenization, and architecture tradeoffs at scale: Michael Ryan posted Limisiewicz et al.'s compute-optimal tokenization thread with @willheld pulling in the earlier vocabulary-scaling paper as counterpoint, and Kaiyue-Wen surfaced the 4-bit-native nGPT paper alongside the multi-power-law LR-schedule prediction paper.