Marin: Week of May 25th summary

The production swarm launched this week, clearing the central deliverable that slipped last week. Will Held reported on #5365 that the swarm went up the prior day, a few days behind schedule, with two operational lessons logged for next time: ingestion onto the datakit store should be budgeted at roughly a week and given more lead time, and Iris should be stress-tested under full-ingestion load earlier, especially on CoreWeave. The swarm itself lands in PR #5958 as an 840-candidate Fisher DSP-aligned two-phase design — D512 MoE on ~100B tokens each over the us-central2 datakit store's 167 mixable domains plus a tail aggregate, with phase weights drawn from an offline D-optimal design and an 80/20 phase split. It ran live as iris-run-swarm_fisher_dsp-20260526-063341 with ~250 sub-tasks at submission. Will Held flagged the PR as not yet review-ready and noted the swarm is capped at 256 concurrent runs, so the D-optimal candidates are scheduled first and the proportional and per-domain baselines that Calvin Xu asked about will follow in a later batch.

Most of the week's thinking was on the metric-and-method side the epic requires de-risking before trusting the swarm, and the recurring conclusion was that there is no free lunch in the mixture. Working from the existing 240-run swarm, Calvin Xu showed sizable eval-loss crossovers — rank Spearman between the first eval and the final of only ~0.31 at 60M and ~0.23 at 300M — which undercut a tempting shortcut Will had floated of training larger models to a small fraction of the run instead of small models to completion scratch that idea then. When Percy Liang pressed on whether downweighting low-quality CC ought to give a free improvement, Will Held showed both HellaSwag and Paloma macro BPB are maxed within the swarm by proportional mixing, so any gains on code/math come as explicit tradeoffs. Calvin Xu separately found that reallocating low-quality budget to the matching high-quality bucket did not uniformly help HellaSwag, which convinced him quality cannot be modeled as a scalar discount.

That "no free lunch" framing motivated a tooling side-quest that produced this week's most concrete artifact. Will Held built a swarm explorer that, in its DSP-driven form, turns each benchmark slider into a steering wheel: dragging a task takes a minimum-norm step on the mixture along the analytical gradient of Calvin's DSP fit, so the mix only drifts as far as the demanded improvements force it make tradeoffs explicitly rather than implicitly. Using it, he hand-steered from token-proportional toward HumanEval, GSM8K, and (more gently) MMLU, and pre-registered the resulting mixture in #6063 before any training — five confident wins (humaneval, github_python/cpp, gsm8k, arxiv_physics), two confident losses on uncheatable news/fiction sets, and 17 tasks inside fit noise, with an explicit reservation that the confident-loss list is the shakiest part. The delta is concentrated in phase 1, which swaps general and high-quality CC for synthetic code, synthetic math, stack-edu, and arxiv. The stated point of pre-registering is the same discipline as the scaling-law work: keep the prediction honest and build confidence that capabilities can be controlled through the mix.

The downstream 10T June datamix issue #6045 remains open and unstarted, gated on this swarm resolving, while the metric-selection sub-issue #5362 is effectively the live debate above. On #2403, Calvin Xu backfilled the Grug-MoE v4 endpoint evals to full 20/20-task coverage at every hidden dim and added a standardized effect-size view showing the v4-optimized direction is positive on average but not Pareto-safe — its largest single regression (boolq, -1.16σ) exceeds its largest gain (medmcqa, +0.85σ).

The week's centerpiece is making Reconcile the single controller-to-worker task-lifecycle channel and then hardening its shape. #5964 deleted the legacy StartTasks / StopTasks / PollTasks RPCs, the UpdateTaskStatus push, and the rollout gate now that every worker is on the new wire — roughly -1500 LOC, with routing now purely by attempt_uid #5921. #5954 trims the Reconcile observation payload to the desired set on both ends, and the still-open #5987 refactor splits the state machine into snapshot reads, a mutation writer, and pure transitions.py logic, with an AST purity guard that forbids cur.execute and attribute reads across the seventeen hot-path functions. #5965 stood up a synthetic benchmark_reconcile that drives the full snapshot → apply tick against an in-process fake worker and surfaced the real costs at zephyr scale — a 4.3s dispatch tick dominated by the apply path, and a 478 MB wire payload at 5000 tasks — giving the refactor a target rather than a guess. Russell Power rolled the reconcile-based scheduler out mid-week with a few controller restarts, confirming the path that terminates unknown slices behaved in production.

The second thread is preemption correctness, which had been quietly burning the failure budget. #6000 (open) has workers poll the GCP metadata server for the preempt notice, latch a drain bit, and SIGTERM local attempts; the controller treats that bit as authoritative and atomically marks the worker's tasks plus coscheduled siblings PREEMPTED, eliminating the JAX peer-loss races that previously looked like genuine failures #5872. Around it, #6004 gives worker-restart a failure budget and classifies "slice not found in zone" as skipped rather than fatal so one TPU reclaim no longer tanks a rollout, and #5977 raises the BackoffDetector short-lived threshold so spot capacity that lives ten to twenty-five minutes decays group health via AIMD instead of reading as normal preemption. This lines up with what users were seeing on the ground — Larry reported a batch v5p-8 job preempted 32 times in three days, making about one-ninth forward progress — and with Will Held's #5953, which lets the v4 reservation back small slices (sizes 8 and 16) with chip-budget-derived slice caps instead of stalling at whatever happened to be pre-warmed.

The finelog migration that began last week with #5912 landed and then got tuned. #5988 drops SetTaskStatusText — the highest-volume controller RPC at ~125/s — and routes status text into an iris.task_status finelog namespace read back by the dashboard with a ten-minute retention filter, while #6003 adds per-namespace storage policies so finished-task rows roll off at 1h / 100MB. Rafal Wojdyla's #6012 then closed the last hot spot: in-task clients had still been writing those rows through the controller's StatsServiceProxy on its 64-thread pool, so a slow finelog backend could starve controller RPCs and fail jobs; the fix resolves the log server directly from the cluster registry. That change also flushed out a pyqwest bug on short-lived RPC sessions, which @rav confirmed the patched client resolves. Operationally, #6020 reaps hung py-spy profilers (and clears the ptrace group-stop a killed tracer leaves behind) that had been freezing zephyr shards for hours on both docker and CoreWeave, and #6078 stamps started_at_ms on the BUILDING transition so wedged image pulls finally fall into the execution-timeout scan instead of holding chips indefinitely.

Smaller hardening rounded out the week: #6018 lifts the RPC handler pool from 64 to 1024 threads so slow sync handlers can't head-of-line-block heartbeats, #5961 and #5952 move the worker-restart driver off a controller RPC and onto the CLI (fixing an AttributeError that broke every restart), and #5966 advertises the controller's SSH auth mode via a GCE label so marin.yaml can flip to OS Login without lockstep client upgrades. On the dashboard, #6017 adds a back-to-parent-job link. Notably, the epic's other named tracks — the zero-trust proxy and a GitHub-to-Iris integration — saw no movement this week; the energy went entirely into the controller core.

The brokered vLLM eval-serving path from last week, #5887, stayed open and unmerged through the week, still waiting on review from Russell Power. Meanwhile Ahmed Ahmed opened #5985 after trying to adopt the separate marin.inference.distributed library — the Zephyr-backed multi-region inference path on the inference/distributed-library fork — by vendoring only its subtree onto main. All cross-package imports resolve at static-check time, but at runtime regional_job._build_context passes a worker_environment kwarg that main's ZephyrContext does not accept. Running it end-to-end at 1e21 (3.4B) and 1e22 (9.7B) showed the cascade is three layers deep, not two: past a one-line lib/zephyr/ patch the worker actor then crashes on ZephyrWorker.__init__() got an unexpected keyword argument 'environment', which the fork only fixes via coordinated lib/fray/ changes that thread environment= through the actor-creation path. Ahmed Ahmed updated the definition-of-done to require those supporting zephyr and fray deltas land in main before the library is usable.

On the reliability side, Russell Power opened #6058, which gives the Iris K8s direct provider atomic gang admission via Kueue plain pod groups — the missing piece for preemption-resilient multi-worker jobs that the epic's def-of-done leans on. Previously the direct path had no atomic startup: a 64×8 job could bind 63 pods and strand the 64th behind the autoscaler, blocking the startup collective, with gang guarantees only on teardown. The PR promotes coscheduled gangs all-or-none on a single pod-group generation, deletes the backing Kueue Workload on teardown so a requeue does not deadlock behind a parked WaitingForReplacementPods quota hold, and bills a vanished coscheduled pod against the preemption budget rather than as a hard failure. It was validated end-to-end on a kind cluster — a gang that fits quota admits whole, a gang that does not stays fully gated with zero partial admission, and tearing down the first frees quota for the second to run.

Two scoping issues opened to shape the eval side of the service. Romain Yon filed #6043 for eval UX including a leaderboard, and we filed #6050 to enumerate and hook up the evals of interest (AA II evals, possibly SWE-bench) ahead of post-training; both still carry placeholder definitions of done. Separately David Hall's draft #5959 adds a Levanter labeled-LM-eval training callback for periodic pretraining probes — adjacent to this epic but a different code path from the served-eval service.

On Discord the epic drew new interest. In #evals Percy Liang connected a prospective contributor interested in helping add evals, and in #infra the same contributor asked what the happy path is for standing up an HTTP endpoint over a multi-host JAX mesh without Ray — exactly the serving shape this epic is converging on. The week's controller churn that the serving path runs under was also visible in #infra, where @rjpower rolled out the new reconcile-based Iris scheduler.

The week turned from building out the gap pool to reading the first real dashboard verdicts on it. David Hall stood up a prompt-format sensitivity suite in #6067, holding task semantics fixed while rendering the same 5-shot examples through 156 surface templates, and after a v2 renderer fix for JSON/JSONL continuations and record-extraction targets the readout was reassuring: Marin 32B sits roughly on par with Qwen3 32B across the slices and is often better, so broad wrapper/template fragility is no longer treated as a worry. The exception was Python REPL/reasoning continuations, which were split out into their own code-interpretation family under #6070 for static target-only records like len('prompt') * 7 producing a final output line, deliberately avoiding famous functions and keeping the scored span to the REPL/doctest result. In #evals on Discord, David Hall flagged this as a real gap area on the function-definition-calls heatmap and asked whether the CWM work might close it.

Underneath the new probes, the scoring machinery got more trustworthy. #6079 added supervised target-only scoring so datasets with separate input and target fields score only the continuation after a prompt, rather than treating each document as one span and contaminating metrics with support-prompt bytes; that machinery landed on main, and the actual prompt-format and code-interpretation suites were retargeted directly to main in #6080. Two BPB-accounting fixes under #5821 hardened the byte math: #6011 aggregates loss against decoded byte totals instead of token-weighted batch BPB and preserves loss from zero-byte token IDs, failing fast when perplexity-gap offsets are not source-aligned, and #6055 adds source-document BPB so TokenMonster control tokens group with the bytes they modify rather than silently dropping their loss. Pranshu Chaturvedi added native Grug checkpoint support to the model-score runner in #6056, handling the alternate max_seq_len and MoE-layout conventions so the d512/d768/d1024 MoE ladder being tracked in #5821 can flow through the same gap workflow.

Two cleanup items also merged: #5946 stops uploading model perplexity score bundles to W&B as tracker artifacts, writing them only to the GCS output path while keeping scalar logging and gap reports, and #5803 documents the end-to-end raw and supervised-target gap workflow from provider registration through Iris scoring, report retry, heatmap generation, and website publication. The FEVER factuality slice #5863 drew a useful exchange with Yiyuan Li on whether evidence-conditioned entailment is the right proxy; David Hall framed it as "throwing crap at the wall" to find ICL proxies for a model's awareness of evidentiary reasoning, with the real downstream goal being hallucination robustness left to post-training. On the adjacent pass@k front, Rohith Kuditipudi's #4549 got a full 36-cell GSM8K mask_00 readout across the Delphi midtrain matrix.

The epic closed this week. Rafal Wojdyla wrapped the last three deliverables — Dolma3 quality scores #5812, Luxical-One domain buckets #5808, and Dolma3 WebOrganizer topic buckets #5811 — into #5822, then closed #5360 on 2026-05-29. The independent metric the milestone was meant to climb finally exists: a blind LLM intruder study, where a judge picks the odd-one-out from four same-bucket docs plus one contrast doc, gives each tagger a single accuracy number. On domain, the v0 Luxical K=40 clustering beat the off-the-shelf Dolma3 WebOrganizer classifier cleanly, 0.710 vs 0.490 with separated Wilson CIs; on quality, the oracle-LLM-distilled v0 classifier edged Dolma3 fasttext 0.292 vs 0.214 (z=2.99, p≈0.003). Both quality numbers sit only modestly above the 0.20 five-way-chance floor, and Rafal Wojdyla framed this explicitly as a starting point — enough to begin, with June's job being to push these up.

The dedup side of "dedup param selection" got its first full-corpus run, reported in #deduplication: 15,089,658,952 docs in, 68.33% singletons, 31.67% landing in some duplicate cluster, of which 22.25% are removable non-canonicals — so the kept set is 77.75% of documents. Without FineTranslations (~3T), that takes a ~15T-token input down to roughly 12T deduped tokens. Doc-level decontamination removes only ~0.04% / 6M elements, which Rafal Wojdyla suspects carries a fair number of false positives, prompting David Hall to ask for examples of what gets thrown out. Notably, the long-standing #3101 ("ensure we have 20T deduped tokens") was auto-closed as stale rather than satisfied — the actual yield is ~12T, so the headline token target was retired rather than hit.

The data-mixing channel surfaced a result that bears directly on how these quality scores get used downstream: starting from proportional and shifting 50% of each low-quality split's budget to its high-quality counterpart did not yield uniform HellaSwag gains, which convinced the team that quality "should not be modeled as a scalar discounting factor" — a caution about treating the new per-doc quality scores as a simple reweighting knob. Forward work is now split out into June follow-on issues filed against this epic: #6036 (get the tokenized, mixing-compatible pretraining data into the CoreWeave cluster for the June run) and #6037 (a time-boxed process for climbing local dedup/decon/quality/domain metrics), with Will Moss's Nemotron-ferry tuning #6065 still open on the throughput front.

The epic's definition of done—a scaling recipe with isoFLOP results good enough to forecast the June run—effectively landed this week. Larry Dial closed out the four-budget isoFLOP sweep in #6074 (1e18 / 3e18 / 1e19 / 3e19 across hidden dims 512–1280 on v5p-32), fit a parabola in log(tokens) at each budget, and tied the optima together into a scaling law: loss(C) = 1.6 + 88.90·C^-0.0941 on the drop-1e18 fit, which carries the same exponent as the v16 README curve and therefore implies a uniform ~2.12× equal-TPS speedup at every budget for the combined MuonH May Recipe. Projecting forward, the May Recipe reaches the v16 baseline's 1e23 loss at roughly half the compute, and Larry published a compute-optimal recommendation table—compute, tokens, batch, steps, and LR per dim—mapping straight onto the June shape (e.g. d=1280 → 3.46e19 / 1.5e10 tokens / projected loss 2.889). He explicitly down-weighted the 1e18 bowl, where Muon does disproportionately well at tiny scale with 1pct warmup, and flagged it as a mild extrapolation he'll revisit if compute opens up above 3e19.

Feeding that table is the refit LR scaling law in #5951, now landed in heuristic.py as muonh_lr = 18.31 · tokens^-0.395 · dim^-0.150 · batch_size^0.5 (R²=0.996 over 17 cells). Larry caught and corrected a real bug along the way: the first fit fed tokens = R·p_active with intermediate_dim = d×4 while the actual heuristic uses d/2, inflating the coefficient ~1.74×—worth noting because the surprising ~60% LR jump it initially implied turned out to be partly that units bug, not a real shift. The corrected law still raises LR ~15–20% over v16 at the optima and steepens the token exponent (−0.395 vs −0.281), which in turn pushes the June compute-optimal points to ~1.7–1.8× larger budgets per dim with ~20–30% lower LR. The notable absence is preregistration: we opened #6046 to determine the path to register a loss guess for the June '26 run, and in #news Percy Liang pressed to actually name and version the dataset (marin-data-0.1) the June model will train on.

The optimizer search that has been running under agent.md autonomy mostly served to lock the recipe by ruling alternatives out. MuonEq #6066 failed gate 1—its pre-orthogonalization equilibration costs 5–6% throughput for no loss gain (effective speedup 0.94 at d512); the AMUSE schedule-free + hyperball tuning #6064 was closed not-planned after failing to beat the MuonH baseline; Kaiyue Wen's KLSOAPH port #5728 got an upstream-parity audit that found five silently-simplified primitives, and a full-matrix SinkSOAP reparam variant #6082 is now training stably at precond_freq=10 (~292k tok/s, ~13% under MuonH's 333k). On the architecture side the sandwich gated post-norm #5938, GLU-vs-SwiGLU #5934, and parallel attn/MLP layouts #5932 all failed gate 1 (parallel layouts bought 7–10% throughput but the loss regression swamped it), and the muon_epsilon sweep #5933 confirmed the recipe is insensitive there—so the default 1e-8 stays and the knob is kept out of the scaling heuristic. None of these displaced the canonical baseline, which is exactly what "land the recipe" needs.

Tooling and the GPU hand-off filled in the rest. David Hall merged final-only checkpoint retention for the Grug launchers #5945 and factored the whole MoE experiment workflow—gates, effective-speedup math, Iris submission, W&B monitoring—into a reusable agent skill #6068, which is what has let the gate-1 experiment fan-out run largely hands-off. The open #6069 reshards the MoE block back to batch/expert layout before the experts run, removing the attention-to-MLP layout cliff under FA4; the motivation showed up starkly in #gpu, where dlwh's B200 profiling found Grug crushing Megatron at d1024 (373k vs 166k tok/s) but collapsing at d5120 (14–24k tok/s)—a routing-layout problem he believes is addressable, and one that matters because the June hero run is a GPU run. Larry also wrote up the recipe's scaling philosophy in #news: keep expert count at 4-active-of-256 (×sparsity always wins on loss; granularity past 4 active hurts MFU), ~1 layer per 100 dims, and init_std = 0.5/hidden_dim to hold activation magnitude constant as width scales. Externally, Poolside's Laguna report—flagged by Larry in #news—independently uses the identical lr = a·tokens^b·size^c·bs^0.5 transfer formula and a proxy-swarm data-mixing approach, a useful convergent signal that the recipe is on a defensible path.

Last week's FA4/CuTe attention backend crossed Transformer Engine on a single B200 shape; this week David Hall pushed on the THD/varlen direction and produced the first full-node B200 profile against Megatron. The headline attention result came out of the probe issue #6039 and the draft backend #6059: reshaping packed BSHD to compact THD with cu_seqlens built from contiguous segment IDs and calling upstream FlashAttention-4 CuTe varlen kernels roughly halves attention-core time on the exact d5120 Grug shape (B=1, S=4096, Hq=40, Hkv=8, D=128, four packed segments), measuring 0.000665s fwd+bwd versus 0.001308s for the current segmented FA4/CuTe path. Getting there took a long sequence of false starts documented in the probe thread: a private JAX cuDNN packed-offset path looked fastest but failed numerics on the GQA shape (Hq=40, Hkv=8) and was discarded, and the win only held once timing used FA4's internal fwd/bwd APIs and cu_seqlens was built ahead of the timed core. The integration in #6059 evolved from a Torch/DLPack bridge to a jax.pure_callback path and finally to a direct cutlass.jax.cutlass_call into the CuTe SM100 kernels, leaving no Torch or DLPack on the JAX-facing path; it remains narrow by design (causal self-attention, fixed max-segment metadata, SM100/SM110, head_dim 128, GQA).

The week's most consequential finding came from the reproducible BF16 B200 baseline #5815, which moved off the old single-GH200 synthetic profile to fair-fit d5120/L8 full-node runs. With Megatron given its native stack (Transformer Engine, flash attention, native DeepEP), the gap is stark: Megatron reaches about 91.6k post-profile tokens/s at d5120/L8 while Grug reaches only about 14.2k with ragged_all_to_all and about 24.4k with a ring/all-gather comparator. David Hall summarized the scale crossover in #gpu: at d1024 Grug is far ahead (373k vs 166k tok/s), but at d5120 Megatron scales much better, and the regression traces mostly to MoE routing. The conclusion in #5815 is that ragged_all_to_all is a real bottleneck but not the whole gap, motivating per-block forward/backward timing across attention, routing, dispatch+MLP, and optimizer.

Two new design issues were carved out of the THD work to push the layout story end to end. #6071 proposes making the Grug block's internal residual stream token-major x_td [T, D] with THD metadata as the attention ABI so attention and MoE share an execution layout, keeping external APIs BSD and measuring pack/unpack and reshard costs separately; its TPU subissue #6072 tracks a packed-THD TPU attention kernel that skips inactive blocks rather than treating the batch as dense padded BSD. The B200 MoE GMM tuning issue #5894 and the segmented-attention issue #5896 were both consolidated into managed experiment TL;DRs this week (the GMM result is a ~1.19-1.20x Blackwell BF16 forward win from an m128_n256 tile, still narrow to one shape class). The longer-horizon FP8 #6048 and pallas xent dtype (#2988, now closed/stale) threads saw no movement, and the parent #6049 sets a June 15 deadline for roughly out-of-the-box Megatron MFU.

No multi-host H100 run was attempted this week, so the definition-of-done — training a June 16B-A2B MoE for ~1k steps on 2+ GPU hosts — is still unmet. The week's substantive move was planning: we scoped the downstream run target #6044, a 10T-token pre-training-plus-mid-training run with a 120B-A2B MoE whose def-of-done is the run going by roughly June 19th conditional on CoreWeave working, and the matching infra-readiness ticket #6052 to make sure the 6/15 H100 cluster works and can carry the June run. Note the scale gap: the epic body still names a 16B-A2B target, while the registered run in #6044 is 120B-A2B, so the bar for "multinode on CoreWeave" has effectively been raised by the run plan it now feeds.

On the kernel side, David Hall reworked the DeepEP MoE backend into a clean, reviewable form. The earlier draft #5957 carried research logs, debug notes, and benchmark scratch files and was closed in favor of #5982, which adds only the production pieces: an implementation="deepep" selector in the Grug MoE backend, the intranode DeepEP path under lib/levanter/src/levanter/grug/_moe/ep_deepep.py, a JAX FFI package for DeepEP dispatch/combine, and a python -m levanter.kernels.deepep.preflight check with install docs. Both PRs are still drafts — #5982 tripped the review bot's draft stop condition — so neither has merged, and DeepEP is intranode rather than the cross-host transport the multinode goal ultimately needs.

The clearest risk signal came from David Hall's GPU MoE benchmarking on B200s, where MoE routing perf was "not great" at scale. At d1024 Grug beat Megatron (373k tok/s vs 166k), but at d5120/L8 the picture inverted hard: Megatron TE+DeepEP ran 91.6k tok/s while Grug's ragged path managed only 14.2k and its ring path 24.4k, a gap a missing flash comparison does not explain. Since the June run is a wide model, closing this large-d MoE throughput gap is now on the critical path to the cluster being usable, not just a perf nicety.

Two adjacent issues were also filed this week. Romain Yon opened #6041 and #6042 for GrugMoE inference support in vLLM on TPU and GPU respectively, with the GPU side not needed before #6044 lands. Separately, Ahmed Ahmed filed the Iris bug #6087, where a slow TPU create-status response causes Iris to delete a live, already-registered slice — orchestration robustness that matters most for paired train/rollout jobs but bears on any large multi-host launch.

A large batch of post-training data work landed outside the milestone epics, all from Tai Vu: SFT dataset additions including SYNTHETIC-2 verified SFT #5944, WebInstruct-verified #5955, OpenR1 Math 220K #5948 and filtered Open R1 code controls #5950, plus a run of Nemotron SFT lanes covering math, science, safety, finance, proofs and agentic tool-use (#5968, #5969, #5970, #5971, #5972, #5973, #5978, #5979). On the RL side, Tai Vu added GRPO and Dr.GRPO losses #5949, a k1 KL loss mode #5943, and an on-policy distillation MVP #5939, and refactored rollout decoding across vLLM and Levanter #5035. Ahmed Ahmed opened LoRA-DPO support for Levanter #4637.

The Zephyr-to-Arrow/Polars migration continued under Will Moss, with scatter internals moved to Polars and Parquet #5963 and a burndown epic #5992 tracking remaining work on RecordBatch exposure, stats reporting, and speculative execution (#5993, #5994, #5995, #5998). On tooling and CI, Russell Power set up grouped weekly Dependabot config for uv and npm #6027 (driving a wave of dependency bumps), drove code quality from the advanced CodeQL workflow #6035, added an ml-slop-test lint detector for low-value agent-written tests #6022, lifted function-local imports to module top to break import cycles #6007, and stood up PyPI trusted publishing for the marin-* libs #5870 — the latter prompted by marin-finelog being missing from PyPI and breaking installs #5867. The recurring nightshift agent landed several multi-cleanup passes and raised the slow-test threshold from 8s to 60s #5980.

Iris control-plane hardening produced a cluster of follow-up issues from Russell Power and Rafal Wojdyla: a wedged BUILDING task that never times out and leaks capacity #6077, making reconcile batch apply order-independent #6081, push-and-forget task-status updates lacking ACK or retry #5921, autoscaler scaledown not communicated to the controller #3042, and minimum-lease handling for batch-priority tasks under preemption (#5942, #5975). Rafal Wojdyla also designed an Artifact.from_id + ArtifactRegistry for execution #6016. Finally, we filed a set of planning issues on post-training strategy and eval direction (#6047, #6051, #6053, #6054) plus an RFC on where design docs should live #5962.

Two contributors from outside the regular team touched the repo this week. @suraj-ranganath opened #3037, a 16-run 50k-step GPT2-small-equivalent sweep reproducing the NAMO weight-decay ablation in Marin, finding the reported endpoint reversed: NAMO/NAMO-D with AdamW-style matrix decay finished ahead of their adaptive-decay counterparts. @Rahul007007 filed #6075 to align the nightshift agent prompts with the author-pr skill, dropping the haiku ritual in favor of plain-text commit-style PR bodies and fixing a pre-existing KeyError in nightshift_doc_drift.py.

The week's Discord activity drew external researchers directly into the active optimizer and MoE work. New arrival Tim Tsz-Kit Lau shared his symmetry-compatible optimizer paper and offered to see how its per-matrix update stack performs on Marin, landing as the optimizer sweep under #5358 was ruling out MuonEq #6066 and AMUSE #6064 against the MuonH baseline. Leena Vankadara (Gatsby, UCL) shared her group's Maximally Scale-Stable Parameterization paper on scaling expert count and active experts, offering to help map its rules onto Marin's setup — directly relevant to the recipe's 4-active-of-256 choice carried into the June run #6044. On the eval side, Percy Liang connected a prospective contributor interested in adding evals, who then asked in #infra for the happy path to serve an HTTP endpoint over a multi-host JAX mesh without Ray — exactly the eval-serving shape #5368 is converging on.

Eighteen members introduced themselves, a mix of incoming PhD students, infrastructure and systems engineers, and researchers leaving frontier labs for open work — several arriving via Percy Liang's ICLR 2026 talk and citing Marin's scaling-law approach. Their stated interests cluster tightly around this milestone's open threads: optimization and pretraining data (Minhak, a Stanford PhD starting this fall), an architecture experiment Percy suggested running on Marin (shahir, also Stanford), and posttraining alignment, evals, and environment scaling (allie, recently of xAI's RL team; Vincent, formerly Anthropic pretraining/RL). By background, Tim Tsz-Kit Lau (DRW, ex-Penn) intersects directly with the optimizer-stability work driving the MuonH recipe, and Leena Vankadara brings MoE scaling theory bearing on the June expert-count decisions. Beyond the self-introductions, several recognizable researchers joined silently through the server-join stream — Martin Jaggi (EPFL), whose distributed-optimization work intersects the optimizer and MoE-throughput threads; Fares Obeid, known for Muon-family optimizer work tying into the same sweep; and Kaifeng Lyu (vfleaking), whose training-dynamics and scaling theory bears on the scaling-law direction.

The research the community circulated was dominated by scaling-law, optimizer, and MoE-parameterization work — from Poolside's Laguna report, which independently uses the same LR-transfer formula, to MoE scaling-rule and symmetry-compatible-optimizer papers offered straight into the recipe discussion.