Marin: Week of June 8th summary

The scaling-recipe critical path was quiet this week: no PRs merged into #5358, no new comments on the epic, and its open recipe sub-issues — the MuonH LR scaling law #5951, the compute-optimal isoFLOP sweeps at 1e18/3e18/1e19/3e19 #6074, per-expert MuonH LR #5799, and SinkSOAPH #6082 — all sat untouched since early June. The recipe is converged enough that the action has shifted to the sibling run epic #6044 (10T tokens on a 67B-A2B MoE, targeting roughly June 19), with this epic's residual housekeeping being a managed TL;DR @dlwh-golem added to the historical AdamH-vs-Adam gate #4042, which records that the optimizer path has settled on MuonH with plain Adam reserved for embeddings. Notably, no preregistration was filed — the #5359 pre-reg the epic's definition of done defers to remains outstanding.

The one substantive recipe-scoped decision happened in Discord rather than on the tracker. Larry Dial opened a thread asking whether to fold partial key offset into the June 67B-A2B run, weighing a measured ~10% compute-efficiency gain from 2e17–3e19 FLOPs (on top of partial RoPE) and zero added parameters against the fact that the technique is unproven at frontier scale and could act as a confounder for the RL stack the model is meant to validate (flagged in moe). Benjamin Feuer, who will be post-training the model, argued for maturing the midtraining→post-training seam first and building a complete e2e pretraining → midtraining → long-context → SFT/RL/eval pipeline before adding experimental features to hero runs; Larry Dial agreed, effectively keeping the June recipe conservative and leaving partial key offset out for now.

On the optimizer line that feeds the recipe, the additional PR #6301 (a PRISM-Berkeley update from @s-h-yang) is an exploratory optimizer change rather than a June-recipe lock, and it surfaced a regression: Kaiyue Wen launched 130m/300m/520m PRISM speedruns and bisected NaN divergences (~step 1240 regardless of LR) to the new use_spectral_norm rescale, whose power-iteration spectral-norm estimate under-shoots when top singular values cluster and over-amplifies the matrix past the Newton–Schulz convergence radius. With the rescale disabled, the 130m run completed cleanly at c4_en_bpb 1.1719, within 0.0016 of baseline — so the optimizer side stayed in experimentation, not promotion, this week.

The headline this week was getting the new CoreWeave US-EAST-02A H100 cluster (cw-us-east-02a) stood up for Iris and validated end-to-end ahead of the 6/15 June run, tracked in #6292. Russell Power's bring-up PR #6293 configured the cluster as a Pod-per-task KubernetesProvider with Kueue gang admission and Topology-Aware Scheduling over the InfiniBand fabric, both pools pinned always-warm at the full region quota of 4x Genoa CPU and 32x H100 (256 GPUs), with Cloudflare R2 (marin-na) holding state, data, and checkpoints. To shard a single model across all 256 GPUs rather than capping at the ~9.5B that fits on one node, it added explicit expert_axis_size/replica_axis_size knobs to build the compact Grug mesh — FSDP over the cross-node data axis plus expert parallelism over the intra-node NVLink devices — and fixed a multi-host shared-expert sharding bug that had crashed any model spanning more than one node at compile time. Live validation on all 256 H100 ran a canary at replicas=32 (20/20 steps, 1.39M tok/s), a 4-node FSDP+expert-parallel smoke, and a real ~90B-total / 5.3B-active sparse MoE (d3072, 48 layers, 128 experts, top-4) for 50/50 steps with loss decreasing and zero OOM or sharding errors.

A scheduling pathology surfaced once gangs started landing on the live cluster: CoreWeave's node-health-check verification pods (8x GPU each, priority -1) are designed to be preempted by the kube-scheduler, but Iris gangs go through Kueue, which counts those external pods as fixed node usage and never preempts non-Kueue workloads. The result on 6/10 was gangs queuing 20–45 minutes behind every NHC sweep with topology "infiniband" doesn't allow to fit any of 32 pod(s). #6310 resolved it by having Iris perform, in the layer it owns, the preemption the kube-scheduler would have done — evicting the preemptible blocker pods whenever it has gang work for Kueue. #6350 added a short marin-gpu quickstart onboarding doc for the cluster alongside the existing operator runbook.

The other major thread was getting the canaries green on H100. David Hall's #6237 closed #6226 by adding the FA4 CuTe and THD runtime dependencies to the CoreWeave GPU task images (previously the FA4 path failed with No module named 'cutlass' and only the reference attention backend validated), wiring the canary workflow to dispatch gpu_fa4_cute or gpu_fa4_thd and letting THD use the upstream SM90 kernels needed on H100. Validation also flushed out and fixed a string of canary bugs: #6252 (the GPU train data-loader referencing an expert mesh axis dropped when expert_axis_size == 1), #6253 (a multislice checkpoint write rejected by TensorStore on a chunk-shape mismatch from a stale resume point), #6298 (the TPU canary inheriting its CPU driver's europe-west4 region, which has no v5p/v4), and #6307 (the MoE train_step OOMing the v4-8 fallback, which has ~1/3 the per-chip HBM of v5p). On the housekeeping side, #3635 switched the GPU canary from SlimPajama to Nemotron CC to match the TPU path and production runs, and #3022 on tuning the CoreWeave GPU ferry was closed out.

Two issues remain open going into the run. #6363 is a TPU-side VMEM OOM where the MoE canary trains fine to step 240 but crashes at the first eval because eval runs the gmm kernel in float32 (it skips the compute-dtype cast that the train step applies), overflowing VMEM by 452K. #6364 is a false-negative in the H100 regression gate: a healthy 100-step Grug MoE canary passed every metric threshold, but W&B run.finish() hung on upload for ~73 minutes and the background worker swallowed the resulting HandleAbandonedError, so tracker_metrics.jsonl was never written and the metrics gate aborted on an otherwise-passing run.

The headline result this week came from #6277, where Larry Dial ran a controlled 8k-seqlen variant of the d=1280 EP=1 compute-optimal baseline and found gradients, train loss, and Paloma macro loss essentially identical to the 4k run. A per-tier analysis showed only ~8.4% of nemotron_mix tokens sit at positions beyond 4k within their own document (the 24.7% hq_synth tier caps at ~1k tokens), so pretraining at 8k on this mix is effectively a no-op for long-context capability. A confusing 10x bisection across Paloma subcategories (programming-languages vs dolma-v1_5) turned out to be a measurement artifact: the eval slices are ordered non-IID sequences, and a reduced eval_batch_size without a matching max_eval_batches changed how many tokens were scored. The controlled takeaways: match eval seqlens across ablations, and a cheap eval-time win is to express the long windows as sliding-window attention over a long seqlen to avoid early-document truncation. Larry walked through the same finding in the long-context channel.

Larry also opened a thread on whether to pretrain at a RoPE base above 10k, reasoning through OOD rotation angles versus retrieval-dim alignment and noting NoPE on the long layers cost almost nothing (3.31 vs 3.30 at 2e18); he concluded he sees no theory problem with a larger theta but also no clear benefit. Amanda Bertsch argued that regardless of pretraining theta the team should increase theta again at extension time (à la Llama 3.1's 500k -> 8M jump), since YaRN or partial RoPE corrections largely absorb the difference. The mid-flight context-extension experiments (#6170, #6171, #6194) remain closed with their stability findings, and the broader long_context_32k sensitivity sweep #6232 is still open. Among the epic's curation issues, #4276 (Levanter long-context training optimization) and #2025 (curating long-context evals) are both now closed.

The week's center of gravity was MFU on CoreWeave's H100 cluster, logged experiment-by-experiment in #6304. The 90B-total/5.3B-active Grug MoE came up from the #6292 bringup at 218K tok/s — about 2.8% active-FLOPs MFU on 256 H100. A baseline profile by Russell Power confirmed the diagnosis from first principles: the step is ~73% NCCL, dominated by ~583 small per-tensor FSDP all-gathers per GPU per step in the latency-bound LL protocol — the model is structurally communication-bound, not compute-bound. The single lever that paid was scaling tokens/device: batch 256→512 gave +85% (5.37% MFU), and batch 1024 (8,192 tok/GPU) reached 709K tok/s / 9.32% MFU, since the per-step gather cost is roughly fixed and tokens ride along nearly free until memory pushes back. save_moe remat was neutral on top of that (712K / 9.37%, the best-known config). Nearly everything else was a negative result, each with a recorded reason: XLA collective-combining/latency-hiding flags (−7%), PGLE, batch ≥1536 and seq 4096 (all OOM, attention activations scale with sliding_window=seq_len per sequence), pure-bf16-params (a deterministic 32-rank gang hang), and microbatching — which fits a larger batch but is MFU-negative because FSDP re-gathers the sharded params on every microbatch, so N microbatches cost N× the gather traffic.

A genuinely sharp finding: the OOM ceiling was traced not to the MoE dispatch but to the token-embedding backward, where GSPMD must reshard the [B,S,H] cotangent between mismatched tile counts (256≠32) and so replicates the full tensor — 3 GiB/device at batch 1024 rising to 18 GiB at 1536. Russell Power wrote and CPU-validated a hand-routed reshard that eliminated the involuntary full-remat (GSPMD warnings 2→0, bit-identical loss), but the explicit replicate-indices→gather→all-to-all path was a throughput regression in practice (−11% at batch 512, and it OOM'd earlier than the unfixed path at 1024), so it was reverted — leaving the root-cause analysis as documentation. The structural conclusion: at the throughput-optimal batch the 90B model fills HBM, the binding constraint is memory rather than a missing flag, and the comm tax is latency-bound, so byte-reduction levers offer only modest upside. The remaining deeper levers — bf16 FSDP gathers with f32 master weights, activation offload, hybrid/expert-only sharding, FA4 — were spun out to #6321 as deliberately deferred multi-session changes. The launcher and checkpointer infrastructure that made the sweep fast to iterate landed in #6312 (runtime-tuning env forwarding through Iris, node-local disposable checkpoints, the save_moe remat knob), with profiling instrumentation in #6213.

Multi-host training on H100s was unblocked at the source. Russell Power filed #6296: the shared-expert residual add crashed at trace time with a ShardingTypeError whenever a model spanned more than one node, because the routed and shared MLP paths canonicalized their expert-axis sharding in opposite order — invisible on a single host, which is why TPU and single-node canaries had stayed green. The fix (reshard DenseMLP's output after the reshape, a no-op on a trivial mesh) shipped in #6297 and was validated end-to-end on 64 H100, completing 20 steps with zero sharding errors. Adjacent plumbing landed too: #6259 derives the MoE data-loader pspec from the actual mesh (the GPU canary drops the expert axis at size 1, which had been deterministically crashing loader build), #6216 routes Grug dispatch through the durable GCS compile cache and fixes two MoE sharding mismatches the cache probe exposed, and #6314 plus #6316 share the Splash mask lowering and teach flash-attention to treat a None mask as full bidirectional attention.

The actual June production shape — the d=2560, 26-layer, 256-expert top-4 "May Recipe" MoE from the epic's recipe comment — is tracked separately in #6367 and has not yet produced a profile-bearing baseline. David Hall cycled through GM2560-MFU-002 through 006 over the night of the 14th: switching from reference attention to gpu_fa4_cute FA4, adding an opt-in compute_with_master mode (bf16 live params with a separate sharded fp32 master tree), and forcing streaming XLA cross-entropy. Runs 004/005 died before step 0 with ~60 GiB allocator requests; 006 cleared that by raising the XLA memory pool to 95% but then hit XLA collective-permute/clique rendezvous failures before any training metric. Abstract sizing shows params and optimizer state already sharded, so the suspicion has shifted toward a distributed collective/resharding path in the full executable rather than replicated state. On the recipe side, two decisions were settled in Discord. Larry Dial proposed adding partial key offset for its ~10% compute-efficiency gain, but on Benjamin Feuer's argument that the midtraining→post-training seam should mature before experimental features enter a hero run — this model's main value is de-risking the MoE RL stack — it was deferred to keep the stack simple. And Larry Dial confirmed z-loss was dropped from the new MoE recipe: it gave ~1% on evals but slightly worse train loss at small scale, and with stability already demonstrated on the 1e23 run the team chose to be less conservative.

Eval infrastructure moved on several fronts. #5959 (by David Hall) merged, adding a Levanter training callback that runs labeled LM-eval datasets and logs per-label losses, wiring train_lm configs to build trace-chat labeled eval caches for periodic pretraining probes. Romain Yon opened #6360 to add a brokered Evalchemy MATH500 hello-world runner that drives Evalchemy as an HTTP client against Marin's brokered vLLM proxy rather than spinning up a second in-process engine. The nightshift scout agent filed #6336, flagging duplicated, behaviorally-inconsistent model-loading between save_logprobs.py and trace_labeled_eval.py (divergent latest-checkpoint resolution and axis_mapping handling) and proposing a single load_eval_model helper. #4389 (by Kevin Li), the soft-proxy-for-agentic-benchmarks idea, was closed with the note that Rohith is pursuing that work elsewhere.

On Discord, Benjamin Feuer announced he will maintain a marin-community/evalchemy fork with lm-eval-harness rebased to upstream main, which together with marin-community/harbor should cover post-training eval needs; Tony Lee asked that the fork be based off the existing teetone/evalchemy to retain Marin-specific changes, and Romain Yon signaled intent to plug Evalchemy and Harbor into the inference service over the coming weeks. David Hall posted a "Marin target metrics" summary in response to f3mi asking what the model should be good at. No leaderboard-UX work surfaced directly this week; the activity is upstream eval plumbing.

The GPU side of GrugMoE vLLM inference was quiet. #6042 carries no sub-issues, no mapped pull requests, and no comments this week, and nothing in Discord touched GPU-specific GrugMoE inference.

The one PR attached to this epic, #6102 from David Hall, adds a Qwen3 inference parity benchmark comparing Levanter against vLLM — with throughput summaries, HLO and profile artifacts, and reference-logit diagnostics — but it is a TPU benchmark, not a GPU one, and remains in draft. No GrugMoE checkpoint loading, correctness testing, or GPU performance work landed against this epic's definition of done.

The TPU GrugMoE-inference epic itself stayed empty this week — no sub-issues, no mapped pull requests, and no comments on #6041 — but the groundwork it depends on moved. Romain Yon opened #6288 to wire Marin's TPU-vLLM dependency path to explicit Marin-owned fork commits instead of the opaque vllm-tpu PyPI package, keeping marin-core[vllm] as a self-contained TPU-vLLM extra while holding the generic tpu and gpu accelerator stacks isolated. The change rides two companion fork PRs (tpu-inference#5 and vllm#6); #6288 stays in draft until both forks merge.

Validation so far is dependency wiring plus a bounded TPU-vLLM smoke: a Qwen3-0.6B generation run on an Iris v6e-4 in europe-west4 reached JOB_STATE_SUCCEEDED, and the resolver checks confirmed the GPU extra carries no TPU-vLLM packages while the tpu+vllm combination correctly fails as incompatible. The PR is explicit that it does not yet prove full runtime correctness for Evalchemy, Harbor, or lm-eval — and it exercises Qwen3, not a GrugMoE checkpoint, so the epic's definition of done remains ahead of this work.

On Discord, Romain Yon gave reviewers some context in code-review: he rebased both forks onto the latest upstream release heads, re-overlaid only the still-useful pre-rebase commits, and smoke-tested the result through #6288.

The diagnostic side of local-metric work advanced via #6207 (merged, David Hall), which adds PPL circuit-coverage v2 generators: 27 deterministic target-only validation slices plus a matching ~1B-token plain-text corpus, covering string/byte transforms, indexing, arithmetic, serialization, state machines, and format-following. It registers the slices in the perplexity-gap dataset registry, adds a Marin 32B vs. Qwen3 32B scoring/gap experiment, and adds a materialization/tokenization entry point so the same tasks can feed both diagnostics and training data — an automated review found no issues. The motivation comes through in the new perplexity-gaps channel, where David Hall noted that much of Qwen's advantage looks like “better code sense” and strong base-conversion / arithmetic / string-manipulation understanding. Note this PR is filed as part of #6070 (code-interpretation PPL eval), so it is adjacent diagnostic tooling rather than a direct climb-metric deliverable.

The epic's named time-boxed components — climb deduplication and decontamination precision/recall on test/synthetic/sampled sets, and climb quality/domain blind LLM-panel intruder tests — saw no direct PR or issue-comment activity this week, and the e2e datakit testbed integration the issue calls for was not touched here. Progress this week is concentrated on the perplexity-gap diagnostics rather than the climb-metric precision/recall loop itself.

Two dataset registrations moved the June CoreWeave pretraining corpus forward. #6210 (merged, ravwojdyla-agent) wires HuggingFaceFW/finetranslations into datakit as a single source, replacing a stale TODO in sources.py and porting the shape of the closed #4328 onto the datakit download → transform → normalize chain. The fold pairs each row's original-language and English-MT text into one text column, with a deterministic xxh3(id) coin flip giving a verified ~50/50 split of original→English vs. English→original ordering. Rafal Wojdyla posted it for review in code-review. #6366 (open, Tai Vu) registers AllenAI's Dolma3 Mix 6T as an opt-in pretraining dataset: it pins the HF revision 689a3ea, downloads via download_hf_step(..., append_sha_to_path=True), and exposes a Llama 3 tokenizer step — deliberately keeping the full 4.4 TB / 6T corpus available for scaling-law work without launching materialization or touching default mixtures.

In parallel, the tokenizer channel worked the practical edges of preparing this data. Pranshu Chaturvedi reported training four 20B tokenizers (gpt-oss/llama, with and without numeric-specific strategies like R2L and place-alignment) without OOM after the 100B jobs kept OOM-ing, settling on a 50B-token corpus that David Hall judged plenty “as long as data was shuffled well” — prompting a check on whether the old shuffling footgun was patched for tokenization too.

The epic's own definition of done — tokenized data staged on the CoreWeave cluster, confirmed regions, non-blocking data-loading performance, and a blessed R2 store — saw no direct comment or PR activity this week; the work that landed is upstream dataset registration rather than cluster-side staging.

Direct activity on the data-mixture epic was thin this week. The one concrete contribution is #6362 (by Calvin Xu), opened in draft, which adds a DCLM Core v2 task inventory and a centered-accuracy scoring helper; the launchable export holds back Winograd, Jeopardy, and BigBench generation tasks until their custom data and filtered-generation metrics are safe, while scoring handles filtered BigBench metrics and rejects unfiltered exact-match values. This feeds the metric side of mixture optimization tracked in #5362 (select a high-SNR metric), but the epic's other open items — the production swarm spec #5364 and its launch #5365 — saw no movement, and the epic drew no comments in-window.

Discord touched the topic only tangentially. Ahmed Ahmed clarified in scaling-laws that his ladder trains on a single DCLM source rather than the mixture pretrain path, and Larry Dial argued that with post-training now planned, decaying LR to zero at the end of pretraining is likely worse than it would be for a standalone run. No production-swarm launch or finalized June mixture was reported this week.

The agentic-SFT and SWE-ZERO line reached its public milestone and then largely wound down. Kevin Li released the full SWE-ZERO 12M-trajectory dataset — execution-free agentic rollouts generated from the SWE-rebench V2-PRs corpus (#4719, #4435) — and in the same sweep closed the surrounding reproduction experiments, mostly with honest negative results. Post-training Marin-32B on the TerminalCorpus #4760 was rejected: after an EOS-metadata fix, an apples-to-apples Terminal-Bench 2.0 eval at a comparable 15%-corpus budget put Marin-32B at 2/87 (~2.3%) against Qwen3-32B's 18/86 (~20.9%). The NemotronTerminal-32B reproduction #4307 topped out at 17.2% on TB2 versus the 27.4% target and was closed unmet, and the execution-based ConTree evaluation #4683 was closed in favor of Daytona-based evals. The TerminalWorld world-model probe #5866 found Pass@1 and world-model fidelity pulling in opposite directions across the assistant-only / user-only / full-transcript loss-mask decomposition.

On the RL infrastructure side, Benjamin Feuer stood up router replay (R3) and truncated importance sampling (TIS) on Qwen3-Next-80B-A3B via MarinSkyRL #6335, reporting a healthy first step on 24 nodes (96 GH200s) on Jupiter at avg_pass@8 0.609. The issue's closing writeup is an unusually detailed postmortem of the 80B bring-up: a weight-sync OOM from materializing the whole 512-expert MoE on one GPU, a silent no-op-learning bug where a transformers BatchEncoding leaked dict keys instead of token ids, uneven EP×FSDP expert sharding crashing the Adam step, and a vLLM Ray compiled-DAG hang that forced a pivot to the mp backend. A 30B-A3B (Qwen3-Coder) run on the validated R3/TIS/grouped-GEMM stack served as proof-of-life, clearing both weight-sync and first-train-step gates.

The Delphi RL scaling-laws study #6279 became the week's central post-training research thread, opened by Ahmed Ahmed to study how math-heavy midtraining, light instruction-following SFT, and RL interact on the K=0.20 Delphi ladder. Before propagating an SFT recipe across all 27 checkpoints, Benjamin Feuer ran a cold-start SFT calibration grid (instruction dataset × learning rate) on the 9e19 p33m67 anchor to install the boxed-answer chat-format contract pre-RL, with a draft launcher for the 100-step MATH-500 probe in #6325. The accompanying "10+ ablations in, still can't reproduce" thread dissected why the hero RL run beats SFT — it alone grows the diagnose→edit→rerun→confirm-green loop — and @Sankalp Jajee suggested potential-based reward shaping with Φ = fraction of target tests passing rather than a hackable PRM or LLM-judge. Separately, the Leonardo non-agentic-reasoning ablations #6218 ran their cross-grids to completion, flipping the earlier read to a clean verdict: on GSM8K with a ~1.5–1.7B student, pure RL beats on-policy distillation on both accuracy and compute-to-accuracy.

A wave of synthetic and math/code data sourcing also landed from new contributor Tai Vu, who opened the GRPO/Dr.GRPO/DAPO loss stack for Marin RL #5949 plus a run of Datakit source pipelines: the full OpenMathInstruct-2 midtraining split #6254, an OpenCodeReasoning-2 SFT pipeline with source-prompt hydration #6368, an OpenResearcher browser-rollout pipeline #6272, and a cluster of math continued-pretraining sources — Open-Web-Math-Pro #6369, AutoMathText-V2 #6370, Olympiad Books #6371, PlanetMath #6372, and StackMathQA #6373. Benjamin Feuer picked up review, asking the contributor to coordinate on Discord and align with the post-training effort. On the parity front, #6341 consolidated a framework-neutral RL feature checklist for reaching parity with the OpenThoughts/Marin RL stack.

Most of the week's non-epic work landed in Iris, the cluster controller, which continued a structural consolidation pass. #6311 collapsed the per-backend control surface behind a single TaskBackend protocol and moved worker liveness onto the reconcile pass, deleting the controller ping and dispatch loops; #6295 folded scheduling demand onto one Scheduler instance and one code path so the autoscaler no longer dry-runs assignment against a second snapshot; and #6344 merged the controller fast loops into a single phased control tick. Supporting changes routed all DB fan-out through one ControlSnapshot in #6294, made vm_create non-blocking #6352 and initialize_jax idempotent #6320, bounded a 1-hour exec_in_container stall #6291, and fixed Kueue gang teardown leaking TAS reservations #6303, #6305. Several deprecated shims and RPC stubs were removed #6306, #6315.

Levanter picked up structured prefix-LM mask support in #6330, lowering static prefix lengths to Splash mask objects and building compact per-example metadata for packed variable-length batches, alongside fixes to config-artifact dumping under stringized annotations #6323, partial-cache rebuild fallthrough #6319, and JAX mesh propagation into the prefetch thread #6317. On the data side, Zephyr moved its Scatter-phase internals to Polars and Parquet #5963, and finelog gained a trigram substring index with a prefix-range optimizer rule for sub-linear string queries #6203.

Code health saw a typing and lint push: #6248 burned down all 159 baselined pyrefly errors so the baseline is now empty and must stay green, and #6249 enabled Ruff PLC0415 to disallow function-local imports repo-wide. The code-health report itself was fixed to surface linter-fed metrics it had been hiding #6290. A nightly agentic canary #6357 now checks that a naive agent can still stand up a launchable training run, and the weekly storage report was automated with week-over-week diffs #6144. Routine dependency bumps moved torch to 2.12.0 #6346 and unified the pyarrow floor across subprojects #6245.

External contributors, almost all from the Stanford CRFM orbit, drove a busy week. Beyond the data and RL work the epics already cover, Michael Ryan merged #6144 to automate the weekly storage report with week-over-week diffs, @muherng landed an Iris user-budget addition in #6331, and Nikil Ravi opened the draft Delphi RL MATH-500 scaling-probe launcher #6325 feeding the #6279 study. Will Moss continued the Zephyr refactor — moving scatter internals to Polars/Parquet in #5963 and adding stage/worker finelog stats in #6126. A first-time external contributor, @esaran1, opened #6269 to clarify Speedrun starter discovery for newcomers, while Rohith Kuditipudi filed #6334 on relative output paths in the ExecutorStep→StepSpec bridge.

On Discord, the sharpest exchange was an optimizers-channel deep-dive in which Ahmed Ahmed asked whether Shampoo and Muon are equivalent and Kaiyue Wen laid out the algebra — Muon as a two-sided whitening versus one-sided Shampoo — grounding Wen's open optimizer issues on KL Soap H #5728, the AMUSE+MuonH kitchen-sink arch #6064, and the Manifold Power Iteration router #6329. In code-review, Rafal Wojdyla posted the finetranslations datakit source that merged as #6210 for review, and over in the math-data push Tai Vu coordinated his Datakit pipeline run (#6254, #6368–#6373) with the post-training effort at penfever's request.

Roughly a dozen newcomers introduced themselves: research engineers and PhD/MS students from UC Berkeley, UT Austin, GaTech, Toronto, UChicago, Paris, and Stanford spanning training-at-scale, interpretability, agents, and data curation, alongside several founders building agent-runtime and eval infrastructure. Two arrivals intersect directly with active work: Malte Ostendorff, a research engineer at the Common Crawl Foundation, joined wanting to contribute data — squarely the datakit corpus-registration line in #6036 — and Alex Dimakis, a UC Berkeley professor and Bespoke Labs co-founder, arrived specifically to work on post-training and RL environments, the center of gravity in #3192 and the Delphi RL study #6279. Atula Tejaswi (UT Austin) brings context on OpenThoughts-Agent, intersecting the SWE-ZERO and Evalchemy/Harbor threads. Two further joiners arrived silently through the welcome-room without a self-intro: Reinhard Heckel (TU Munich/Rice; deep-learning theory and scaling), whose background intersects the scaling-laws and optimizer discussion, and Yu Su (Ohio State; LLM-agent and Mind2Web work), whose background intersects the agentic-RL line in #3192.

The week's shared reading clustered on optimizer and scaling-law theory — Shampoo/Muon threads, VPNLS and Chinchilla-revisited references, and an RL primer — interleaved with ongoing discourse on automating AI research.