Marin: Week of June 1st summary

The recipe got consolidated onto main this week: #6153 ports the full May Recipe (half-RoPE on every layer, PKO, the MuonH kitchen-sink) onto the current MoE template, and Larry Dial told the team that main's Grug MoE and its README are now up to date. With the isoFLOP sweep and compute-optimal table from last week in hand, attention turned to what a forecast for the June run actually requires. In #moe Larry sketched the plan to train d512→d768→d1024→d1280→d1560 at the big run's ~100x-overtrained ratio on the new datamix once the data-mixing swarm hands off a mix, generating a preregistered forecast and catching any unexpected interactions. The long debate that followed (with Will Held and @Kaiyue-Wen) landed on a sharp point: as you overtrain, the asymptote dominates earlier, so a fixed-size loss improvement that's worth its MFU cost at compute-optimal can stop being worth it — and the smallest isoFLOP curve is a cheap way to check how each in-flight change shifts the parabola.

The agent-driven optimizer search under agent.md kept probing for anything that beats the locked MuonH baseline (paloma c4 3.5931 at d512/2.19e17). SinkSOAPH #6082 was the week's deepest dive: on its own the Gram-Sinkhorn direction plateaued ~+0.22 vs MuonH, but inserting Newton–Schulz orthogonalization after it dropped that to +0.067, and swapping Sinkhorn balancing for a plain SOAP-Adam second moment in the reparam eigenbasis closed the gap to just +0.008 — close, but still not a win. The first thing to actually edge below MuonH came from EMA-Nesterov (arXiv:2605.25395), tracked in #6167: a constant negative lookahead (β=−0.2) reached c4 3.5895. The diagnostic is the interesting part — the lookahead-loss landscape is monotone negative, the exact reverse of the paper's prediction, with the basin vertex near β≈−90 and the per-step advantage collapsing to ~0 by end of training, which Kaiyue Wen flagged as worth investigating. HybridMuon (opt_a/opt_f) was ported to JAX in #6118 with the Qwen3-scaling LR sweep written up in #6119 (opt_a beats opt_f but neither beats Muon), and Varun Srivastava closed the Dion port #5625 as not useful for Grug's untensor-sharded configs. On the LR law #5951, Larry Dial corrected the hidden-dim dependence plot to normalize for batch size, which had been overstating the model-size term.

On the throughput side, the B200 Grug-vs-Megatron gap closed substantially. The reproducible BF16 baseline #5815 was marked complete, and the d5120/L8 follow-up #6139 hit its target: David Hall measured Grug ring EP + gpu_fa4_thd + an outer global clip at 82.6k tok/s post-profile against Megatron's 91.6k — inside the 10% band, up from the 14k–24k of two weeks ago, helped by the MoE block resharding in #6069. The DeepEP MoE backend landed in #5982 with profiling scopes #6175, and an assigned-token DeepEP dispatch #6251 (closing #6215) avoids ring EP's per-rank global activation buffers, measuring 1.94 ms vs ring's 2.32 ms on 4×B200; a plain-XLA ragged_all_to_all variant was much slower, confirming CUDA-side transport is the scalable path. #6140 added Qwen3 MoE perplexity support so HF Qwen3-30B-A3B can run through the scoring path, and a pair of canary/profiler fixes landed in #6111 and #6124.

The headline merge this week was #5987, which replaced the transitions.py monolith with a layered reconcile package: a pure two-pass kernel over a closed-world snapshot, an ops command layer, and a single load/commit boundary. Along the way it fixed coscheduled gang-splitting, jobs left RUNNING after a worker failure, capacity leaked by zombie and stranded attempts, and preemption/reservation placement that ignored hard constraints. With reconcile now the only task-lifecycle path, Russell Power pushed a wide cleanup and capability-driven refactor: #6209 introduced a single TaskBackend contract so the controller dispatches by declared capabilities (placement and capacity ownership) rather than isinstance checks, laying the groundwork for the Slurm backend tracked in #6178. The finelog store also finished its migration to a native Rust wheel (#6135, #6204, shipping marin-finelog 0.2.9 and deleting ~15k lines of dead Python server and DuckDB store), with the controller cut over to a PyO3 embedded server.

Preemption and provisioning correctness dominated the live-ops work. A cluster of fixes stopped Iris from destroying healthy hardware: #6090 submits the TPU create operation without blocking and lets worker health, not GCP's slow create-status, decide readiness, after Ahmed Ahmed and Rafal Wojdyla showed a create-status timeout was deleting live, registered slices and starving RL train/rollout pairs (#6087, #6098). #6122 gates worker registration on the node reporting READY, #6117 reaps READY slices whose backing allocation vanished, and #6198 throttles slice creation project-wide after dozens of scale groups scaling in one cycle blew past GCP's CreateNode quota. The reconcile scheduler rollout was bumpy in #infra: @rav restarted the controller to deploy Russell's over-preemption fix, and by week's end could report "you should not see thrashing anymore on the TPUs" — though the cluster remains capacity-constrained and the team is still awaiting a GCP response on internal provisioning errors.

On the H100/CoreWeave side, the readiness picture is mixed. #6058 merged atomic gang admission for the direct K8s provider via Kueue plain-pod-groups, closing the gap where a 64x8 job could bind 63 pods and strand the last one behind the autoscaler. But the actual H100 Grug MoE canary is not green: David Hall spent the week in a long debugging loop on #6237, where successive FA4 THD runs failed on fixed-shape segment metadata, a sharded cumsum, an explicit-mesh resharding assert, and finally CUTLASS shared-memory and M-tile limits in the SM90 backward kernel — each fix surfacing the next. Separately the canary itself is broken in several ways: a reproducible SIGSEGV at W&B profiler upload (#6108, still reproducing on main), a mesh-axis mismatch where the single-expert canary references a dropped expert axis #6252, and a TensorStore chunk-shape collision on the resumed multislice checkpoint #6253. The multislice Megascale path was at least restored #6142 after David Hall minimized a TPU page-fault down to a Marin-side mesh workaround.

The throughput target is the soft spot. The B200 baseline closed (#5815, #5894): at the fair-fit d5120/L8 shape, Megatron with TE plus DeepEP reaches 91,574 tok/s while Grug reaches 14,230 (ragged_all_to_all) and 24,350 (ring), and David Hall confirmed the residual gap is MoE transport, not FA4 attention, which already clears its block-time target. The May kernel epics #5356, #5357 and #6049 were all closed and rolled forward into the June milestone, with David Hall publishing an MoE-speedup blog post summarizing the effort. Closing that ~4-6x MoE-transport gap to roughly Megatron MFU remains the gating risk for actually training the June run on H100s by the 15th.

Quiet week. The fp8 training epic saw no direct activity — it remains downstream of the recipe and cluster work and has not been picked up. The nearest adjacent thread is the FP8-stability characterization folded into the GPU systems work under the cluster epic #6052, but no fp8 training experiment has been scoped or run.

This week turned the epic from a plan into a running experimental campaign, driven almost entirely by Larry Dial across its four sub-issues. The first wave studied training stability when extending context mid-run on the May Recipe MoE. In #6170 he resumed d512/d768/d1024/d1280 checkpoints and swept extension knobs (sliding-window size, RoPE theta, and YaRN attention rescale): standard YaRN on the long (no-window) layers was the most stable path from 4k to 16k on the nemotron mix, with an immediate loss drop, and most of the long-context gain was captured by adjusting only the global-attention layers. Spikier gradients appeared after lengthening sequences (fewer documents per batch), and at expert-parallel 2 the overflow fraction roughly doubled when sequence length 4x'd, flagging document-diversity and token-dropping as the risks to watch at 128k+ and higher EP. #6171 isolated the data-mix shift, finding QB routing very robust to swapping in the longmino mix (entropy restabilizes in roughly one step) but Paloma macro-loss degrading sharply on a pure longmino swap, with a 50/50 mix splitting the difference. #6194 then combined both stressors at once on a d1280 EP8 run extending to 32k with a half-mix, which came through clean overall (one expected grad spike).

On the eval side, #5923 landed, adding the long_context_32k perplexity bundle (raw PG19/GovReport/QuALITY plus target-only QASPER/NarrativeQA/BookSum) that #6232 immediately exercised. Larry characterized the corpus (noting QuALITY tops out near 8.7k tokens and PG19 is the only slice that meaningfully probes 32k), then ran controlled inference-time YaRN comparisons: training 15% of the run on the long mix at 16k buys a consistent ~0.02 loss improvement over never training long, while a clean 32k-seqlen test (4k sliding window vs YaRN to 32k) showed the extra context helping only modestly and unevenly across scales (a small boost at d768, little to none at d512/d1280). Discussion continued on Discord, where Will Held noted whole-repo NLL as a long-context driver and @penfever shared 131k-context MiniMax agentic traces and onboarding for help on the experiments. Three of four sub-issues are now closed; #6232 remains open as the go-forward baseline characterization.

The mix-selection handoff from the production swarm reached a decision point this week: Will Held announced on #data-mixing that vibe mixing is in, landing the hand-steered candidate #6063 as the leading path for the June run. After repeated W&B timeouts he stood up a lightweight dashboard at oa.williamheld.com to track the swarm with less noise, and shared a quality-filter validation showing that Rafal Wojdyla's higher-quality code quartiles are in fact better correlated with coding perplexity. Discussion drew on the Microsoft AI tech report's candid account of alternating global/local mixture optimization over a swarm, reinforcing the view that mixing is useful but not magic for pretraining.

On the ingestion-and-throughput side, Will Moss's Nemotron ferry optimization #6065 merged, keeping the full set of normalize output shards in flight and using extra threads to process them in parallel — faster across the board on task wall time — alongside a discussion with Rafal Wojdyla about resource hinting and stats reporting #5995 to let Zephyr size workers automatically. Tai Vu opened a batch of new datakit and midtraining source PRs — a Nemotron Code v3 materializer #6257, OpenCode instruction staging #6256, Nemotron Specialized v1.2 sources #6255, and the full OpenMathInstruct-2 corpus #6254 — all currently in draft, plus a Rafal Wojdyla finetranslations source #6210.

Deduplication and contamination scrutiny sharpened the quality bar: on #deduplication Rafal Wojdyla shared near-dup examples and noted dups are currently resolved by keeping a random copy, with Percy Liang agreeing that loose synthetic-data duplicates are fine to drop given a token surplus. Separately, Ahmed Ahmed reported heavy near-duplicate contamination of the Nemotron math val carve-out under #4547 and is rebuilding a clean doc-level val set. Upstream, the long-running data-sources epic #3100 was closed as substantially done — Isaac Hodes noting the corpus is now beyond 19T tokens, more than needed for the foreseeable future — confirming there is ample raw material to draw the 10T June mix from.

The June hero run got a definite shape this week. The milestone and this epic were retitled to a 67B-A2B MoE trained on 10T tokens — firming up a target that had drifted between 16B-A2B and 120B-A2B variants in prior weeks — with the stated definition of done unchanged: the run should be going by ~June 19th, conditional on the CoreWeave H100 cluster working, and should include midtraining data. The epic issue itself saw no direct activity; its lone sub-issue #6046 (registering a loss guess for the June run) remains open, and the run stays gated on three siblings that are all still open: the scaling recipe #5358, the 10T datamix #6045, and cluster readiness #6052.

The real motion was in the supporting infrastructure. The May recipe was ported onto the current MoE template in #6153 (half-RoPE, PKO, and the rest of the v16-vs-May deltas). On the kernels-and-transport side, the DeepEP backend that landed in draft last week gained profiling named scopes #6175 and assigned-token dispatch work #6251, a multislice mesh-sharding fix landed in #6166, an experimental FA4 attention backend in #6059, and a router summary-stats regression was fixed in #6124. Midtraining datamix components began staging in draft — full OpenMathInstruct-2 in #6254 and NVIDIA OpenCode instruction data in #6256.

The largest open risk is the cluster gate. The CoreWeave H100 Grug MoE canary segfaults reproducibly ~18 minutes in, around the W&B profiler upload #6108, and the daily GPU and multislice canaries also broke this week on a data-loader pspec mismatch #6252 and a TensorStore checkpoint chunk-shape mismatch #6253. @rav reported the TPU side is more stable after a controller restart but capacity-constrained and oversubscribed for v5ps. Separately, Will Held, @Kaiyue-Wen, and Larry held an extended discussion on the overtrained regime for the big run — whether the optimal architecture for a 100x-overtrained model (MLP router, more expressive attention) differs from the compute-optimal arch the ladder is tuned against, and whether to add a small overtraining ladder to the preregistered forecast.

The headline UX/leaderboard goal itself still has only a placeholder DoD ("eval experience is fine"), but two infrastructure pieces moved. #5887, the brokered vLLM eval-serving path that lets one HumanEval run fan out across Iris TPU workers behind a single OpenAI-compatible endpoint with lease-based preemption recovery, finally merged after Romain Yon addressed Russell Power's review (threadpool vs async, context-manager lifecycle, and a serving-plumbing rename). Separately, David Hall merged #6092, fixing lm-eval harness multihost broadcast placement so multi-host TPU workers stop feeding non-addressable JAX arrays into device_put.

Most of the week's eval energy went into the PPL circuit-coverage gap line from #6070, where the gap dashboard showed Marin 32B trailing Qwen3 32B on basic deterministic skills (arithmetic, string/token manipulation, exact index tracking). David Hall opened three contributor issues to expand ppl_circuit_coverage_v2: #6235 to collect task proposals, #6236 to add new task families, and #6238 to add template diversity; he also floated in #evals generating a few billion tokens of synthetic drill data, with Will Held backing aggressive dataset additions since mixing/tagging should sort out usefulness. On enumerating evals of interest #6050, Will Held noted AA II can't be set up directly since it's closed-source. The related raw perplexity-gap report issue #4934 was closed as complete.

The GPU side of GrugMoE-in-vLLM saw no direct work this week, which is by design: #6042 is explicitly gated behind the hero pre-training run #6044 landing, with Isaac Hodes noting it is "needed no earlier than" that point. The epic remains an empty shell — no sub-issues, PRs, or comments — and what inference momentum exists lived entirely on the TPU sibling #6041, where David Hall reported that throwing goal-mode at native Levanter TPU inference made it ~10x faster and within ~12% of vLLM-TPU, potentially obviating part of that path. The one issue nominally orbiting the GPU side, #6226, is in fact unrelated FA4 CuTe/THD attention-kernel work for CoreWeave training canaries (now passing for both backends), not GrugMoE serving. The related TPU decode-corruption findings #6136 and the port-to-flax playbook #6143 likewise sit on the TPU/native-serving track rather than this GPU epic.

The core of this epic landed this week as the #6106 GrugMoE-vLLM-TPU experiment, driven end to end by Romain Yon with Levanter as the correctness oracle. The first cut added an unfused PyTorch GrugMoeForCausalLM in the vLLM fork with exact QB routing semantics, plus a tpu-inference auto-router and a Marin parity harness comparing tiny seeded GrugMoE components and full hidden states against the Levanter/JAX reference. Local component and composed parity passed, and Iris validated the strict-parity harness on a v6e-4 slice in europe-west4 after working through SSH-key, TPU-runtime-extras, and TPU matmul-precision setup issues. The harness reported GrugMoeMLP matches Levanter moe_mlp and GrugMoeModel hidden states match Levanter Transformer.

The implementation was then pivoted off PyTorch entirely: the vLLM PyTorch model was removed and replaced by a native JAX/NNX tpu_inference.models.jax.grugmoe.GrugMoeForCausalLM registered through tpu-inference flax_nnx loading, exercising the sliding-window branches. From there Romain Yon built out the checkpoint boundary the DoD requires: a canonical config.json + sharded HF safetensors export through the standard Levanter/Marin path (to_hf_config, export_lm_to_hf, convert_checkpoint_to_hf_step), native tpu-inference load_weights for both single-file and model.safetensors.index.json sharded layouts, a full realistic checkpoint roundtrip from a seeded GrugTrainState, and deterministic greedy generation parity against the loaded native artifact. All of this stands on the grugmoe-vllm-tpu-support branches across Marin, vLLM, and tpu-inference rather than on main.

Surrounding this, David Hall pushed hard on the shared Levanter TPU inference stack that GrugMoE serving will ride on: a selectable TPU paged-attention backend wired to tpu-inference kernels #6099, a backend-neutral PagedAttentionConfig family #6115 closing #6112, merged batch-merging-before-decode #6168 and streaming greedy decode by default #6130. On Discord David Hall noted that goal-mode optimization made Levanter inference ~10x faster, within ~12% of vLLM-TPU. The main caution comes from the serving path itself: @penfever filed #6136 reporting tpu-inference decode corruption on multi-host DP=2 with max_num_seqs ≥ 4, plus a porting-playbook proposal #6143, a generic correctness risk in the same vLLM-TPU stack GrugMoE depends on.

Quiet week. The June "climb local metrics" data-pipeline epic — a time-boxed process for pushing the local dedup, decontamination, quality, and domain metrics up from May's baselines — saw no direct activity. The upstream attribute deliverables it would build on did land under the mixture epics this week (Dolma3 quality #5812 and WebOrganizer domain buckets #5811), so the inputs exist, but the metric-climbing work itself has not been picked up.

The end-to-end machinery to produce the June run's data landed: #5822, the v0 datakit stage pipeline that normalizes, deduplicates, decontaminates, embeds, runs the quality and domain classifiers, tokenizes, and emits Levanter stores, merged on June 5. The two attribute deliverables it depends on also closed this week: #5812 (Dolma3 fasttext quality scores) and #5811 (WebOrganizer domain buckets), with Rafal Wojdyla confirming both are now feeding the ongoing mixing swarm and that the June run will use the 40-bucket luxical domains and the oracle-LLM quality signal. In #data-curation Will Held showed a clean monotonic gradient validating that the higher quality quartiles correlate with better coding perplexity in the mix.

What did not move is the actual definition of done: getting the tokenized, mixing-compatible data physically onto the CoreWeave datakit store. There was no ingestion or staging chatter in #infra, #data-mixing, or #deduplication this week, and the flagged ~week of ingestion lead time and the Iris full-ingestion stress test have not started; the cluster itself spent the week capacity-constrained and thrash-prone. Tokenization work that exists is the separate vocab study (100B-token subsets at varying vocab sizes), not the run's production tokenization. The datakit testbed #5200 remains parked until later in June. So the producing pipeline is built and attributes are flowing into mixing, but nothing is yet ingested onto CoreWeave.

The code-interpretation track narrowed from "can Marin do Python" to a much sharper diagnosis. David Hall landed #6080, the static target-only prompt-format and code-interpretation PPL suites plus 32B gap-runner scripts (REPL/doctest-style continuations like len('prompt') * 7), and #5337, which threads per-dataset token counts and runtime through the perplexity summaries and pins eval batches to dataset boundaries so the timing is attributable. Two new probe families followed: a tokenizer-coverage suite #6103 that exercises every representable Marin token id and flags byte-fallback ids that cannot round-trip through text, and the open v2 generators PR #6207, which adds deterministic slices for string/byte transforms, indexing, arithmetic, serialization, and format/style instructions, plus a matching ~1B-token plain-text corpus for the same skills.

David Hall deployed the resulting Marin 32B vs Qwen3 32B circuit-coverage gap dashboard, with per-character span heatmaps, and read it on #6070 as a representation/tokenization competence gap rather than a broad coding deficit: the gaps concentrate on tiny deterministic targets (arithmetic_repl +1.38 bpb, character_indices +1.32 bpb), where Marin knows the surrounding format but spikes on the exact character-level answer while Qwen stays smooth. In #evals he proposed materializing a few billion tokens of synthetic drill data, and Will Held endorsed adding it aggressively and forcing in eval versions so the mixing pipeline keeps it. A still-open thread is #6096, testing Qwen 2.5 base on the PPL sets and correlating with coder-model SWE-Bench performance, which is the epic's actual correlation target.

The production data-mix swarm #5365 that launched the prior week kept grinding through the week, but the headline event — reconciling Will Held's pre-registered vibe mix #6063 against actual swarm outcomes — has not yet landed. That preregistration remains the open bet on the table: derived from a DSP regression fit to the 241-run 300m swarm, it calls 5 confident wins (humaneval, github_python, github_cpp, gsm8k, and arxiv-shaped reasoning, all with |Δz/σ| well past 2) against 2 confident losses (uncheatable/bbc_news and fiction), with the wins judged far more credible than the losses because the win Δz's run an order of magnitude above the in-sample residual noise while the losses sit right at the edge of the σ buffer. Until the production runs report, that scorecard stays unsettled. To keep an eye on the runs through a rash of W&B timeouts, Will Held vibe-coded a standalone swarm dashboard at oa.williamheld.com.

Most of the week's substance in #data-mixing was a close read of Microsoft AI's data-mixing paper, which Will Held praised for its candor in trying and then abandoning a regression-based approach in favor of empirically trying many candidates and alternating global (across high-level buckets) and local (within a bucket, e.g. code) optimization — "useful but not magic." Notably their final mix was the best candidate at all scales, though they saw scale dependence on high-quality but low-diversity buckets. Calvin Xu dug into the rank-(non-)invariance question, observing that capping max epochs at 8 seemed to restore rank invariance and spinning off a "Scale Dependent Data Duplication" thread — the goal may still be to read the best mix off a single-scale swarm, just accepting that not all scale dependence is modeled. Will Held also surfaced the existing Dolma swarm's rank-correlation matrix and admitted growing skepticism that pretraining has as many exciting compositional mixing effects as hoped — to which Percy Liang replied, "It's called a 'prior.'"

On the data side, both domain-bucket and quality-score deliverables landed: the Dolma3 WebOrganizer 24-class topic classifier #5811 and Dolma3 fasttext quality scores #5812 both closed after running across all 104 datakit sources, supplying the quality and topic buckets the mixing optimizer mixes over. The handoff target — the 10T June datamix #6045 — still gates on the swarm resolving and a curriculum being chosen; Will Held clarified it does carry long-context data even though long-context is not yet weighted in the swarm architecture. Meanwhile Isaac Hodes floated closing the umbrella automatic-mixing issue #2403 as substantially done, even though a specific mix has not yet been chosen.

Synthetic-data generation moved decisively onto Marin's own compute this week. @penfever stood up #6133 to track agentic trace generation on pre-emptible Iris TPUs through the OT-Agent launcher — documenting how to launch datagen, collect and upload traces, and reading off throughput grids and pre-emption stats — and opened #6191 as a running index of the resulting HuggingFace trajectory datasets, with twelve teacher-model cohorts (e.g. MiniMax-M2.7 @ 131k, with Qwen3.5-122B-FP8 and Gemma4-31B incoming) registered so far. On the prep side, two post-training data PRs merged: #4997 wired NuminaMath-TIR into the datakit download/transform/normalize pipeline, and #4637 unified DPO and LoRA-DPO under a single Levanter entrypoint with a durable reference-eval cache, broadening the preference-tuning surface for post-trained checkpoints.

Two long-running data-consumption experiments closed out. #2956 finished SFT'ing Qwen3-8B on 5K fresh SWE-smith Rust trajectories generated by GPT-5-mini through the mini-swe-agent v2 harness, and #2441 closed the DeepMath-Zero-7B AIME25 replication on the decontaminated DeepMath-103K corpus. Downstream, @penfever wrapped the 96-way single-dataset RL ablation #6187 on Qwen3-8B, uploading winning checkpoints under the laion/a3-rl-*-8B namespace and surfacing which Nemotron-Gym and math datasets actually move reward versus those that collapse or are too easy/hard; freshly-landed Nemotron RL datasets are being converted to Harbor format and quality-checked for a TaskTrove V3 release. Distillation teacher-selection #3956 and the logprob-distillation formalization saw no new movement this week.

The bulk of the week's unscoped work was a sustained code-health and static-analysis push by Russell Power. A long pyrefly campaign drove the type-checker baseline to zero (#6248, #6223, #6219), enabled missing-import and interpreter queries #6221, killed jaxtyping false positives natively #6224, bumped pyrefly to 1.0.0 #6220, and added torch/sympy to the ignore-missing-imports list #6266; follow-up triage and a round-two ratchet are tracked in #6225 and #6242. Ruff gained local-import lints #6222 and a ban on function-local imports #6249, and the lint catalog itself was rebuilt to run as parallel agent lanes with a composer and a read-only --review mode wired into PR code review (#6241, #6262, #6267, #6268). The codehealth tooling that classifies PR comments was batched, parallelized, cached, and hardened against non-serializable artifacts (#6239, #6250), and the author-pr skill was folded into the commit skill with tightened format rules and a required review step (#6260, #6193). An issue to move off Python 3.11 toward requires-python >=3.12 was filed in #6233.

On dependency and build hygiene, Russell Power restricted dependabot to grouped security updates only #6152, removed the experimental kitoken tokenizer backend and its dependency #6158, and unified the pyarrow floor to >=23.0.0 across subprojects #6245; dependabot landed grouped npm and uv-security bumps in #6145, #6165, and #6208. Russell Power also opened #5682 to rename the lib/marin PyPI distribution to marin-core after PyPI rejected the bare name. Elsewhere, David Hall added a hero-run management skill and a checkpoint-candidate discovery helper for recovery scripts (#5981, #5991), and Michael Ryan automated a weekly storage report with week-over-week diffs running on non-preemptible capacity #6144.

Outside the core team, external GitHub activity skewed toward data and post-training plumbing. Tai Vu opened a batch of datakit and midtraining source PRs — a Nemotron Code v3 materializer #6257, OpenCode instruction staging #6256, Nemotron Specialized v1.2 #6255, and the full OpenMathInstruct-2 corpus #6254 — while @penfever drove the agentic-datagen and RL-ablation tracks #6133, #6187, #6191 and filed a tpu-inference decode-corruption report #6136. Several first-time contributors landed small fixes: @Kymi808 added a missing @classmethod to Gemma3Config.from_hf_config #6097, @Lang-Qiu modernized type hints #6246, @awen11123 made artifact serialization fail gracefully #6250, and @mvanhorn fixed LoraLinear handling in get_lm_head() #6161. On evals, Nikil Ravi opened a WIP RULER eval #6110, Yiyuan Li filed a Marin-vs-Qwen3 tokenizer-coverage comparison #5842 feeding the byte/tokenizer gap work, and @surface-integral proposed reproducing the MANO optimizer claims #3335.

On Discord, #data-mixing hosted the week's sharpest exchange: Will Held walked through Microsoft AI's data-mixing report — its candid pivot from a regression fit to empirically alternating global/local optimization — and voiced growing skepticism that pretraining holds as many compositional mixing effects as hoped, to which Percy Liang replied simply, It's called a 'prior'. That thread ties directly to the still-open vibe-mix reconciliation #6063 and the metric-selection debate #5362. @penfever also used #introductions to make an open call for the community to post-train marin-8b/32b-base against the instruct and OLMo-3 baselines, and to pick up any of the ~500 open issues.

New arrivals self-described as drawn to fully-open LLM development: an Amazon scientist on agentic post-training and evaluation, a Cisco AI Defense lead researcher, a YC-alum founder building an agent virtual-filesystem, a Mila PhD on cross-datacenter distributed optimization, a Tübingen post-doc with prior MoE-scaling work, an EEG-foundation-model PhD visiting Princeton, and independent researchers from ML Collective and elsewhere who found Marin via CS336 and Percy Liang's PyTorch keynote. Two intersect cleanly with active work: Ben Feuer joined as post-training lead, having co-led OpenThoughts-Agent, intersecting with the agentic-datagen and RL ablations now running on Marin TPUs #6133; and Darrow, a Stanford senior doing test-time RL on frontier-math problems, intersecting with the math-RL datagen sweeps #6187. The welcome-room also saw a dozen silent joiners with no intro; none resolve to a recognizable author of a widely-used kernel, architecture, or optimizer, so none are surfaced here.

Shared reading skewed toward optimizer/scaling-law theory and post-training signal — hyperparameter-transfer and parameterization-dependent scaling laws, length-generalization notes, the Microsoft AI mixing report, number-tokenization, and RLVR results — alongside Larry's MoE-speedup write-up and the MarinDNA observatory.