Marin: Week of October 19th summary

The run crossed two token marks and changed its data mix. From the W&B history, it opened October 19 at step 287,631 and 13.27T tokens and stood at step 328,186 and 15.14T tokens at 17:21Z on October 25, about 40,500 steps and 1.87T tokens in the week. The 14T mark fell near step 303,440 at 14:42Z on October 21 and the 15T mark near step 325,120 at 05:17Z on October 25. Daily median MFU ran 30.3–30.6% and median step time 13.79–13.89 s, unchanged from last week's 16K build. The learning rate decayed from 0.00095 to 0.00067 on the linear schedule. Daily median train loss moved from 1.079 on October 19 to 1.067 on October 22, then 1.044 on October 23 and 1.037 on October 25. Paloma macro loss went from 2.0706 at step 287,999 on October 19 to 2.0442 at step 299,999, 2.0445 at step 311,999, and 2.030 at step 326,999 at 12:41Z on October 25; UncheatableEval macro loss from 1.544 to 1.521. The eval_regressed alert fired on the 0.0008 uptick at step 311,999 and cleared at the next eval.

The week had one logged stop and one unexplained pause. At 00:55Z on October 23 the coordinator was preempted on purpose to rotate credentials, and the 176-task child gang then failed twice at startup with an XLA CoordinationService/WatchTasks connection refusal; Iris requeued all 176 trainers at 01:00Z and 01:05Z, and the second attempt restored the rolling checkpoint step-312162 at 01:11Z with no manual intervention, as loom-oa-dev logged. The same refusal hit again at 02:03Z, and the coordinator was retried at 02:07Z to reload keys after a revocation; the replacement child started at 02:13Z and the follow-up at 15:29Z found 177 tasks running and Finelog at step 315,506. The initiating cause is still unconfirmed. W&B logging paused from 00:54Z to 01:17Z and again from 02:01Z to 03:11Z, about 1.5 hours in all. The rotation followed #10475 by Russell Power, merged 18:18Z on October 22, which makes CoreWeave jobs carry the submitter's own keys and notes that running legacy jobs, hero runs included, must be resumed with personal keys before the controller rolls out. W&B also shows a 27-minute gap from 06:32Z to 06:59Z on October 20 with no thread entry. #10257, merged October 19, finished the runbook fix from last week: a retryable Finelog query failure now yields non-firing rows for hero alerts while FinelogFleetUnhealthy reports the shared failure.

The phase-2 data mix started on schedule. The hero plan in #8435 places the mix boundary at 80% of the run, step 312,201, and the W&B history shows the 100-step median train loss dropping from about 1.062 to 1.040 there, in the hour after the October 23 restart; Larry Dial posted the last-20% mix update in #hero-run-2026 at 03:16Z. The team weighed turning off the 500M-token block shuffle at the same boundary, after Larry Dial tied the waves in the hero's train loss to it and #10412 showed a sequence-level shuffle removing a 128-step sawtooth at small scale; Larry Dial decided to leave the run as is, since the waviness is smaller at 16K and the gain is under a day of compute. The outside forecasts of the final loss posted this week, and the finding that 4K-era forecasts start about 0.03 high after the 16K switch, are covered under the hero engineering section.

Work for the 65K phase went ahead without touching the live job and is covered under the hero engineering section; the operational changes were smaller. The vibe-check sampler now runs at temperature 0.7 from step 270,000 under #10224, merged October 21, because 0.2 gave repetitive text; loom-oa-dev's #10469 proposing 1.0 drew a no from Larry Dial. In #10428, David Hall's agent averaged the step 282,000, 288,000 and 294,000 checkpoints with LR-weighted coefficients, seeded five chat tokens and ran 64 LoRA updates to produce a provisional chat checkpoint for RL capacity planning without a cooldown; all 100 panel responses ended at EOT. #10221 merged October 19 and routes CoreWeave scratch to a separate bucket while the hero keeps its rolling checkpoints in hero-checkpoints, with cleanup staged in #10279 for no earlier than November 2. The logbook again posts no finish date or date for the 65K switch; this page's estimate from W&B is that with 62,065 of 390,251 steps left and the week's 6,030 steps a day, the run would reach step 390,251 near November 5 if it stayed at 16K, a day later than last week's estimate because of the October 23 stop.

The context-parallel stack for the 65K and 262K phases moved from draft to main. #10145 merged on October 19: the shared and routed MLPs now flatten tokens inside a shard_map, so under context parallelism (CP) each device flattens its own rows and sequence shard and the compiled CP2 forward has four all-to-alls instead of eight. On one GB200 rack at 65K on CP4 and expert parallelism (EP) 16, the step went from 15.938 to 15.586 s and the drop fraction from 1.17% to 0.23%. The change moved which tokens each expert-parallel group holds, which broke three context-sharding tests that CI had not selected because they import the hero model only in a subprocess; #10313 fixed the references and added module-level imports so the selector sees them. #10121 merged on October 23 after Amy Sartran approved it: document packing gives rows very different attention costs, and on real 16K batches the busiest of 64 shards did 2.10× the mean global-attention work, so the train step can now sort rows by attended query-key pairs and deal them to shards in snake order. Restored from step 300,000 at 65K on CP4/EP16, step time fell 3.63% from 15.352 to 14.743 s; at 16K on EP64 it fell 0.40%. Sartran asked whether it needed a flag since it had not run on more than one rack, and Matt Wittmann put it behind balance_attention_rows, skipped it when each shard holds one row (as at 262K), and fixed a bias that made drops likelier for rows of many small documents. The cost is an all-gather of the whole batch, about 0.55 GB per GPU per step at the 11-rack 65K layout; Wittmann's pessimistic projection is about 100 ms exposed across InfiniBand, and he plans to watch it at deploy. Two PRs then prepare the switch. #10610 makes the zigzag query swap from #10170 configurable, because it pays at 262K but at 65K the contiguous layout ran 14.639 s against zigzag's 14.743 s, about 0.7% and near the step-time noise; the README now says zigzag at 262K, contiguous at 65K. #10618 teaches the production launcher, which had fixed the expert axis at all 64 GPUs of a rack, to run CP: the degree and row balancing become checked-in constants so the 65K deploy changes one line each, and the eval batch becomes one sequence per batch shard, 176 at 65K CP4 instead of 704, keeping 16,384 tokens per GPU on the dropless eval mesh. A Codex review flagged that the CP1 layout test pinned live defaults and would fail at the 65K deploy, which was fixed. On W&B, a pair of 500-step rehearsals from step 300,000 on main compares the two 65K layouts: CP4 at capacity factor 1.15 ran 14.69 s per step at 36.5% MFU, CP1 at capacity factor 1.5 ran 15.45 s at 34.7%, with losses 1.0111 and 1.0105 at step 300,499. The dropless eval also ran under CP4 at 65K on a rack for the first time, logging a Paloma macro loss of 2.046 to 2.049 against 2.051 for a 16K control from the same checkpoint.

A kernel campaign from boba shop cashier had mixed results on the rack. #10311 computes the cross-entropy gradients during the forward while each block of logits is still in memory, as Liger-Kernel does, so the loss takes three matmuls instead of four; on one B200 at the hero's per-GPU shape that is 243 ms against 351 to 356 ms, and a sweep showed blocks of 2,048 to 8,192 rows within noise while 512 and 1,024 were slower. Wittmann's agent benchmarked it on one GB200 rack at the hero's 16K configuration from step 240,000: 13.079 s per step against 13.142 and 13.148 for two base runs, +0.50% tokens per second at +0.6 GiB, with a loss difference against base of 1.6e-3 rms where the two base runs differ by 1.3e-3. It merged on October 22. #10312 fuses RMSNorm and GatedNorm into one op with Triton backward passes, cutting activation reads and writes from 27 to about 23.5 per step and saving 83 ms on one B200, but the same rack test ran 0.85% slower at 13.258 s, and the two together landed at −0.02%. Wittmann asked whether an XLA fusion had broken; Larry Dial said the same technique speeds up a new model with a different architecture and guessed the hero is tuned to its memory profile; Wittmann declined further rack rounds and asked that the Levanter op land without the hero wiring, to be tried in the next optimization campaign. In Discord, boba shop cashier had hoped the pair would take MFU from 30.5% to about 31%, and after the rack results boba shop cashier concluded only #10311 was good. The larger bet is #10415, which folds in #10414: an opt-in ExpertPrecision.MXFP8 that runs the six routed-expert GEMMs of the ragged EP backend on cudnn-frontend 1.31.0's fused grouped-GEMM kernels, with E4M3 values and one E8M0 scale per 32 elements that Blackwell applies inside the MMA at twice the bf16 rate. On one B200 at one hero expert chunk under remat, the expert MLP took 31.9 ms against 51.0 ms for the bf16 QuACK kernels at a lower peak, 13.5 against 15.9 GiB; the receiver buffer now aligns each expert's rows to 256, checkpoints restore under either precision, and the next step is a one-rack A/B. Larry Dial reported 36% MFU on one rack with MXFP8 for a new model, with the fused norm included in the 36.8% figure, and #10559 ports the kernels to that model's ReLU² experts, where boba shop cashier expects about 3% MFU.

David Hall's pipeline-parallel (PP) line for the hero's post-training merged in three parts. #9279 is the H100 foundation: a stage-local JAXPP pipeline over the full MoE blocks with 1F1B as the default, and each routed expert bank now shares one Frobenius norm across its experts to match the stacked EP model. #9283 adds CP to the pipeline mesh, with next-token labels formed from global token IDs so a context slice can target the first token of the next slice; the historical 48-layer run completed ten finite updates on 192 H100s at PP24 × EP4 × CP2 and 65,536 tokens, 68.5 s per step at 9.88% MFU. #9662 adds the GB200 launcher, a pinned JAXPP commit plus host-startup patch, and fresh Harrier data; an eight-GB200 gate passed two updates, while the 64-GB200 rack attempt failed during install, and the reconciled branch has not been rerun on GPUs. Romain Yon asked about PP8/EP8, which would trade EP64 traffic for EP8 plus activation transfers; whether it wins on GB200 is open. The RL-readiness work around this launcher, including NCCL channel limits in #10368, pinned-CPU activation offload in #10376, and checkpoint averaging with compact SFT in #10539 and #10556, is covered under Other Changes.

On the science side, two outside forecasters posted on the epic. @mindscanlabs forecast a final Paloma macro loss of 1.979 with a 90% interval of 1.965 to 1.994 from the ladder plus the hero's own evals, or 1.935 from the ladder alone, using a symbolic-regression law with no assumed floor; the formulas are sealed by SHA-256 digest. Qirun Li, whose preregistered forecast is #9977, measured that the 16K switch alone lowered the Paloma macro by about 0.031 on nearly the same weights, 2.1234 at the 4K eval of step 252,000 against 2.0929 after restoring it at 16K, so any 4K-era forecast scored against the 16K final starts about 0.03 high; Larry Dial committed to re-evaluating every checkpoint at 4K once the run ends, without claiming a context-extended model evals at 4K like one that stayed there. In data-mixing, Larry Dial suggested the waves in the hero's loss come from the 500M-token block shuffle, since many documents run to about 500K tokens and each is its own distribution; David Hall pointed to #2917, where block shuffling cut class B traffic about 90% at minimal loss cost, and said it probably should have been off for the hero and could be switched mid-run. Larry Dial judged the waviness smaller at 16K and the gain under a day of compute, so the run keeps it; the phase-2 mix for the last 20% was due within days. For the checkpoint completions in #8827, #10224 raised the sampling temperature from 0.2 to 0.7 from step 270,000 because 0.2 gave repetitive text, and #10469 proposes 1.0.

Marin released its first Snowball checkpoint this week. Benjamin Feuer sealed all 936 cells of the eval-policy tracker #10197 and reran the tournament; the new winner is Smartie RL step 14, with a mean paired win rate of 0.684 across 22 Snowball and Grug opponents against 0.633 for Smartie step 12, which it beats 4-1-18 head to head #10283. That replaces last week's DARE-Linear merge, now frozen as the v0 record in #10199. The weights are public as Marin Snowball 67B-A2B v0.1, and the 15.5 GB trace archive behind every tracker cell is sealed. The run that produced it, written up in #10305, started from the Campaign 2 DARE-TIES-02 merge rather than v0: asynchronous leave-one-out policy gradient (RLOO-N) with truncated importance sampling on 64 policy and 48 inference H100s, learning rate 4e-6, 512 prompt groups of 16 per update, over four NVIDIA Nemotron RL domains plus 57,364 synthetic structural-compliance tasks. By update 14, native validation pass@1 rose from 41.21% to 50.22% and NVIDIA-task reward from 40.59% to 55.50%, while structural reward fell from 39.09% to 32.92%; the mixture ran unshuffled, so Feuer reads the result as task learning with a recorded regression, not balanced joint optimization. A continuation run from step 14 on repository-setup and structural tasks was stopped after 12 updates when SWE-Smith validation went 27.2 to 34.3 to 9.8 and TaskForge 72.9 to 6.3 to 0.9. A manual safety audit of 100 head-to-head chats against Qwen 3.6 35B-A3B found no unsafe actions by either model but Qwen ahead 59-19 with 22 ties, with Snowball claiming success without tool evidence in 16 cases against 7 #10269. An identity SFT in #10491 taught the model to name itself as Marin Snowball from Open Athena (identity panel 7/60 to 45/60) and shipped as v0.1.1, but it cost MATH500 461 to 449 and GSM8K 1224 to 1038 of 1319, and user testing then exposed fiction refusals and unsolicited self-attribution; a larger V19 run was rejected after MATH500 fell to 371. Feuer's attempts to move the identity back by on-policy distillation (OPD) and by a teacher-repair Group Relative Policy Optimization (GRPO) pilot both scored 0 of 20 held-out identity conversations, and he traced the GRPO failure to rare identity tokens whose gradient coefficient is about 92,000 times smaller than other teacher tokens', with the code preserved in #10652. Gonzalo Benegas noticed the v0.1 BixBench runs used Pi 1.3.0 where the policy pins 0.87.0; Feuer called it a missing pin and is patching the launcher.

How v0.1 behaves in agentic harnesses got close scrutiny. Karan Sagar found it takes about twice as many steps per task as v0 #10338: on 100 Terminal-Bench Lite tasks it solved 23 against v0's 20 but hit the 1,800 s limit 49 times against 8, solving 40% of the trials it finished against 22%; on SWE-bench Verified random-100 it solved 26 against 33 and ran the submit command in 37 trials against 70, ending 59 of 100 at the 32k context limit. A GLM 5.3 reference row scored 78/99 and 79/97 on the same sets. Reruns with a 5,400 s limit (24 solved) and a 64k context (26 solved) moved the score by one to three tasks, inside the run-to-run churn, so Sagar concluded the timeouts are a symptom of long trajectories rather than lost score. Feuer's context ablation across four models #10493 put v0.1 at 0.243 on SWE-bench Verified at 32k/8k, 0.322 at 65k/16k and 0.334 at 131k/16k, with character-budget summarization cutting every model by 0.26 to 0.35, and he proposed moving the policy default to 131k/16k without summarization. Qirun Li questioned the tournament's unpaired tie rule on #10197: both models answer the same tasks, so the paired standard error is 0.63 to 0.94 times the unpaired one, and 51% of benchmark comparisons in the two tournaments are ties; the v0.1 pick holds at any scale factor down to 0.5, but the v0 pick would have flipped to Campaign 2 at factors of 0.96 or below. David Lundgren documented that Grug checkpoints emit stacked or stray think segments that leak into replies, in 9.1% of Smartie step 14's thinking messages on BixBench against 2.5% for the September 21 SFT #10400. He scanned 2.1M SFT assistant turns and found no source there, then traced the habit to the swe-zero-12m pretraining source, where 0.8% of documents close a reasoning span and keep going; SFT suppresses it and RL rewards it because the answer extractor reads past the last marker. The RL-side fix rejects malformed rollouts, a planned pivot rerun tests whether that alone stops the AIME24 drift #10585, and the SFT-side patch was closed as not planned #10521.

The preregistered agentic SFT shootout #10213 returned its first results, and no arm met the pass rule. Sagar first regenerated the GLM 5.3 pool at reasoning effort high after the max-effort traces cut 65.6% of SWE-smith trials at the 30k window (high: 32.8%, low: 6.6%, pass rates unchanged at about 61 to 69%), switched the base to v0.1 because it carries no relay graft, and required demonstrations to end with the agent's own submit. Every arm trained on GLM traces then scored below its parent on the agentic screens: from v0.1 (26 SWE-bench r100, 23 TBLite-100), plain SFT on high-effort rows scored 13 and 8 and relay on low-effort rows 25 and 15; from v0 (33 and 20), plain scored 17 and 9, RS-multi 11 and 10, recovery 12 and 9, relay 13 and 14 and plateau dropping 14 and 13, while GSM8K and BFCL-Parity rose in every arm. The only method-versus-twin interval that excluded zero was relay over plain on low-effort rows, +8 on SWE-bench [+1, +16]. The mechanism, read off trajectories, is that students keep the rows' per-step style but take twice the steps, hit the 32k window in 75 to 97% of episodes and rarely submit (3 to 6% for high-effort students against 36 to 56% for the parents). Karan Sagar said keeping the arms token-matched under the policy's 32k context was the hard part, since Snowball uses most of it. Next the cheap arms rerun on the contributor Kimi and Qwen traces, where the learning-rate sweep's 1e-6 run had beaten v0 (composite 31 to 32 against 26.5 to 27, TBLite 29 against 20), with a 3e-7 check on the real rows. Marianna Nezhurina reported a parallel finding on Antidoom step 12: across five teachers, GLM-5.3-flash is the strongest and most distant teacher and produces the worst student, and closeness to the student predicts SFT success better than teacher strength. The SWE-bench preregistration #8933 also resolved: Ricardo Olmedo predicted 40.6% pass@1 for the released base fine-tuned on 7.55B Nemotron Lightning tokens and measured 38.4% ± 1.1, a miss of 2.2 points that sits almost entirely in the bpb forecast (0.1331 predicted, 0.1380 measured); at the measured bpb the link laws err by under a point. Ricardo Olmedo asked for the pre-SFT checkpoints of the Snowflake scaling ladder, two tiny Snowball replicas Will Held published for RL debugging.

Several inputs to the next candidate matured. Lundgren closed the mixed-domain RL with verifiable rewards (RLVR) experiment #10214 with a second seed: the arm that mixes selected PivotRL pivots into 20% of prompts reaches 0.348 and 0.328 on SWE-bench Verified at step 8 against 0.260 for the October 5 SFT, beats random turns by 10.0 and 12.6 points at both seeds and plain RLVR by 9.2 and 3.0 (pooled +6.1), with no reasoning loss; by step 12 every arm loses Math500 and MMLU-Pro, and his malformed-span scan ties the arm with the largest loss to the highest malformed rate at both seeds. David Hall built a fast RL loop around a frozen 100,000-task curriculum in five equal families #10314: a 100-update RLOO-N continuation from step 14 lifted the 910-task development panel's equal-family mean from 30.96% to 39.98% by update 25, and 50 updates from the September 21 Datakit SFT and from Dr.Doom reached 40.09% and 42.29% from 24.68% and 17.57%, on reused development tasks rather than a held-out split; David Hall noted cold-start SFT may be the stronger initializer. The selection and launch code merged in #10561, #10562 and #10605, with structural closure tasks ingested in #10319. Sagar's transfer follow-ups weighed against distillation: a 4x learning rate and 2x updates did not move the multi-expert OPD arm in #9984 at a cost of 989 H100-hours, and in #9920 rejection-sampled SFT reached the expert's CruxEval level at 43.5 H100-hours against 203 for OPD. On serving, Sagar found the TPU-vs-H100 gap was the tpu-inference config applying the 2,048 window to the seven global attention layers; with a layer_types override, MATH500 on TPU went from 0.878 to 0.940, v0.1 matched H100 on 16 benchmarks on a v6e-8 and on six on a four-host v4-32, and Levanter served it at parity but 35x slower than vLLM #10270. The public Snowball family tree merged in #9737 with 130 nodes, Sagar proposed main as its single source of truth #10598, and #10599 adds the ten transfer checkpoints. Post-training and release of the 67B-A2B model, originally slated for September as an ambitious goal, continues with v0.1 out and the shootout and RL loop feeding the next candidate.

SkyRL moves into the Marin tree. Russell Power vendored the SkyRL fork into vendor/skyrl in #10306, a 123,000-line import that keeps SkyRL's Torch and CUDA dependencies in their own lock behind an isolated marinskyrl executable, so RL changes land with Marin instead of through a fork pin. The gate was a full pipeline on two H100s: train Qwen2.5-0.5B to step 2 with Megatron, export, restore in a fresh job, train to step 4, and score the export on 128 GSM8K examples through Marin's evaluator. #10458 deleted the migration scaffolding, David Hall's #10463 let CI find the 270 SkyRL unit tests by glob, and #10588 removed an Open-MOPD harness whose images could not be built. The move also closed #10117, where the external lock had kept a stale Rigging and stopped the CatCount nightly.

The transplanted SkyRL stacks. About thirty open MarinSkyRL PRs were replayed as Marin PRs with their source commits recorded. Merged: #10437 shares BF16 weights with their FP32 masters in Megatron checkpoints, 134 GB off a 939 GB checkpoint; Karan Sagar's CatCount distillation canaries on CPU #10361, four H100s #10373, and two teachers #10378, plus #10522 after two canaries on one host collided on port 18080 and one accepted the other's teacher. Ahmad Qamar reopened his stacks on the vendored tree: faster startup in #10661 through #10664, worker-side batch assembly in #10666 through #10668, and exact vLLM-trainer numerics in #10634 through #10636, whose online check counts sampled tokens whose float32 log-probability bits differ between engine and trainer; an 8-H100 run asserted zero. Romain Yon opened score centering in #10429, off by default: on Discord Romain Yon explained that the paper's under-1% overhead assumes inference and training share HBM, while across nodes he measured about 30% and no learning gain on Snowball, so #10658 adds probes that report what centering would have done.

RolloutEngine and the task runtime. The harness refactor #10203, by a Claude agent supervised by Rafal Wojdyla, merged after Russell Power asked how a running harness is cancelled: async cancellation in one process, with commands killed on the Shellbox machine. #10264 makes the same engine a managed agent, a registered harness plus any OpenAI-compatible chat endpoint, and #10442 lets a harness cap each command's output after a build program printed an entire TaskSpec into a 250,000-token request. Hall landed explicit prefill in #10406 and assistant prefixes in #10528; when Power asked why, Hall answered that about a third of the RLVR1 (RL with verifiable rewards) tasks the Snowball mix is based on end in an unfinished assistant turn. David Lundgren's draft #10399 adds four protocol probes to TaskSpec 0.29, grading file submission, real tool execution, thinking-off compliance, and URL loading apart from answer correctness, after the BixBench audit in #9258 found answers left in chat and narrated shell sessions. Benjamin Feuer found in #10570 that math grading crashes on chained relations of unequal length and on \neq.

Taskforge. Mark Muchane's synthesizer is landing as a stack cut from the 11,000-line draft. Merged: the stage coordinator #10087, the GLM-5.3 client with agent loop and call recording #10291, which also moved default sampling from temperature 0.7 to GLM's published 1.0, and the JSONL and Finelog ledgers #10292. Under review: typed proposals #10295, triage #10296, a restricted program loader with no open, exec, or eval #10297, the 10,000-line build runner #10298, and solver-and-adversary validation #10300. An analysis of 1,468 gen1000 environments, posted on #10265, found about half of sub-1.0 rewards are grader strictness rather than policy failure. #10504 folds a shell task's instructions into its last user message, because in a Qwen3-30B-A3B test 36% of rollouts answered without reading their input file.

Agentic Taskforge and the Baby recursive self-improvement (RSI) loop. Wojdyla's agent built the second flavor for the RSI loop: #10265 turns failed rollouts into capability ideas, and #10444 gives one GLM agent per idea a bubblewrap shell and six tools, with an evidence gate the agent cannot bypass: a wrong control must fail, DeepSeek V4.1 Flash must pass 10 to 80% of at least eight solver rollouts, and a blind GLM review decides last. #10445 also grades perturbed wrong answers, which found the same 16 loose graders as a hand audit over 373 tasks, and #10621 keeps each task's build program so new instances can be made, after a Qwen3-1.7B run learned to guess a family whose answers were always 3, 4, or 5. The loop closed in #10576: a vendored SkyRL job trains full-parameter Group Relative Policy Optimization (GRPO) from a live S3 task feed while the gate solves with the live policy. With pool and synthesized tasks mixed 50/50, Qwen3-4B went from 0.145 to 0.684 on probes of the synthesized families against 0.119 for a pool-only control, and Snowball 67B-A2B from 0.298 to 0.820 against 0.549; both RSI arms lost about 0.05 on held-out Smartie tasks, having seen half as many pool tasks. The stream loop #9626 and feed training #10622 remain open.

Hero post-training readiness. Hall's checkpoint averaging, chat-token donor initialization, and 64-update compact LoRA SFT for the 535B hero in #10428 produced a provisional checkpoint whose 100 panel responses all ended at EOT. The 18,500-line reproduction #10539 was superseded by #10556, which keeps only the FP32 routing-bias export and frozen-weight gradient fixes; the rest lives in the new marin-logbooks repository under the workflow #9467 defines, which on Discord David Hall said should stop him opening spiraling PR stacks of exploratory code. Yon posted the pinned 192-H100 learner recipe on #10388: pipeline parallelism (PP) 48, expert parallelism (EP) 4, context parallelism (CP) 4 with recorded-expert replay cut the gradient change from reordering examples from 71.96% to 1.37%. Two ported knobs await GPU qualification: eight NCCL channels in #10368 took a warmed 65K update from 50.74 to 29.08 seconds, and pinned-CPU activation offload in #10376 fit 64 sequences per update for 32.8% more tokens per GPU-second. Hall's native JAX learner, #10557 with JaxPP execution in #10600, stays in draft.

Post-training research. Feuer's on-policy distillation (OPD) versus SFT controls experiment is scaffolded in drafts: an Axolotl launcher #10490, one chat template and budget for all four bases #10505, the X and R task families #10510, pinned probe sets #10511, a binned sequence-KL estimator #10483, and a 30-cell matrix in #10513 where step s of the SFT arm trains on exactly the prompts step s of the distillation arm rolls out. Lundgren found in #10520 that 2.3 to 7.1% of rollouts per step in a plain RLVR run had malformed reasoning spans and were still rewarded; an 8-step rerun with rejection cut the rate to 1.4 to 1.6% at unchanged validation reward. After Will Held argued on #10542 for a soft penalty like the length penalty rather than discarding signal, the design moved to #10624, with a 0.1 default charge on the offending token in #10626. In #10546 Lundgren ran PivotRL on Qwen3-30B-A3B-Thinking and raised SWE-bench Verified from 4.6% to 15.6% in ten steps with AIME24 flat, then corrected the numbers after finding a 73,728-token eval window had truncated the trained checkpoints' longer runs. @raspberryice reported in #10320 that 1,142 of 11,817 instruction-following rows in the Nemotron RLVR1 training set reward all-caps or letterless answers.

Evals and Harbor. Power vendored both evaluators, Harbor in #10287 and Evalchemy with all 42 dataset generators in #10288, each with its own lock and run as a subprocess; Held noted that reproducing an old eval policy now means checking out the old Marin commit. #10471 removed LiteLLM and blocks its installation, after #10466 pinned mini-SWE's copy because a fresh sandbox could resolve the compromised 1.107.0 release of October 21; #10481 notes the retained logs cannot show which inner version was installed. Held's #10169 recovers brokered evaluations after inference preemption, and Sagar's #10526 restored Snowball v0.1 BoolQ from 0.529 to 0.834 by keeping a generation cap off loglikelihood requests. For the eval policy #10197, #10650 grades FinanceBench and OlympiadBench with MiniMax-M3 by default, and Feuer's ablation in #10493 supports a 131k SWE-bench context with summarization off. A first-time contributor, @XiaoyuanWang111, followed the IFBench audit with a grader-audit skill in #10422 and checker fixes in #10423, one of which had sliced character indices where every prompt asks for word indices. Gonzalo Benegas added three no-tools LAB-Bench components in #10582 and recorded on #9258 that Snowball v0.1 scored 9.3% on BixBench, with 156 of 205 tasks never writing the answer file.

Kernels and AMD. Amy Sartran's #9880 adds ring_filtered, an expert-parallel MoE backend that sends each token only to the GPUs owning its experts; at batch 64 each collective moves 537 MB per GPU instead of 1,174 MB, and on #9812 it reached 15.82% MFU on 8x MI355X against 14.30% for the default. #10344 reorders Pallas-Triton flash-attention launches so the longest blocks start first and retiles dK/dV, cutting attention time per step from 245 to 167 ms. Still open: #10345 derives the routing-weight gradient from the down projection's backward instead of saving expert outputs, freeing 16.5 GiB so the production per-GPU batch fits, and #10673 adds XLA overlap flags for multi-node runs, worth 13% on two MI350X nodes where a quarter of the step was an exposed all-reduce. For Blackwell, boba shop cashier's #10559 runs Belay's ReLU² experts on cudnn-frontend MXFP8 grouped GEMMs, 61 ms against 91 for QuACK on one B200, mostly from fusing the activation backward; on Discord boba shop cashier estimates about 3 MFU points.

Levanter and training framework. Hall's #10564 moves ArrayStacked layer stacks out of Haliax so array-native Grug models run without it. Feuer's #10112 repairs DPO exports, which were scheduled by step rather than completed optimizer update, and his #10347 adds an MLX build pipeline for Apple Silicon: the published 8-bit and mixed 4-bit Snowball builds match the vLLM reference at top-1 agreement of 0.994 or better and decode at 9 to 12 tokens per second on a 20k-token prompt on an M5 Max. #10484 would unify HF release exports across Levanter and SkyRL after #10478 showed v0.1 shipped BF16 in 39 shards and v0.1.1 FP32 in 79. Two canary failures were fixed: BF16 safetensors ignored dtype_override in #10267, and TensorStore could not read mounted S3 credentials in #10571. Ben Kolligs's SigLIP encoders in #10076 got a first review, and Sagar documented in #10453 that resuming into a root holding uncommitted partial checkpoints fails the next save.

Pretraining research. Sartran began tidying the Belay registry: every trial carries an id in #10268 and commit fields must be hex SHAs in #10447. Wojdyla's quality-classifier fast track #10108 scores a 10.7B-token raw pool and gates selections on 13 cloze tasks plus gold-answer BPB rather than Paloma macro loss, since across 99 dense runs Paloma ordered selection pairs like the capability target on only 11 to 40% of pairs. Feuer archived two negative results: the Noise backprop-free method in #10581, whose readouts cost 390 times a forward-plus-backward, and the tied-drafter study in #10654, where native training at 12,288 updates beat every four-expert draft at 16,384 in both loss and cost. Rohith Kuditipudi published recipes for 20 more Delphi seed-repeat checkpoints in #10555. Larry Dial showed in #10412 that sequence-level shuffling removes a 128-step loss sawtooth and improves Paloma by 0.0135 at d512. Hall proposed a metadata-only release of the hero's 292-source document pool in #10527.

Iris, credentials, and security. Power's #10474 delivers application secrets as per-attempt tmpfs files and pod-owned Kubernetes Secrets instead of environment injection, and #10475 requires every CoreWeave job to carry the submitter's own storage keys, removing shared operator keys from task defaults; the vLLM nightly was the first casualty in #10566. Sagar found in #10440 that launch provenance had written raw -e secret values into 25 .artifact.json records in object storage; they were redacted in place and the owners told. #10473 keeps host credentials and loader variables out of bubblewrap sandboxes. The open #10538 adds an egress allowlist enforced by Cilium after a survey found 23 of 346 running pods reaching hosts outside the base list, 11 of them Harbor evals. After private keys for eight eMudhra intermediate CAs proved recoverable from their certificates, #10435 bumped certifi, #10574 rebuilt the Iris, Finelog, Marina, and Grafana images across six trust stores, and #10580 covered uv, Node, and third-party images. Held's #10465 enforces parent budgets across federation, and #10221 routes all CoreWeave scratch writes to the new marin-us-east-08a bucket. The cw-rno2a peer went unreachable for two hours on October 21 in #10433, stalling evaluations whose model traffic crosses the hub proxy.

Finelog, FineStore, and Grafana. Power raised Finelog forwarding throughput 1.44x, from 284k to 409k rows per second in replay, by skipping oversized parent encodes and decoding zstd frames into a bounded buffer in #10271, and #10289 shrank the embedded server wheel 61%. #10573 stops compaction and index backfill from filling the object cache after the hub kept downloading about 0.8 GiB a minute for them, and #10550 emits JSON metrics only from process zero after Belay jobs produced 137 of 158 MB of sampled logs. Qamar merged the sync and async RL boards into one in #9715, and #10397 cut Grafana's Home page from 14.77 to 1.80 seconds by sharing Arrow results across panels and aligning live windows to 15-second boundaries. #10553 would let FineStore caches recover after lifecycle rules expire their shards.

Shellbox and ShellSim. Muchane's #10155 gives local bubblewrap machines copy-on-write system directories, so generated tasks that write under /opt run locally at no measurable per-command cost. His open #10506 keeps one sandbox per machine and joins it with setns per command: at 96 machines a node went from 127 to 2,066 commands per second, after 1,668 concurrent rollouts in the Taskforge RL run had queued behind namespace setup until deadlines expired. Power's #10640 builds on that with private service groups and prepared images for tau3-style tasks, and his #10628 would route all Harbor task and verifier execution through Shellbox, removing installed-agent paths that copied host model keys into guests. Feuer's #10332 prebuilds Daytona snapshots and retries throttled creates. #10578 publishes ShellSim's 138-package port catalog at public.oa.dev, so readers need no credentials or proxy.

Marina, Loom, and serving. #10383 keeps Echo chunk IDs stable across corpus rebuilds, which had been upserting up to 92k of 103k chunks hourly into shared Postgres. Loom gained a claude.ai connector in #10507 and a pm profile in #10321 that rewrites the goals document's status section every three hours and keeps the project board in line with issues. Sagar's draft #10543 adds paged Snowball inference across JAX hosts: on a four-host v4-32, Step12 scores 468 of 500 on MATH500 against 0.944 on H100, but at 14 tokens per second, 35x slower than vLLM on the same slice. Yon's #10645 runs Snowball performance grids through Iris coordinators for an InferenceX-style dashboard. On Discord Russell Power told Atula Tejaswi, whose newest DSpark drafter reaches 1.9x on RLVR prompts, that the team wants draft models in default RL runs once drafting holds across a task mix.

Data and Datakit. Himanshu Kriplani's #10310 truncates Harrier documents at read time, cutting the largest window from about 500 MB to 4 MB with byte-identical embeddings; his draft #9564 adds a Ray backend to Fray for Datakit on clusters without Iris, which Power said needs thought since Fray was meant as a migration shim away from Ray. #10541 found that the hero's decontamination Bloom never checked MMLU or GSM8K, because the Hub no longer resolves the un-namespaced gsm8k id and preparation skipped failures silently; the v4 corpus makes those tasks required. #10523 repairs orphan </think> tags in SWE-ZERO and daVinci-Dev that Lundgren traced in #10518 to the model's habit of closing a think block twice. #10395 adds JAX Harrier inference at 391 documents per second on one GB200. The Datakit canaries failed most days on placement, with 32 and 64 GiB workers pending past the CI deadline in #10336 and #10425; #10569 lowers the requests to 16 GiB. On Discord David Hall shared a gallery of agentic SFT traces showing bad behavior and proposed a filter.

Agent workflows and dependencies. Power consolidated the agent skill catalog from 42 entry points to 25 in #10462, and #10587 makes advisory reviews name their calling agent and documents that the mandatory pre-PR review may send diffs to either provider. Feuer's #10337 requires a responsible-use paragraph on every Hugging Face model card. Hall raised security floors for Brotli, Zod, DOMPurify, Pillow, and virtualenv in #10619, #10620, and #10627, and Feuer's open #10647 would audit all eight tracked locks nightly for affected Brotli releases and any LiteLLM package. The standup in #10649 adds what the PRs do not: Sartran lists the AMD dependency tangle, with jax-triton 0.3.1 troublesome on ROCm and AMD's fork diverging from it; Yon reports LMCache KV offloading raising throughput in early tests; Will Held released two tiny Snowflake SFT models for debugging RL tooling; and Ryan Williams deleted 60 TB staged by Hall and brought disky's CoreWeave scan indexes from 1.4 TB to 183 GB.

Two Discord regulars opened their first pull requests. boba shop cashier, who has posted kernel timings in #architecture since August, merged #10311, which computes cross-entropy gradients while the logits are still in memory, and has MXFP8 expert kernels for Blackwell under review in #10415 and #10559. @XiaoyuanWang111 turned last week's IFBench audit into a grader-audit skill #10422 and seven checker fixes #10423, and @raspberryice reported in #10320 that 1,142 Nemotron instruction-following rows reward all-caps or letterless answers. Michael Ryan's weekly sync on #2351 cleared last week's worry: every Qwen3-8B extraction for the Common Crawl curation ran on the correct torchax path, so only ReScraper hit the vLLM TPU RoPE bug, and the deduplicated pool stands at 9.95B documents, 17.15T Llama 3 tokens.

On Discord the task-curation threads carried the most substance. Franziska Weindel found that MiMo had re-uploaded its Docker environments with the Git-history leak largely fixed, added timestamp normalization to the task Dockerfiles, and noted Multi-SWE leaks the same way; four days later Franziska Weindel filtered Scale-SWE from 16,786 to 2,655 tasks by Qwen3-30B-A3B pass@8. Michael Siu compared pass@4 across Qwen, cold-start SFT, and Dr. Doom on that set and cautioned that dropping every task Qwen never solves would discard tasks Snowball can. Marianna Nezhurina saw RL reward climb on 32 TMax tasks but stay flat on 1k TMax and on calibforge, which she reads as tasks too hard for the compute without pass-rate filtering. In #mechinterp, @likelytobelaura offered to lead neuron and circuit hunting on the public Marin 32B checkpoints, and two days later @likelytobelaura posted a first neuron atlas.

Forty people joined and three introduced themselves: Alexander Cohen, a math professor at NYU who found Marin through CS336 and wants to work on pretraining or RL; Chanwoo, an MIT PhD student in multi-agent RL who has read from Marin 8B to the hero run and asked where to start; and Dunli Chan, a high-school student in Malaysia with NeurIPS workshop and ACL papers, now on model optimization and robotics manipulation at Tsinghua. Cohen's background intersects with the loss forecasting on #8435, and Chanwoo's with the RL loop in #10316.

Shared reading ran to new open releases and what their weights reveal: Meta's Muse Glimmer 30B, which Bilibird proposed as a teacher for task selection and whose weight RMS norm, Ahmed Ahmed relayed, sits near 6e-3, plus a paper Larry Dial flagged for a future run on pretraining sliding-window layers at very small context. Joy Jing closed RSVPs for the October 27 New York gathering on Marin.