Marin: Week of October 12th summary

The run changed context length this week. From the W&B history, it opened October 12 near step 245,645 and 11.33T tokens and stood at step 285,651 and 13.18T tokens at 16:15Z on October 18, about 40,000 steps and 1.85T tokens in the week. The 12T mark fell near step 260,092 at 12:16Z on October 14 and the 13T mark near step 281,768 at 01:01Z on October 18. Daily median MFU ran 29.1% on October 12 and 13 at 4K and 30.4–30.6% from October 14 at 16K, with median step time rising from 13.47 s to 13.84 s as the batch dropped from 11,264 to 2,816 sequences. Daily median train loss was 1.153 on October 12 and 1.150 on October 13; at 16K it read 1.099 on October 14 and 1.084 on October 18. Paloma macro loss moved from 2.119 at step 246,000 to 2.111 at step 255,000, the last 4K-packed eval, and UncheatableEval from 1.586 to 1.582. The first 16K-packed eval, at step 258,000, read 2.087 and 1.561; by step 285,000 on October 18 they were 2.074 and 1.548. Eval now packs at 16K, so the series step down at the switch and are not comparable across it, as the thread notes. W&B shows one gap over 20 minutes all week: 22:16Z on October 13 to 02:18Z on October 14, the cutover itself. The 16K run has not stopped since.

The cutover began as a pause. At 22:10Z on October 13 Matt Wittmann said on Discord the hero would stop for a few hours for a sweep over the QK multiplier for the 4K to 16K switch, #9615. mcwitt-agent cancelled hero-mfu30-step231k at 22:34Z after a requested permanent checkpoint, step-257553, committed at 22:22Z; four 100-step arms, at qk_mult 1.3, 1.48 and 1.58 on all layers and 1.68 on the global layers only, then ran one at a time on the eleven racks from permanent step-252000, with the hero due to resume at 4K afterward. The arms' step-252,100 eval losses fell within eval noise, and Larry Dial chose 1.48 on all layers; the engineering section has the reasoning. At 01:33Z on October 14 the plan changed: the hero would stay paused and the 16K switch would hand off from step-257553. The launch controls had merged the day before in #9689, with #9890 and #10086; the engineering is in the sibling section. The deploy, #10101, merged at 02:06Z and Matt Wittmann requested the launch at 02:10Z: run hero-ctx16k-step257k, commit eadf76a1bc, forking the W&B parent at step 257,552 and restoring step-257553 on cw-us-east-08a's eleven NVL72 racks at 16K context, qk_mult 1.48 and batch 2,816, with 4,194,304 tokens per rack per update and the 390,251-step schedule unchanged. The parent stopped at the handoff, so there was no same-batch control window; gates were agreed instead. The GO came at 04:08Z: first logged step 257,553 on all 704 ranks, cross-entropy at most 1.132 over the first 10 steps and a 1.105 median over the last 20, median MFU 30.7%, drop fraction 1.21e-3, peak memory 123.42 GiB of a 143.75 GiB limit, no retries, and the first 16K eval at step 258,000 done in 72 s. From the parent's last step to the child's first was about four hours, in line with the 3.5–4 hours estimated for the sweep alone. Rollback remains commit f2bc69bf6c under the old run from step-257553. Two findings from the independent review of the deploy are filed: #10100, where the vibe-check sampler would stamp pre-switch checkpoints with qk_mult 1.48 if its release were bumped, and #10024, merged October 14, which bounds W&B host copies after a one-rack hero restore grew each process from 1 to about 1,700 py_xla_callback threads in 20 minutes and made py-spy dumps 3.5x slower.

Checkpoint storage was thinned. #10218 by Larry Dial, merged 18:35Z on October 17, replaced the hero-* deletion deny on marin-us-east-02a with 39 path globs so 17 superseded permanent checkpoints across six finished phases became deletable, 70.3 TiB in 552,013 objects. Each phase keeps every multiple of 12,000 steps, step-6000, step-270000, and the checkpoints on either side of four changes: weight decay at 54,000/60,000, the ragged all-to-all at 78,000/84,000, the data mix at 108,000/114,000, and the 16K context at 257,553/258,000. On Discord, Larry Dial described it as deleting every other checkpoint while keeping the ones around stage transitions. The bucket has versioning suspended, so the deletion is permanent. #10245, merged 23:08Z the same day, reports the 17 gone and the 30 retained permanent checkpoints intact, and restores the full wildcard deny. #10221 by Russell Power, still open, routes CoreWeave scratch writes to a separate bucket and keeps the hero's rolling checkpoints and resume searches in hero-checkpoints. mcwitt-agent opened #10180 on October 15: six checkpoints from the original run at steps 6,000 through 54,000 store a router bias equal to the bias derived from their own pending quantile-balancing betas, which a consistent step-N checkpoint cannot do, so they may mix state from two steps, perhaps through the async save path under pinned-host fp32 master params; every checkpoint from step 114,000 on behaves as expected, and the cause is unverified. Checkpoint completions moved too: #10224 raises the sampling temperature from 0.2 to 0.7 from step 270,000 after 0.2 gave repetitive responses, and Larry Dial posted the temperature-1.0 samples from that step on Discord.

Alert runbooks caught up with the September stops. #10049 merged October 13 with the corrected telemetry-gap and retry guidance described last week, and #10257, opened October 18, adds that a retryable Finelog query failure yields non-firing rows for hero alerts while FinelogFleetUnhealthy reports the shared failure, and separates healthy from warming_up in the loss-spike runbook. The coverage gap in #10048 stays open. In #long-context, Benjamin Feuer relayed an offer of outside compute to cool down an intermediate checkpoint without stopping the run; Will Held judged that moving the data off CoreWeave and adapting to InfiniBand would take about a week, more than the gain, and Larry Dial put the pretrain at about 22 days left and already at 0.3 of peak LR, which points to about November 7. The logbook posted no finish date; this page's own estimate from W&B is that with 104,600 of 390,251 steps left after 285,651, at the 16K build's roughly 6,130 steps a day the run would finish near November 4, the same date as last week's estimate, with the cutover pause and the slower 16K step roughly cancelling the week's gain.

The three-PR context-switch stack merged on October 13, with a fourth PR added the same day. #9888 shuffles Harrier in blocks of 2^20 tokens and adds prior_context_phases, so each source resumes where it left off; without it a 16K relaunch would have re-read about 2.85T of the remaining 9T tokens. #9890 adds the FLOPs baseline that keeps cumulative throughput continuous across the switch. #9689 answers last week's review note about a second switch: the hero's context history is now checked in as HERO_CONTEXT_PHASES, a deploy edits Python only, trigger_hero.sh is unchanged, and the launcher holds 4,194,304 tokens per rack per update so the 390,251-step schedule and learning rates stay fixed as the sequence batch shrinks. #10086 adds qk_mult_long_scale, a second query multiplier applied only on the global full-causal layers. Q and K are RMS-normalized, so the multiplier sets the softmax temperature, and since the local layers keep their 2,048-token window at any context, the YaRN-style value 1.3 × (0.1·ln 4 + 1) = 1.48 is only clearly motivated for the global layers.

The choice of multiplier was argued in #9615 before the sweep ran. Qirun Li showed that the 67B's QK arms had nearly tied and that PTB and WikiText carry 77% of the hero's Paloma macro eval-to-eval variance, and proposed keeping 1.48 unless another arm beat it on both Uncheatable and the Paloma macro without those two subsets. Larry Dial agreed on 1.48 as the default while declining to automate the call, and plotted the 67B's 20-step probe, where scaling only the long layers had the lowest training loss. A Claude agent supervised by Matt Wittmann posted the sweep design: same-seed 100-step arms from one checkpoint, so losses compare step by step on shared batches. Matt Wittmann paused the hero for the sweep on October 13. Per #10101, the arms restored permanent step 252,000 on all 11 racks and the 16K dropless eval loss after 100 steps was 1.81349 at 1.3, 1.81156 at 1.48, 1.81200 at 1.58, and 1.81282 at 1.68 on the global layers only, within eval noise. Qirun Li read the same arms paired per step and found clearer separation: 1.58 tied 1.48, while 1.3 and the global-only 1.68 trailed on every step after step 20, a gap the 67B's 20-step probe could not have shown. Larry Dial chose 1.48 on all layers, noting the field is insensitive at a 4× extension. #10101 then forked hero-ctx16k-step257k from step 257,553 with the sequence batch at 2,816 instead of 11,264 and a FLOPs baseline of 1.779e24; the gate record shows 30.7% median MFU and 123.42 GiB peak memory. The arms also exercised the 16K dropless eval on GPUs for the first time. Two loose ends: #10100 notes the vibe-check sampler would stamp pre-switch checkpoints with the current qk_mult if the store were rebuilt, and #10024 merged, bounding the W&B host copies whose threads had grown to about 1,700 per process.

Work for the 262K phase, which needs context parallelism (CP) of at least four, started on one-rack restores from step 240,000. #10170 merged on October 16: a contiguous split of a causal sequence gives the shard holding a document's end far more attention work than the one holding its start, so the sharded FA4 wrapper now swaps half of each shard's queries with its mirror shard in a zigzag. At 262K on CP4 and expert parallelism (EP) 16, tokens per second rose 27.8%, step time fell from 29.66 to 23.22 s and its standard deviation from 3.19 to 1.39 s, at +0.8 GiB and a loss 4.6e-4 rms from main. An earlier version stored the whole sequence in zigzag order (+477 lines in 10 files against +109 in 2) and ran within 0.2% of the final one. Amy Sartran approved and asked about coverage at 64K to 130K; Matt Wittmann raised the kernel parity test from 512 to 2,048 tokens. Three draft PRs follow. #10145 flattens MLP tokens locally under CP, removing four of eight all-to-alls per layer; at 65K on CP4/EP16 the step went from 15.938 to 15.586 s, the drop fraction from 1.17% to 0.23%, and CP4 became faster than CP1, though CP8 at 262K still runs out of memory on NCCL buffers. #10121 deals packed rows across the 64 batch shards in snake order, cutting the busiest shard's global-attention work from 2.10× to 1.17× the mean at 16K (13.131 to 13.055 s per step). #10122 evaluates the FA4 SM100 segment mask 32 columns at a time, taking the local-layer forward from 3.16 to 1.96 ms against upstream's 1.87 ms, 13.055 to 12.956 s per step on the rack; the remaining 0.035 s per step is in the one-CTA backward. Together the two gave +1.66% on a second rack. #10232, found by Karan Sagar in a Snowball sweep, shows held-out eval still fails when tokens are context-sharded; #10233 is the open fix.

Amy Sartran's MoE kernel work moved to NVIDIA: #10098 runs the tile-map ragged_dot kernels on Hopper and data-center Blackwell, faster than group-grid in 54 of 54 GB200 cases (1.03 to 2.47×) and 49 of 54 on H100, worth 3.9% on an eight-layer June model; #9894 adds the benchmark behind those numbers, #10099 lets callers pick a family explicitly, and #10217 teaches the AMD Pallas flash kernel to read the hero's precomputed window bounds. #7934, open since August, merged and stops the base MoE template's MFU counting sliding-window layers as full attention. On checkpoint consistency, #10180 reports that the original run's checkpoints from steps 6,000 to 54,000 store a router bias equal to the one derived from their own pending quantile-balancing betas, to 4.8e-7, which a single-step snapshot cannot do; every checkpoint from step 114,000 on differs as expected, and the cause is unverified. #10174 makes inference and export use the stored bias rather than the pending estimate, matching training-time eval, and #10175 makes the reference and XLA attention backends honor the local-window bounds they had silently ignored. On data, Mark Muchane's #8303 merged, registering the hero's quality scores as Datakit steps after a census of all 162,535 shards across 292 sources confirmed row order, and Rafal Wojdyla's #10108 proposes a quality-classifier fast track that scores a raw pool and trains the ladder on the top documents.

Benjamin Feuer published the October eval policy #10197, which Benjamin Feuer called the canonical policy for the Snowball release, and under it the results that pick the release candidate #10199. An uncertainty-aware tournament scored 21 Snowball and Grug checkpoints on 23 benchmarks, counting a benchmark as a win only when the paired difference's adjusted 95% interval excludes zero. The Campaign 3 merge DARE-Linear-02 ranked first with a mean paired win rate of 62.1% (144 wins, 33 losses, 283 ties) and the Campaign 2 merge DARE-TIES-02 second at 62.0%; head to head they split 3-3-17, the winner ahead on MBPP+ (68.8 against 58.7), SWE-bench Verified (30.9 against 22.6) and BixBench (17.6 against 10.3), the runner-up ahead on GSM8K (89.3 against 85.2), MRCR (20.0 against 17.3) and τ³-bench (11.2 against 5.0) top-two card. The winner is now called Snowball v0. It came from Campaign 3 in #9975, which grafted the allkimi relay-SFT specialist into DARE-TIES-02 under a rule that at least 75% of base scalars stay bit for bit: of 32 merges, DARE-linear-02 raised SWE-Bench-100 from 27.27 to 33.00 and Terminal-Bench 2.1 from 9.92 to 13.36 while changing 2.36% of scalars. Campaign 4 then asked whether a narrow edit to Antidoom step 12 could recover tool calling from Reference-KL step 20. Across 71 audited merges that touch only routed-expert rows, adaptive-delta-08 raised the sum of HumanEval+, MBPP+, OlympiadBench and Berkeley Function Calling Leaderboard (BFCL) Parity from 174.90 to 220.40 (BFCL-Parity 28.46 against 5.70) while changing 9.47% of scalars, with MATH500 at 93.60 against 95.00; these are single runs with one retention canary. The merging library landed in #9974 and the Campaign 4 methods in #10135, and #10039 fixed a Hugging Face mirror that had declared a cache hit with 28 of 39 DARE-TIES-02 shards present. On #9976, Colin Raffel wrote that merging looks on par with on-policy distillation once mixing coefficients are tuned and pointed to Marco Ciccone's forthcoming coefficient-tuning method; Feuer replied that merges can look strong under the policy and still acquire behavioral tics, and that he is hardening the policy against that. From a HallBench comparison of 2,000-plus GLM-simulated long-horizon chats between v0 and Qwen 3.6 35B A3B, Benjamin Feuer named nuanced instruction following, factuality, perseverance and complex tool use as the weak spots.

Karan Sagar preregistered an agentic SFT shootout on Snowball v0 in #10213: plain SFT, rejection-sampled SFT that resamples across epochs, relay SFT #10027, recovery SFT and plateau dropping #10028, all from one base, one task pool of about 2,000 TMax environments and 2,100 SWE-smith tasks, one teacher (GLM 5.3, eight attempts per task) and one supervised-token budget, with the learning rate chosen by a sweep whose decision rule was fixed in advance. An arm passes only if it shows no decisive loss against plain SFT on GSM8K, BFCL-Parity and GPQA-Diamond and a decisive win on SWE-bench Verified, Terminal-Bench 2.1 or TBLite 2.0 under the policy's rule; one policy run resolves gaps of roughly 4 to 5 points on SWE-bench and 8 to 10 on the terminal benchmarks. Building the pool turned up answer leaks. The SWE-smith Harbor packages leave the repository's git history in the agent's container, so a diff against main prints the fix #10212: 860 of 5,237 passing Kimi-2.5 trials (16.4%) read history and 94% of the testable ones saw a line of the fix, as did 491 of the 2,665 passing relay trials in the allkimi data that Campaign 3 grafted into v0. mini-swe-agent's own tool environment and the base images also hold clean copies of eight task repositories #10223, and seven paramiko packages have verifiers that score the unpatched tree 1. Sagar rebuilt the packages with a single-commit repository and verifier-side test restoration, cut the agent-phase sandbox off from every host except the model proxy, dropped the eight repositories and 181 TMax environments with hidden verifier assets, and froze pool v5 at 2,015 TMax and 2,100 SWE-smith tasks, public as Snowball-SFT-Shootout-Pool-2026.10. GLM 5.3 solved TMax at pass@1 0.77 in a 200-task pilot, with 12.8% of conversations over 32k tokens. Plain generation finished on October 18 with a budget of 28.93M supervised tokens over 2,880 eligible tasks, 4% under the planned 30M floor because never-solved tasks contribute nothing; relay generation continues after a preempted student server silently failed 198 of 200 trials in one round #10237. The code is a draft stack: pool builder #10258, sweep data #10252, trace driver #10251, row builder #10253, verification #10254, a recovery arm that inserts masked failed attempts into 20% of rows #10259, and plateau dropping as a Levanter train_lm option that drops a token once two consecutive relative loss changes fall under 10% and resumes bit for bit #10255. #10247 and #10248 preserve the finished CruxEval and two-expert transfer experiments.

Sagar also measured what the expert-transfer arms did to the weights in #9920 and #9984. No arm copies the expert's change. The CruxEval expert moved 0.036% of the student's weight norm, mostly in attention, the router and the LM head; plain SFT on its traces moved 0.123% and touched 41% of weights, rejection-sampled SFT 0.094% and 35%, and on-policy distillation 0.035% and 20%. Cosines between each arm's delta and the expert's sit at 0.05 to 0.12 in attention and the expert MLPs; only the LM head lines up, at 0.68 and 0.71 for the SFT arms and −0.32 for distillation. Four different weight changes all recover the expert's score, so transfer re-learns the behavior and does not recover the parameters. Feuer read the smaller, sparser distillation update as the safer one for task interference; Sagar noted its direction is still nearly orthogonal to the expert's and that the retained tasks did not regress. David Hall said merging experts trained under conflicting harnesses could produce the kind of pathologies Feuer worries about. Marianna Nezhurina ran her 40M-token recovery SFT mix on Antidoom step 12 and found it slightly ahead of the same mix on Grug 09.21, with submitting after visibly failed tests the main remaining failure; the checkpoint is public. Harshit Varma reported 18.2 on Terminal-Bench 2 after 20 steps of agentic RL from a plateau-dropped SFT checkpoint of the 09.21 Snowball, with runs on the new base queued; the SFT gains repeat there except on SWE-bench Verified, where Antidoom is already strong. Karan Sagar asked both for their failure-injection and plateau-dropping code so the shootout's phase-2 arms match the originals.

Feuer closed the structure-closure campaign #10079: joint training on JSON, tool-delimiter, reasoning-block and code-fence closure, using final-token preference optimization (FTPO) plus supervised loss on verified repairs, lifted BFCL-Parity under Pi from 30.76% to 36.04% over six runs per model, met the 75% syntax-failure-reduction target on four of six non-tool families and missed it on nested JSON and strings; MATH500 fell from 92.33 to 91.17 and HumanEval+ rose from 66.87 to 72.56, with code in #10078. The cost of that lineage is now measured in #10040: Antidoom step 12 scores 31.1 on SWE-bench Verified, 20.9 on OT-TBLite and 10.9 on Terminal-Bench 2.1 against 6.00, 1.00 and 0.000 for BFCL-Recovery and 6.01, 3.47 and 0.820 for Continuation-Tool-Retention. The loss is already present at BFCL-Recovery, and a same-pin comparison would be needed to blame a particular DPO update. On #10043 Feuer handed his Smartie follow-on RL run, async RLOO with truncated importance sampling from DARE-TIES-02 on 112 H100s, to David Hall and David Lundgren; its native scored-only reward rose from 41.14% to 46.81% by step 4, but none of the 90 Harbor validation tasks has scored because the endpoint requires exact token IDs on retry, so no BFCL, SWE or terminal gain is established. Lundgren's #10214 fixes in advance three matched mixed-domain RL with verifiable rewards (RLVR) arms from the October 5 Datakit SFT to test whether PivotRL pivots at 20% of prompts lift SWE-bench Verified and OT-TBLite, with datasets in #10215 and curation in #10181; a smoke run is in progress. The SWE-bench pre-registration #8933 gained a second forecast for Nemotron 3.5 Lightning data: 0.1334 bpb and 40.3% pass@1 on SWE-bench Verified for the Snowball base fine-tuned on 7.55B Lightning tokens. Two serving results bear on candidate evals. Sagar found TPU serving of step 12 diverges from H100 past the 2,048-token sliding window, with MATH500 at 0.944 and 0.950 on H100 against 0.878 to 0.884 on TPU #10250, and Romain Yon refreshed the Snowball GPU reference scores after rounding barriers changed three next-token choices #10069 and passed all 64 parity prompts on TPU vLLM in #9400. Atula Tejaswi posted DSpark drafters for Snowball that decode 1.8 to 2.2 times faster at concurrency 16 #10227, and Hall closed his inference draft PRs, among them the EAGLE drafts #9680, #9682 and #9685 to focus inference work on exact training and serving numerics.

RL task stack. RolloutEngine, on main since last week, grew the surface SkyRL and Taskforge need. A Claude agent supervised by Rafal Wojdyla landed an observer that reports each step of an attempt and carries the conversation in its events in #10139 and #10177, so a crash no longer loses the transcript; a per-attempt token budget in #10143; and output files returned to the caller and grader in #10140. #10142 adds a messages trace mode for chat endpoints, which re-tokenize every request: behind the Orion router, 2 of 7 GLM-5.3 ShellSim rollouts had aborted on one non-canonical token. A spill mode that kept long command output in the task machine merged in #10144 and was reverted the same day in #10184: paying for unbounded output is what teaches a model to redirect it to a file. The open #10203 replaces task sessions with registered harnesses that own the agent loop. Russell Power brought real CPython and NumPy into ShellSim's WebAssembly sandboxes with one PackageSpec for bubblewrap and ShellSim machines in #10071, and grader environments now declare pinned requirements, files, and env in #10104, which deleted the 16,000-line checked-in lock, with Dockerfile fragments for toolchains such as Lean in #10088. #10159 uses ShellSim 0.1.38 cancellation tokens to interrupt a timed-out action instead of closing the machine, after one slow computation cost 58 Taskforge trials in two hours.

Taskforge. Mark Muchane's RL environment synthesizer began merging in pieces cut from the 11,000-line draft #9993, kept open as the reference for review. #10068 adds the GLM-5.3 client and an experiment that prompts GLM with a capability, runs the program it writes in Shellbox, and keeps the TaskSpec it emits; #10081 adds the author and build stages as a memoized step DAG; #10091 stops repairing a reply cut at the 131k output limit, since replaying it had built a 175k-token prompt. The stage coordinator in #10087 is under review. On Discord Mark Muchane shared about 750 generated math, physics, chemistry, and biology tasks from a run of about 2,000, and #10125 would register the accepted set as eight Atlas sources, with too-easy tasks kept apart so RL mixes can exclude them. The run found the cluster's limits: #10111 records the cw-rno2a controller saturating at about 2,000 live sandboxes, since every Shellbox command is an ExecInContainer RPC through it. #10118 cuts that load, parsing a 28 MB pod list in 41 ms instead of 296, and #10119 proposes moving sandbox exec off the controller; bubblewrap already handles Python-only environments without it.

RL data and training infra. Power's curation pipeline closed its loop with the Atlas. #10055 validated the recipes for about 150 datasets; #10075 samples each source, up to 100 retained tasks and 20 excluded, and gets one structured GLM judgment per item; #10157 embeds tasks and curriculum entries with the same pinned Harrier model and maps coverage; and #10208 publishes a stable catalog of 206 source declarations, 146 assessments, and 12,107 judgments, which #10205 shows as a By source table and a By curriculum heatmap. Benjamin Feuer found the move to TaskSpec had dropped most of the Nemotron Ultra graders and restored them in #10077 and #10080, giving all eight ungraded components of RLVR1 (RL with verifiable rewards) executable graders over 50,544 rows. New sources: Gonzalo Benegas pinned Skill2Env's 7,496-task Harbor Hub release in #10120, Tim O'Donnell's #9881 registers PDBThink's 100,000 coordinate-reasoning tasks, and David Lundgren's #10181 curates PivotRL pivot turns with in-process graders. #9911 switched SkyRL's recipes to the TaskSession runtime, with the async canary passing on four H100s. Karan Sagar found in #10212 that SWE-smith's Harbor packages leave 944 git branches, including the unbroken main, in the agent's container, and 16.4% of passing Kimi-2.5 trials had read them; #10223 adds that clean copies of eight task repositories sit outside the testbed, three in mini-swe-agent's own install and five in the base images.

Iris, access, and storage. Romain Yon's #10070 makes Iris resolve opaque principal IDs from a private directory at controller start, so access policy stays in Git without emails; #9846 is the first personal Batch grant on it, and David Hall admitted a new researcher to the H100 and GB200 clusters in #10056 and #10186. Will Held showed in #10082 that garbage collection counted its one-hour retention from pod creation, so a long task could lose its pod before the controller saw it succeed and be retried as a worker failure; #10083 counts from the last container's finish. Sagar found in #10240 that every GPU job defaulted to NCCL_DEBUG=INFO: four 32-GPU SFT jobs wrote about 80,000 log lines per second against a forwarder that accepts 34,500, and the hub fell 90 minutes behind. #10241 defaults to warnings and #10238 raises forwarding throughput.

FineStore, Zephyr, and Datakit. Power rebuilt FineStore's write and compaction paths. #10209 adds a simulated S3 backend to count requests, uploads shards straight to their final key, elects one compactor per archive, and brings a one-row flush from eight requests to three. #10188 bounds compaction to a 400 MiB Arrow accumulator after Power rejected a draft that disabled prefetch; #10216 makes compaction a sorted-run merge that resolves key versions in Arrow instead of a Python dictionary; #10226 prefetches small inputs concurrently, 5.76 to 0.85 seconds in a 17-shard benchmark. In Zephyr, #9995 adds opt-in intra-shard map execution so idle workers can split a shard, kept off by default after Sagar worried about cutting inputs too finely, and #10167 shows per-shard input progress in the dashboard. The Datakit canaries kept failing on placement: the store stage's 32 GB workers cannot fit the 16 GB us-west4 fallback in #10064 and #10149, and a Luxical model archive under a one-day TTL vanished mid-embedding in #10150.

Evals. Feuer published the Marin eval policy in #10197, superseding the September drafts and setting what counts as a reportable score for Snowball and its baselines; on Discord Benjamin Feuer noted that infrastructure error rates are low while over 15% of τ³-bench jobs on Pi failed, and presented David Hall's conversation lab, where GLM plays user and judge across 2,000 long chats, as an informal benchmark. A Stanford undergraduate, Xiaoyuan Wang, audited the IFBench checkers behind #10060: 37 of 58 instruction types have at least one defect, moving strict accuracy by −1.7 to +4.7 points across 16 open models. The runner changed shape: #9813 runs each evaluation as a StepSpec with its own lock and cache, #10222 awaits inference asynchronously, adds a /status page, and runs several serving instances behind one broker. Held's #10169, resuming after inference preemption, overlaps Sagar's relay in #10033; they agreed to pick one. EvalDash fixed displays that misrepresented stored results: #10151 restores per-sample grades where a GPQA Diamond run reported 43.1% with all 594 responses shown ungraded, #10192 shows the GSM8K reference value instead of the full rationale. Larry Dial's #10179 scored the 67B-A2B 65k checkpoint at 0.7812 Paloma macro BPB against 0.8397 for Qwen3.5-35B-A3B-Base, lower on 15 of 16 subsets.

Inference serving. Yon's #9746 settles how GPU vLLM wheels reach Marin: the fork builds and qualifies from reviewed main, then one adoption PR checks Snowball's token probabilities against stored references. The first such release, vLLM 0.30 on Torch 2.14.1 and CUDA 13.2, passed its H100 and GB200 gates and waits in draft #10022. #10069 refreshed the saved Snowball GPU scores, since the compiler barriers that make inference round like training had changed three next-token picks, and #9400 now passes all 64 prompts of TPU parity on a v6e-8. Sagar found the production problem behind that work in #10250: Snowball Step12 on TPU diverges from H100 from token 2048 onward, the sliding-window size, and MATH500 drops from 0.944 and 0.950 on H100 to 0.878 to 0.884 on TPU. Hall closed his nine EAGLE and GPU paged-decode drafts, #9656 and #9675 through #9685, to focus on exact training-and-serving numerics; the branches are kept. Drafting continues elsewhere: Atula Tejaswi trained two 280M-parameter DSpark drafters for Snowball and reports in #10227 that the one trained on the SFT mix's replies reaches 2.16x tokens per second over plain decoding, though acceptance after an RL weight sync is unconfirmed.

Kernels and AMD. Amy Sartran's MI350X stack merged. The gather-based ring combine in #9860 is now the default MoE implementation on GPUs after she measured it a small win on H100 and a loss on TPU, which keeps ring; #10009 adds tile-map grouped-GEMM kernels for ragged_dot on AMD and #10098 turns them on for H100 and data-center Blackwell, where they ran up to 2.5x faster than the old kernels; #10014 is the Pallas-Triton flash attention, landed because any jax_triton kernel still hangs multi-GPU ROCm training. On #9812 two nodes keep 78 to 83% of one node's per-GPU throughput under data parallelism and 69 to 77% under expert parallelism or Fully Sharded Data Parallel (FSDP), and a standalone measurement explains why the Triton kernels matter: hipBLASLt has no bf16 grouped GEMM for gfx950, so XLA's default reaches 35.6 TFLOP/s on the June shapes against 784 for Triton. Sartran's standup notes list the blockers: Pallas is deprecated in the JAX version Marin runs, MI300X nodes are scarce, MI350X nodes throttle under poor cooling, and command buffers must stay off.

Marina, Loom, and review. Power rebuilt Marina's release path: #10137 releases every application from versioned packages in Postgres so a runtime restart no longer republishes apps, #10132 replaces the row of app links with a searchable drawer, and #10178 adds a persistent Python playground with project storage and MCP access. Agent review of PRs gained a second lane: #10127 adds a whole-change holistic review that asks what a hunk-scoped review cannot, #10133 bounds it to one initial run and one follow-up per PR, and #10211 updates the overview in place instead of posting a new one on every push.

Levanter, data, and research. Matt Wittmann's #10024 fixed a W&B logging change that had grown each hero training process from 1 to about 1,700 host-copy threads in 20 minutes, since PJRT's work queue adds threads and never retires them; copies are now bounded at eight in flight. Feuer's #10039 stops a cache marker in a Hugging Face repo from making a 28-of-39-shard mirror look complete. Sagar's draft #10255 adds plateau dropping to train_lm, removing a token from the loss once its loss stops moving. Kaiyue Wen tested the copy-paste language model head on the July MoE recipe in #10229, with a first d512 run improving 23-domain macro NLL from 4.579 to 4.092 at 1,000 updates; the draft loss is #10230. Ben Kolligs opened #10076 with SigLIP encoders and image-text data loading, the first step toward a Delphi-based vision model in #9524. Marin Belay in #9451 accepted a new baseline checkpoint and began testing a MuonH momentum warmup suggested through the intake issue, where Qirun Li paired 404 seed-twin runs to argue for reporting the Paloma macro without ptb and wikitext_103. Jeff Hammerbacher proposed offline web search in #10189 and text-interface computer use in #10231, neither of which Marin has trained for. The standup in #10148 adds items the PRs do not show: Dial reports 28% MFU at hero scale on one rack with attention residuals, FP8, MLA, and KDA; Ryan Williams deleted more than 300 TB of storage marked for sweeping; and Held, back from COLM, asked for a set of first issues on environment creation so interested PhD students have somewhere to start. About forty issues from May through July were closed in a backlog sweep.

Outside contributors worked mostly on plumbing. Himanshu Kriplani merged #10073, one aggregation in place of a per-file loop in Datakit's cluster planner, and drafted #10106 and #10074. Bryce Lin fixed #9065, a Levanter launch failure @prashant-rathod had reported, in the one-line #10084. Kostiantyn Dementiev offered a tested fix for #7889, Iris's lost ConfigMap cleanup, with an orphan sweep, since evicted and force-deleted pods never enter the cleanup queue. Tim O'Donnell opened the EvalDash grade repair #10151. Jenna Russell reported in #9073 that AI-generated text in the Uncheatable eval has the highest macro loss throughout pretraining and no sign it masks loss on human text; Ricardo Olmedo pre-registered the 40.3% SWE-bench forecast in #8933; and Michael Ryan's notes on #2351 trace most of his TPU-versus-GPU ReScraper gap to a RoPE bug in the vLLM TPU build.

On Discord, Percy Liang announced a community-meeting talk by Rohith Kuditipudi on hinted scaling laws, for forecasting accuracy from small models that score zero; slides and recording are in the follow-up. Charlie R asked for a comparison of the hero run with open-weight base checkpoints, #8824; Larry Dial ran the bits-per-byte numbers in #10179 and cautioned that a 2B-active, very sparse model may look better on perplexity than it generalizes, while Will Held noted Qwen 3 wins on arXiv and code and loses on news-like English. Zoey Li read through the RL with verifiable rewards (RLVR) campaign #9359 and found that several answer checkers reject Snowball's output format and that rewards are not normalized per task, so the language-mixing reward runs to 5.5 against 0 to 1 elsewhere.

Thirty-seven people joined and six introduced themselves: Abhijith Neil Abraham, a founder in LLM inference and vLLM contributor, which intersects with Marin's vLLM fork; Saket, a UCSB postdoc in reinforcement learning for robots; Gideon, who builds an atom-level multimodal model at Manas AI; Joey, a Berkeley PhD student in the Learning Mechanics group, who asked in #mechinterp what open checkpoints have shown so far; Open Athena's Karan Sagar; and Qirun Li, a former quant trader who pretrains small models and wrote his own Group Relative Policy Optimization (GRPO) trainer. jjmachan of Vibrant Labs, an RL-environment company, offered help on Taskforge.

Shared reading ran to pretraining recipes: Microsoft AI's frontier pretraining post, Meta's Muse Spark Open, whose post-expert norm Larry Dial and Kaiyue Wen discussed, a batch-size-and-optimizer paper Will Held posted, and David Hall's essay arguing there is no post-training data commons. Open Athena announced an October 27 New York evening on scaling Marin.