Marin: Week of October 5th summary

The run passed 10T tokens on October 7 and changed builds on October 9. From the W&B history, it opened October 5 near step 205,300 and 9.47T tokens and stood at step 244,208 and 11.27T tokens at 18:27Z on October 11, about 38,900 steps and 1.79T tokens in the week. The 10T mark fell near step 216,750 at about 01:17Z on October 7. Daily median train loss fell from 1.176 on October 5 to 1.155 on October 11. Daily median MFU held at 26.3–26.5% through October 9 on the old build and ran at 29.3% on October 10 and 29.2% on October 11 on the new one. Paloma macro loss fell from 2.157 at step 204,000 to 2.140 at step 231,000, the old build's last eval, and to 2.118 at step 243,000 on October 11; UncheatableEval macro loss fell from 1.608 to 1.589 over the same span. On Discord, Will Held pointed to Larry Dial's investigation in #9874 of a one-eval drop in Paloma's Penn Treebank loss; a rebuild of the training stream found PTB's own 6-gram files, inside a GitHub repository dump, read about 500 steps before the step-210,000 checkpoint.

One stop this week. On October 6, Kubernetes deleted task 16 on node s14fys64 under a NoExecute taint, and Iris requeued the 176-task gang. loom-oa-dev reported all tasks running and training under way by 21:00Z; the condition behind the taint is unconfirmed, and the incident record is at marina wiki 641. W&B logged no new step between 215,855 at 20:33Z and 215,856 at 21:25Z, 52 minutes.

The planned handoff to the MoE optimization stack happened on October 9. #9958 merged at 00:49Z and dropped two memory overrides from the launch script: with them, a hero launched from main would have compiled with host-memory offloading at a 0.75 memory fraction and slop factor 85, a combination that aborts compilation on every rank in HloRematerialization on a one-rack restore of the step-180,000 checkpoint. #9960 merged at 16:21Z with the handoff itself: one GB200 NVL72 rack restoring that checkpoint trained 4K sequences at 12.70 s/step against 13.97 s/step before, 30.9% against 28.1% MFU, and a d768 validation tracked its September baseline over 11,420 steps. At 16:26Z Matt Wittmann announced the deploy on Discord as good for about 9% more throughput. mcwitt-agent requested the launch at 16:53Z: run hero-mfu30-step231k, commit f2bc69bf6c, on cw-us-east-08a's eleven NVL72 racks. The old run was cancelled at step 231,664 after a 200-step window past the handoff checkpoint, step 231,464, which was forced on it, marked permanent and copied to the hero-checkpoints bucket; the new W&B run forks from the old one at that step. The GO came at 17:53Z after the new run trained the same 200 batches, steps 231,464–231,663: median MFU 29.44% against 26.32%, median step time 13.33 s against 14.91 s, median iteration 13.43 s against 15.14 s, peak HBM 123.4 GiB against 120.8 GiB. Mean training-loss difference was −8.5e-5 with a largest gap of 4.2e-4, and router z-loss, load-balancing loss and routing entropy stayed within 0.5% of the old run. Single-rack tests cannot exercise the eleven-rack collectives, so this trial was the first multi-rack test of the stack. From the old run's last step at about 16:50Z to the new run passing it, the cutover cost about an hour of wall clock, replay included. The GO left two things untested: the run's first eval and a resume from its own checkpoint. The first eval, at step 234,000, ran at 02:58Z on October 10 and reported Paloma 2.134 and UncheatableEval 1.593, in line with the trend. #9990, opened the same day, adds a preflight to the cutover checklist: a one-rack run of the deploy commit should train through a dropless eval and the steps after it, since the 200-step trial ends before the first eval and the switch between eval and train executables has failed before in #8861; it also notes the public hero report showed the new run's whole inherited history until its report_phase_start was set to 231,464. In #infra, Matt Wittmann pointed Furkan to #9829–#9834 as the deployed round and to #9378 for the round before.

Alerting work followed the September stops. #10048, opened October 11 by loom-oa-dev, records that a failed attempt followed by an Iris retry can pass without a critical Grafana alert: TrainingProgressStalled resets its clock for the new execution and allows 45 minutes to initialize, TrainingTelemetryGone stops firing once the retry publishes phase=initializing, and the warning-only task_retried check needs fresh telemetry and an event from the last 15 minutes. The September 30 checkpoint failure resumed about 27 minutes after its last step and neither critical rule fired. #10049 corrects the runbook to match: an empty 30-minute telemetry query can classify an active run as silent, operators should use the 24-hour phase window the alert bridge uses, and health warnings group by alert name and cluster while critical hero alerts group by run. #9745 merged October 5 so the hourly hero-completions workflow keeps two sampling jobs active instead of one; a sample set takes about 40 hours, mostly queue wait for 64 GB200s at batch priority, and steps 156,000–180,000 had no attempts. The context-length switch did not happen this week: #9689 stays open, and #9890 would fix the held-out eval at 4K and carry the parent's FLOPs count across a context change so the eval and MFU series stay comparable. The run stays at 4K. The logbook posted no finish date; what follows is this page's own estimate from W&B. With 146,043 of 390,251 steps left after 244,208, at the new build's first full day of about 6,190 steps, the run would finish near November 4; at the clean 13.33 s step time it would finish near November 3, four days earlier than last week's estimate.

Last week's one-rack campaign reached the production run. On October 6 David Hall asked Matt Wittmann to split #9708 into about eight PRs, and it became #9829 plus the stack #9830 through #9836, with the short convolution and router top-k back as raw Triton kernels, faster than the Pallas ports. Six merged on October 7 and 8: #9829 replaces the Pallas short convolution with a streaming Triton kernel (4.45 to 1.76 ms per step in a single-GPU hero block); #9830 selects the router's top-k in Triton at 0.096 ms per call instead of 0.856, bitwise equal to jax.lax.top_k; #9831 stops rematerialization recomputing about 10 GiB of values that fit in HBM (+1.04%), runs collectives one at a time without the carry offload, and adds the profile-guided latency estimation (PGLE) builder; #9832 leaves unread ragged transport rows unwritten (+1.41%); #9833 computes the routing-weight gradient on the expert side (+4.12%, 13.541 to 13.005 s, peak HBM 125.4 GiB of 143.75), which David Hall asked to make the default for the supported kernel; and #9834 fuses the SwiGLU backward into the dh GEMM epilogue and skips the pre-activation store under remat (+0.89% and +0.33%). Two PRs closed instead. The hand-placed transport ordering in #9835 was worth 0.13% with a PGLE profile and 0.54% without, so its scheduling is left to PGLE; the shared-expert QuACK kernels in #9836 measured 12.685 against 12.699 s, within noise, and with a profile aborted compilation and then ran out of memory. #9879 tracks running production with a profile, worth 1.5 to 1.9% on one rack; the thread records an XLA remat-offload bug fixed on a marin-community/xla branch and a runtime OOM whose cause is untraced, and asks for a preflight compile with a fallback before any long run.

Before deploying, a Claude agent supervised by Matt Wittmann reran the #9615 context experiment on the stack: 4K went from 13.97 to 12.70 s per step (28.11% to 30.91% MFU), 8K and 16K sped up 10.0% and 9.6% with unchanged drop fractions, and 16K still adds 0.449 s over 4K. #9958 fixed a launch bug first: the trigger's explicit 0.75 memory fraction and slop factor 85 overrode the stack's 0.78 and 105 defaults and would have aborted compilation on every rank. #9960 then handed the run off to hero-mfu30-step231k from a checkpoint forced at step 231,464, after a d768 validation tracked its September baseline over 11,420 steps (held-out loss 2.6783 against 2.6791). Matt Wittmann announced the deploy on October 9, and after a 200-step paired trial on the same batches Matt Wittmann confirmed the losses matched and the new run stayed on. The go record puts median MFU at 29.44% against 26.32%, median step time 13.33 against 14.91 s, mean loss difference −8.5e-5 with a largest of 4.2e-4, and peak HBM 123.4 GiB. The run's first eval at step 234,000 and a resume from its own checkpoint had not yet been exercised. A side effect of last week's #9600 surfaced on a one-rack restore: prefetching every W&B metric at once grew each process to about 1,700 py_xla_callback threads in 20 minutes, and #10024, still open, bounds the in-flight copies to eight.

The context-length switch is now a three-PR stack, all open. #9888 carries the Levanter change: shuffle blocks of 2^20 tokens rather than 256 sequences, identical at 4K so the live order is unchanged, and a prior_context_phases field under which each source skips the unread rest of its current shuffle window at the switch, about 52B tokens or 0.65% of the rest of the run at a 216,000-step handoff. It rejects a phase that changes the active sources or repeats the previous length. #9890 pins the held-out eval at 4K regardless of training context and adds a FLOPs baseline so cumulative throughput does not jump at the switch. #9689 adds the launcher controls: --seq-len holds 4,194,304 tokens per rack per update (2,816 sequences at 16K on 11 racks) so the 390,251-step schedule and learning rates are unchanged, and a changed context requires an explicit --qk-mult and switch step. A review note on October 9 found that the trigger records only one switch step, while the plan has two; a second deploy that kept the first step would compute offsets as if the run had gone straight from 4K to 65K and re-read part of a window. It proposes a checked-in context history instead. The production run still trains at 4K, no GPU rehearsal of the switch has run, and the query-key scale trials Larry Dial asked for last week did not appear this week.

On the science side, a community member posted an independent, preregistered forecast of the hero's final dropless Paloma macro loss in #9977, frozen at 58% of training from public data: median 2.035, 80% interval [2.023, 2.047], 95% [2.016, 2.054]. The August 19 preregistration of 2.039 in #8435 sits at its 68th percentile, and the author traces about 72% of the gap between that figure and a 2.10 ladder fit to whether lm_head FLOPs count as compute. Larry Dial added the first 4K tokens of Paloma's Penn Treebank to the hero vibe checks in #9776 to explain a sudden PTB eval improvement, and #9874 found the cause by rebuilding the exact training stream for steps 204,000 to 216,000: a GitHub dump of quanpn90/torch-nplm in source c13q3 holds the full PTB test and validation sets as sliding 6-grams, 1.39M tokens read at steps 209,272 to 209,669. A follow-up notes PTB drops at steps 104,999 and 191,999 that the known reads do not explain, and the same pattern on WikiText-103. #9745 keeps two checkpoint-completion jobs active so older checkpoints get sampled. Romain Yon's #9701 merged on October 10 and shares the bounded Hugging Face shard writer between the hero export and the Levanter converter, with resume verified by shard checksums; full-size accelerator exports are untested. David Hall's #9460 merged on October 6 and saves and resumes pipeline-parallel (PP) hero checkpoints; probes at PP2 with eight-way expert parallelism (EP) on two H100 nodes resumed bit for bit on CPU and within 2.9e-6 on GPU, but the full 48-layer shape has not been checkpointed, and the GB200 launcher #9662 is still draft. Amy Sartran's #9942 routes the non-hero MoE models through the same top-k kernel, with timings on H100, MI300X and MI350X. In Discord, Mark Muchane explained that the NVL72's large fast-communication domain let the team move to EP shortly before launch and train a model 50% bigger with twice as many experts.

Benjamin Feuer closed two campaigns on tool calling. The Berkeley Function Calling Leaderboard (BFCL) multi-harness recovery campaign #10040 did not reach its goal of bringing the worst harness to 70% of the best. Offline DPO in Levanter lifted a regressed Grug Antidoom step-12 lineage to 35/123 on BFCL-Parity under Pi, 29/123 under Claude Code, 24/123 under OpenCode and 16/123 under Mini-SWE, as Benjamin Feuer summarized, but the extended native DPO run in MarinSkyRL fell from 34/123 to 10/123 after four updates and 0/123 after eight, and he stopped it. Broken traces in the preference cohort taught the model to give up on tool calls, and DPO has no online reward to penalize that. The best initializer is public as BFCL-Recovery, the code is consolidated in #10045, and three follow-ups are open: online multi-harness RL #10041, mining same-history local DPO pairs with Datakit #10042, and final-token preference optimization (FTPO) aimed at tool-call closure #10043. Model merging recovered more. In #9975 Feuer evaluated 113 Snowball checkpoint variants across eleven merging recipes, combining a NUPA specialist (the step-92 checkpoint from mixed-domain RL with verifiable rewards (RLVR): 38.0 NUPA200, 16.3 BFCL-Parity) and a BFCL specialist (Reference-KL step 20: 0.5, 43.9) into the Antidoom step-12 base (0.5, 5.7). Sequential merging by selective expert transfer gave Shared100 at 36.5 NUPA200 and 48.78 BFCL-Parity, but it kept only 0.76% of the base's parameter values and scored 89.8 on MATH500 and 52.2 on GPQA-Diamond against the base's 95.0 and 67.3 #9976. A second campaign that required at least 75% of base weights unchanged found DARE-TIES reached 45.53 BFCL-Parity while changing 6.04% of parameters, with NUPA200 at zero; Benjamin Feuer concluded the NUPA checkpoint is a poor merge or distillation target because its tokenizer diverged. The implementation is in #9974, and #9976 asks for outside advice on capability-preserving merges for MoE models.

Karan Sagar finished the single-expert transfer study #9920. From the Mixed-RLVR step-92 student (CruxEval input/output 55.4/57.8), an expert built by SFT on 1,454 GLM 5.3 traces reached 79.4/71.4. All three ways of moving that gain back into the student recovered most or all of it on the 1,600 seen prompts: plain SFT on the expert's traces 73.9/72.0, rejection-sampled SFT 80.8/77.9, and on-policy distillation (OPD) 79.6/75.2 #9920. Rejection sampling and OPD beat plain SFT by 3 to 7 points; one run per arm cannot order the two. Training cost was 61.6, 57.4 and 287 H100-hours to the selected checkpoint, and no arm fell below the student on GSM8K or MATH500, though these numbers measure transfer on seen prompts, not generalization. The follow-up #9984 builds a second expert on MMLU-Pro and compares multi-teacher OPD, mixed SFT, mixed rejection-sampled SFT and weight merging for transferring both at once. The second expert passed its gate at 60.6 H100-hours: 66.4% on the MMLU-Pro training subset against 43.9% for the student and 64.0% against 44.1% on 10,426 held-out questions, while its CruxEval output score fell 4.9 points. The multi-teacher OPD run launched on October 10, cut to two epochs (about 900 H100-hours) after a smoke run measured 9.6 minutes per update; the first task-arithmetic merge at coefficients 1.0/1.0 produced runaway generations, with IFEval p90 output length at 103K characters against 0.7K for the student.

Harshit Varma proposed plateau dropping for agentic SFT in #10028: track each token's loss across epochs and mask the gradient of tokens whose loss has stopped moving, letting them re-enter if it moves again. On Grug 09.21 with the run7 mix of 6,301 MiniSweAgent trajectories, 20 epochs of plateau dropping scored 14.1 pass@1 on Terminal-Bench 2 and 26.6 on TB-Lite against 9.8 and 21.8 for three-epoch plain SFT (paired +4.2 ± 2.1 and +4.8 ± 1.7); the gain narrows by pass@8, so the rule makes the model solve the same tasks more reliably rather than new ones. Dropping a random 65% of tokens matched plain SFT and dropping by confidence alone cost 11.5 points on TB-Lite, so the loss history matters. A preliminary SWE-bench Verified run on Open-SWE-Traces shows about 38% pass@1 against about 17% for plain SFT with 40% of trials in. Harshit Varma expects it may also be a better starting point for RL, which he has not tested. Marianna Nezhurina posted a 40M-token SFT recovery mix: 1.3 epochs on Grug 09.21 raised SWE-bench Verified random-100 from 19.3 to 40 under Terminus-2, Terminal-Bench 2.1 from 5 to 9 and TB-Lite from 10 to 24, and cut Terminus format errors from 82.6% to 1.5% of turns, but AIME24 fell from 59.0 to 8.7 and GPQA-Diamond from 60.4 to 38.6, which she attributes to stripping reasoning from the Nemotron math and science pool. Luke Lee reported that the relay recipe that worked for Terminus-2 is not working for mini-SWE-agent, and opened #10027 to register the 10,308 CalibForge relay conversations as a Datakit source for decontamination; the converted source does not yet apply the teacher-only loss mask, so it cannot yet reproduce the reported 46.6% SWE-bench Verified. Benjamin Feuer asked both to rebase on Grug Antidoom RLVR1 step 12, the strongest Snowball under the current eval policy at 31, 21 and 11 on SWE-bench, OT-TBLite and TB2.1.

Gonzalo Benegas closed the biology SFT experiment #9775 with no demonstrated improvement. Three epochs on 5,120 windows of GLM-5.3 Skill2Env traces (73.9M target tokens per epoch) left the model at 0/5 on a five-question BixBench diagnostic under Pi with thinking, against 1/5 for the parent, with every SFT attempt ending in an unexecuted tool call inside the reasoning field #9775; under the Terminus schema the parent's first responses were accepted 36/36 and the SFT model's 0/36. The GLM-5.3 teacher baseline on BixBench-Verified-50 under Pi is 39/46 graded (84.78%) #9258. On eval infrastructure, Feuer found that Iris retries overwrite sealed evaluation records #9972: a Snowball relay-SFT checkpoint's GPQA-Diamond changed from 0.636 to 0.599 after a retry replayed the benchmark, and Russell Power asked for the Evalchemy port to resume natively from finelog instead. Will Held proposed splitting the policy's benchmarks into dev and test sets, using external grader code for the test set, and Russell Power and David Hall agreed. Benjamin Feuer fit 24 policy benchmarks across ten baselines and five Snowball candidates against training FLOPs with a shared slope of 1.43: MATH500, GPQA-Diamond, TB2, OT-TBLite and BixBench scale predictably, while many others fit badly and warrant inspection. Held also pinned the OlympiadBench judge to GPT-OSS-120B #9755 and the Pi startup fix for the September campaign #9707. David Hall noted Reflection AI's new model has specs close to Snowball's and will serve as a comparison point, with Marin's post-training compute an order of magnitude smaller.

The RL task stack that was mostly drafts last week is now on main. Rafal Wojdyla's rollout engine merged in #9623: one task per attempt runs through TaskSpec → LoweredTaskSpec → TaskSession, with Shellbox machines for commands and private grading, after David Hall had objected that the executable-task path bypassed the direct-chat lowering and then stripped TaskCompendium down to typed grading in #9608, removing its own Harbor execution in favor of the shared engine. The open #9911 would switch SkyRL's recipes to that runtime. Russell Power then rebuilt RL data curation around it: #9642 converts pinned task datasets into audited TaskSpec Parquet with Zephyr, and #9798 makes every Atlas source a Python declaration, 146 of them, with the 192-file datasets directory cut to 33 and TaskSpec.verifier replaced by a typed grader union. The Atlas itself is now generated from those declarations with its 206 source IDs preserved in #9969, TaskTrove conversion runs locally from them in #9970, and the old pipeline's 10,274 lines went in #9973. Schema 0.26 in #10021 separates what a task requires from how Shellbox runs it, and each source now declares its own pipeline in #10015. TaskTrove math moved to sandboxed verifyit in #9966 because the archived scorers passed model text to SymPy's sympify, which evaluates Python. Verifyit, TaskCompendium, and RolloutEngine are published as marin-* packages in #9982, which Russell Power expects to reduce dependency trouble for SkyRL. The grader debate on Discord settled a policy: David Hall asked whether verifyit should always be as correct and tolerant as possible, citing a jee_bench tolerance bug that rejects 1 against 1.01 but accepts 0.01 against 0.02; Benjamin Feuer said no, since instruction-following benchmarks depend on strict graders; Russell Power proposed keeping bad graders for evals and fixing them only for training ingestion, and Will Held added that held-out evals should run external eval code. Hall's #9803, exact rational arithmetic for numeric answers, merged under that rule. Two design documents frame what comes next: Mark Muchane on generating RL environments and Russell Power on ingesting existing datasets.

Sandboxing for graders got stricter. Local Shellbox machines now run in bubblewrap over a private root instead of under Landlock on the host filesystem, with fresh PID, IPC, and UTS namespaces per command, in #9949; #9954 fixed a dangling uv interpreter symlink inside the sandbox that had made about a fifth of grader trials on the curation workers fail with ModuleNotFoundError, and #9955 mounts only explicit read-only roots. Rafal Wojdyla's agent closed holes on the grading side: archives are collected with the agent's own permissions, symlinks and hardlinks are rejected, and expansion stops at 1 GiB or 100,000 members in #9940 and #9988; Harbor shell tasks must grade in a separate verifier in #9937; Docker output is bounded and timed-out Daytona machines survive for later commands in #9938 and #9941; a ShellSim panic now becomes an exit code 1 instead of aborting the trial in #9965. Still open is #9873, Mark Muchane's request that graders be able to kill processes an agent left running, which Russell Power expects to be awkward on the Docker and Iris backends. Benjamin Feuer's batch of TaskTrove source repairs, posted for review on Discord and merged by Power since he had to port them to the new pipeline anyway, honors CodeContests' per-case time limits after a scan of all 8,222 released tasks in #9864, recomputes sorting gold that shipped in the wrong direction in #9865, rejects conflicting NL2Bash counts in #9866, requires structured-output answers to be grounded in the supplied document in #9868, excludes four E2EGit and three Stack tasks whose contracts contradict their instructions in #9869 and #9871, and keeps SweSmith environments below pytest 9 in #9870. Pytest startup failures count against a candidate only when they come from editable candidate code in #9923. Feuer closed the verifier-gap audit #7418: the team will source its own RL data rather than fill TaskTrove's gaps.

Mark Muchane's Taskforge is the generation side of that plan, and none of it has merged. The skeleton in #9993, 11,266 lines, runs end to end on ShellSim with scripted models; beneath it, #9783 adds a GLM-5.3 client that retries, waits out an empty router pool, and continues replies cut at max_tokens, and the drafts #9784 through #9789 and #9847 through #9850 carry a capability idea through proposal, triage by structural checks and a seven-axis GLM rubric decided by strict majority, builder programs of memoized steps, validation by solver, adversary, and control replay, a pure rule table for accept, reject, repair, or retry, and a resumable queue that runs on a laptop or in an Iris task. Supporting verifyit pieces are split out: judge output budgets for reference and checklist rubrics merged in #9772, while weighted and gated reward aggregation in #9767, repeated checklist judgments resolved by vote in #9770, grading a supplied final state without inference in #9838, and a text token contract for chat endpoints in #9781 wait on review. Running model-written commands on the cluster exposed the credential problem in #9816: sandboxes under the old gVisor profile received the object-store keys and the submitter's Hugging Face and W&B tokens, any task could reach the controller as the anonymous admin through trusted_cidrs, ExecInContainer checked no owner, and a gVisor container on GCP could take the VM's service-account token from the metadata server. #9817 adds CONTAINER_PROFILE_SANDBOX, which withholds all of that, gives every job an egress policy, and requires the caller to own a task before executing in it; #9818 gives each task a token with the job's owner as subject. The old profile came back temporarily in #9855 after the deploy found running clients still submitting it. Follow-ups cover narrowing trusted_cidrs in #9992, sandbox pods that cannot pull from the task registry in #9996, and a cached job environment that hands scrubbed secrets to child jobs in #9997.

Iris had two controller incidents. On October 6 on cw-rno2a, GetTaskStatus and ListTasks took 58,207 requests in five minutes and a thread sample found 56 of 64 RPC handlers waiting for a read-pool connection; #9843 drops the repeated ListTasks calls from inference polling and #9828 moves container exec onto its own pool of 128 workers. On cw-us-east-02a, concurrent writes kept restarting the SQLite backup so controller checkpoints ran past both the 300- and 1,800-second deadlines; #9842 holds one read snapshot across the copy. From the hero incident record, #9720 forwards preemption to the peer that runs a federated task, since the hub had acknowledged the override while leaving a hung trainer running, and #9721 cordons a node whose S3 probes keep failing, but only after a fresh majority of nodes succeeds against the same bucket. Worker credentials now renew a day before their 30-day expiry in #10010, after an expired token had gone unnoticed behind a cached Finelog endpoint. Matt Wittmann found that the periodic py-spy thread dumps paused every training process: 7–10 seconds per sweep on a 704-GPU run, about 1.3% of throughput; with --nonblocking in #9998, a GB200 rack's sweep fell from 0.67–0.93 seconds to at most 0.05. A setup scan that exited 1 after a successful install took down the TPU, Datakit, and Grug multislice canaries in #9809, #9810, #9811, and #9821; #9815 removed it. Romain Yon's #9856 prunes the GHCR build cache, which held 2.0 TB across 13,534 Iris versions, and David Hall documented in #9913 that actor RPC's cloudpickle payloads mean both sides must trust each other.

On AMD, Amy Sartran closed the porting issue and opened #9812 to track MFU, where each experiment is one comment with its commit, command, and result. The MI350X went from the 3.88% baseline at batch 8 to 12.70% with batch 96 and save_moe remat, then to 13.03% with a gather-based ring combine and a routing top-k kernel, and to 14.31% at 1.580 seconds per step with ring_dedup; MI300X sits at a confirmed 8.66% at batch 16. The merged pieces are the Slurm wrapper for the HPC Fund cluster in #9877, which turns off XLA command buffers because they produced NaN gradients and segfaults on this ROCm stack, the synthetic-token benchmark in #9693 with its remat and watch flags in #9852, allocator preallocation off in #9901 after a 1 KiB fragmentation gap, a step timeout that exits instead of hanging in #9903, and #9777, which stopped the ragged all-to-all backend from parsing gfx942 as a float. The kernels are still open. #9880's ring_dedup sends each token once to each GPU that owns one of its top-4 experts, 537 MB per collective per GPU against 1,174 MB for ring, on top of #9860, which replaces the combine's scatter-adds, running at 0.8–1.2 TB/s on AMD, with gathers. A Triton flash attention launched through jax_triton in #9853 runs one layer 12–24% faster than its Pallas version, 6.77 to 5.11 ms for the 2,048 window on MI350X, but any jax_triton kernel hangs multi-GPU training on ROCm because jaxlib's loader deletes the compiled kernel file while sibling threads are still reading it; #9859 waits on a jaxlib fix while #10009 and #10014 land Pallas-Triton versions in the meantime. Two measurement PRs closed without merging but settled a question: JAX reaches 1,229 TFLOP/s on a dense bf16 matmul on MI350X, 97% of PyTorch, so dense matmul does not explain the gap in #9840, and all-gather reaches 392 GB/s in #9886. MI300X-specific tuning in #9904 and #9905 was closed because the work targets MI350X and MI355X. On Discord Sartran noted that JAX out of the box leaves compute on the table against PyTorch.

Eval serving lost work to three failure modes. Will Held traced North Mini Code's MRCR eval to a port collision in #9862: two inference jobs on one node picked 127.0.0.1:39535, North accepted Nemotron's server as ready, and all 602 requests failed; the open #9863 reserves a task-owned port and checks /v1/models for the configured model. Karan Sagar found in #10030 that when Kueue preempts a serve job and its retry registers on another host, the eval keeps retrying the old address; nine groups on cw-rno2a were lost that way on October 9 and 10, some after 16–20 hours, and #10033 routes Evalchemy through a loopback relay that resolves the endpoint from the Iris registry. In #10036 a Kueue-gated pod, which Iris reports as BUILDING, counted as placed, so the 2,400-second startup budget ran while the serve was still queued and eight groups failed in one night without a GPU; #10037 and #10038 start the clock at placement. Benjamin Feuer showed in #9972 that an Iris retry can reuse an evaluation's run ID and overwrite a sealed record, and Russell Power asked for the Evalchemy port to resume from Finelog by default. Smaller eval fixes merged: distributed RunAI checkpoint loading in #9961, an explicit OlympiadBench judge in #9755 because Together rejects the default gpt-4o-mini, CruxEval's 1,600 items no longer reported ungraded in #9750, and Pi pinned to Node 22 in #9707. Romain Yon's drafts #10022 and #10023 stage vLLM 0.30 for H100, GB200, and TPU from one source, adding the size-one expert and context mesh dimensions that Snowball's weight layout needs; the TPU refresh is blocked in #9751 on an AuxOutput API the protected vLLM source lacks. Evaldash keeps comparisons within a cohort in #9752 and stopped clearing its page on every 60-second poll in David Lundgren's #9956; Hall's searchable model picker in #9897 is a draft. The chat dashboard warns when vLLM cuts a reply at the output limit in #9804.

Datakit and Zephyr got faster and shed code. Himanshu Kriplani's #9694 scores quality from the token IDs the tokenize stage already stores, with the classifier retrained on the Marin tokenizer: 6.4 seconds and 32 CPU-seconds on a 35,000-document bench against 44.7 and 115 for the text path, at a holdout AUC of 0.837 against the deployed e5 model's 0.921, with 99% of documents within one bucket of e5. His fuzzy-dedup changes send connected-component labels only from nodes whose label changed in #9773 and drop singletons before the attribute shuffle for 8× fewer shuffled rows in #9845. Rafal Wojdyla's agent removed verify_fuzzy_dups and the Zephyr memory store, 8,849 lines, in #9738 and #9739, gave each worker attempt its own ID in #9741 so a shard no longer stays in flight after Iris replaces a lost worker, and stopped a failed producer from committing a partial Parquet file that a rerun would then skip in #9895. Russell Power bounded shuffle buffers and exported coordinator counters to Finelog in #9884, gave the Zephyr dashboard per-pipeline progress bars and shard heatmaps in #9906, and removed FineStore's per-worker SQLite key index in #9876; his opt-in intra-shard map execution in #9995 is open. Marina gained Filer, a viewer for private S3 and GCS buckets with paged Parquet tables, in #9935, and the shared XProf viewer now runs on Pinak's xprof-rs in #9826, after #9827 trimmed the bundle under Iris's 25 MiB limit. The tier-1 Datakit ferry then timed out in #10047 when the store asked for 106 workers and Iris ran one. On the decontamination recall question in #6852, a real case arrived from the hero run: the document that contaminated Paloma PTB is a repository dump holding the whole validation set as sliding 6-grams. Luke Lee's open #10027 registers 10,308 Terminus-2 relay SFT conversations as a Datakit chat source, which Benjamin Feuer asked for so the data is decontaminated under the policy before the Snowball post.

CI got measurably quicker. Amy Sartran found the levanter-tpu venv lived on a different filesystem from the uv cache, so uv copied all 218 packages every run; hardlinking in #9899 cut the install from 1 minute 32 seconds to 294 milliseconds and job start to pytest from 103 to 10 seconds, with the lane's timeout raised to 35 minutes in #9893 and the runner's pre-populated 4.6 GB cache finally mounted in #9910. unit (marin) was the slowest job in 50 of 98 PR pushes sampled that week at a median of 5.3 minutes; #9924 shards it across three runners, the longest taking 3:32. FFmpeg packages are cached after one job spent 11 minutes downloading them in #9902, service tests pull through Google's Docker Hub mirror in #10001, Shellbox changes now select Shellbox and TaskCompendium tests in #9914, and #9909 stops an old gh from masquerading as "no checks yet", which had left one PR monitor waiting twelve hours. Open: Hall's #9867 would run only explicitly marked tests on TPU, 14 files instead of 121, and Sartran's #9985 instruments a 2–19 minute stall that hits 11 of 83 jobs in a TPU VM's first hour and none after three. In Levanter, Hall's context-parallel queries for the native SM90 attention backward merged in #9282, Matt Wittmann's #10024 bounds the device-to-host copies that a W&B watch step released by the thousands, Benjamin Feuer's #10039 keeps a cache marker from declaring a 90 GB mirror complete at 28 of 39 shards, and Mridul Agarwal's #9806 floors the AdamH norm after a zero-norm parameter poisoned a 67B SFT checkpoint on its first update in #9054. David Lundgren's PivotRL pivot selection and SFT baselines, #9933 and #10000, are drafts. Six external-runtime bumps, two Dependabot updates, and security fixes for fast-uri in #9912 and Notebook, JupyterLab, and virtualenv in #9918 round out the week.

Himanshu Kriplani merged three Datakit PRs: the token-id quality scorer #9694 and two fuzzy-dedup cuts, #9773 and #9845. Qirun Li, who posted the hero forecast in #9977, read every Paloma and Uncheatable eval for #9874 and listed PTB drops the torch-nplm reads do not explain, then brought that document to the decontamination recall question in #6852. RavnaBergsndot suggested in Belay Intake #9514 warming up MuonH's Nesterov momentum from 0.85. Baixin Guo offered to build the admission check for local preference pairs in #10042. jeremybarreto710's first contribution, the PleIAs SEC 10-K filings in #9275, drew a note from Will Held that they are a subset of the SEC-Edgar source Datakit already downloads. Tim O'Donnell opened #9881 to give PDBThink and Skill2Env one release-intake path in Atlas, and Ben Kolligs described Delphi vision experiments in #9524.

On Discord, Pinak shared xprof-rs in #random, Russell Power said the team runs a shared xprof server, and Pinak saw it in use and offered a PyPI release. Matheart posted a paper on a low-rank router for MoE models with thousands of experts, noting that reading and writing routing scores gets expensive at that count, which bears on the router top-k kernels in #9830 and #9942. In #openthoughts-next, Charlie R joined the RL data effort, asked Franziska Weindel about the OT-next-data pipeline, and found more reward hacks in the MiMo SWE environment than the vals.ai post Bilibird had shared.

Five people introduced themselves: Tane, a Stanford sophomore in Percy Liang's CS221, from creative writing and a summer as a venture partner at S32, drawn to post-training, RL and alignment; Guillaume Delore, a full-stack and cloud engineer in Europe with an ML and applied maths education, here to build pre-training and post-training skills; Himanshu Kriplani of Hark Labs, formerly on Uber's Michelangelo platform, who found the channel after his first PRs merged; Alex, an SJSU student with a computational linguistics background, new to LLM research; and Abhijith Neil Abraham, founder of Vitalops.ai, who has built LLM inference at scale. Kriplani's work already intersects with Datakit; Abraham brings context on inference serving. About seventy others joined without an introduction.

The links in #news, #data-curation and #reinforcement-learning were things to measure against: a newly released open model with specs close to the hero's, catalogs of RL environments and terminal-task grading data, and a report on reward hacking in a public coding environment.