Marin: Week of October 26th summary

The hero run opened the week at step 328,186 and 15.14T tokens and stood at step 369,382 of 390,251 and 17.04T of 18T tokens at about 16:00Z on Nov 1, roughly 41,200 steps and 1.9T tokens in seven days despite one outage and one planned relaunch. At 46,137,344 tokens a step it passed 16T near step 346,800 on Oct 28 and 17T near step 368,500 on Nov 1. Train loss was 1.0119 at the week's end; step time 14.08 s and MFU 29.9% on the 16K build, a little below last week's 13.8–13.9 s and 30.3–30.6% because the new run includes checkpoint-write overhead in its averages. Dropless Paloma macro loss fell from 2.0019 at step 353,999 on the parent run to 1.9650 at step 368,999 on the child. With 20,869 steps left at 14.08 s the run would finish near 03:00Z on Nov 5 at 16K; the planned 65K phase (below) moves that to about 15:00Z the same day per #10821.

The week's one unplanned stop began at 17:18:42Z on Oct 29, when trainer task 1 lost its heartbeat at step 352,561 and Iris requeued the 176-task gang, per the incident log. Fourteen automatic retries restored the rolling checkpoint step-352321 and then died with NCCL CUDA out-of-memory before logging a step. Host diagnostics found about 18 GiB per GPU still held on 12 of 16 nodes in rack 392 while idle: a stale nvidia-imex fabric peer had disconnected at 17:17:32Z and never released memory imported from the departed node, per the root-cause and migration note. Cordoning the rack alone would not help, since Kueue needs eleven complete 16-node NVLink domains and the only spare rack, 399, held 13 nodes of other jobs. The operator cancelled those four jobs, cordoned the affected nodes, and attempt 15 was admitted on ten original racks plus rack 399; it stalled on one node whose init container hung after cgroup OOM-kill and QSPI timeout events, and attempt 16 resumed training from step 352,321 at 20:47:17Z. The stop cost about 3.5 h of wall time plus 240 redone steps, roughly 4.4 h in all. CoreWeave recommended node restarts to pick up an IMEX fix; eight hero racks still run the old revision and will be swept while the run is stopped for the context switch. Storage checked during the outage showed US-EAST-08A at 86.75 TiB of a 500 TiB quota, 53.26 TiB of it hero checkpoints. #10761 from David Hall and the Grug checkpoint-on-SIGTERM PR #10774 (open, 60 s grace, validated so far on two H100s) address the other side of this: a preempted trainer today loses everything since its last hourly temporary checkpoint.

Late on Oct 29 the run's gradient norm reached 20.84 at step 353,060, 41.6 times the recent median of 0.501, and loss jumped from 1.018 to 1.101 at the next step before recovering over about 100–150 steps; an earlier 4.49 at step 352,190 left no mark on loss, per the analysis in #10677. Larry Dial proposed a global clip and merged #10783 at 13:41Z on Oct 30, which clips the total gradient norm at 1.0 outside the optimizer so checkpoint layout is unchanged, and logs train/grad_norm and train/grad_clipped every step rather than every tenth. The hero's code is fixed at launch, so the change required a relaunch: a permanent checkpoint was requested through the training-control endpoint, the parent kept running to record a control window, and at 13:51Z the child hero-gradclip-step356k was launched from step-356531 at commit 8aac0db3d9. All ranks loaded the checkpoint at 13:54:29Z and the first step logged at 14:02:35Z with loss 1.007708, identical to the parent's, per the first-step check. The parent stopped at step 356,621. The 50-step paired gate passed: loss delta mean −3e-5 (max 1.8e-4 over 91 overlapping steps), step time 13.78 s against the parent's 13.96 s, peak memory 123.2 GiB, no retries. The clip had not fired through step 356,727; the pre-clip norm median was 0.458. Larry Dial also reported no loss change from an FP32 attention-output test, #10785, so that change was not deployed; details are covered under the hero engineering section.

The switch to 65,536-token context is the next planned step. Matt Wittmann opened #10821 and the deploy PR #10820 (draft) on Oct 31 with a handoff planned for 17:00Z on Monday, Nov 2, near step 375,500 (96.2% of the run), leaving 0.68T tokens at 65K. The live job keeps its mesh, context parallelism (CP) 1 by expert parallelism (EP) 64, and changes only qk_mult to 1.66 and the router capacity factor from 1.15 to 1.5; a one-rack rehearsal of the deploy branch restored from step-360000 ran 60 steps at 15.571 s a step with three dropless evals and no OOM. Expected hero step time is about 16.3–16.8 s, a 15% drop in tokens per second, and about 50B tokens are skipped at the switch as each data source abandons the rest of its shuffle window. The checklist follows the Oct 30 relaunch: request a permanent checkpoint about 15 minutes before stopping the parent so it records a 50-step control window, stop it, run the IMEX sweep on the eight affected racks while no hero pods remain, then fork W&B and launch hero-ctx65k-step375k. Alerts stay on through the first 65K eval and permanent checkpoint at step 378,000, about 13 h after launch; because evals pack at the training context, the Paloma macro loss will step at the switch and the eval_regressed alert may fire. Rollback is to relaunch hero-gradclip-step356k from the same checkpoint. The CP4 configuration and multi-length evals (#10618, #10711) continue toward the 262K extension and are covered under the hero engineering section.

The gradient-spike investigation #10677 reached a mechanism, and the hero now clips. Larry Dial replayed the step-324000 batch on one rack and pinned its 2,320 gradient norm to one token in one sequence: near-tied top-8 routing in layers 10–28 turns a one-ulp bf16 change in 0.6% of embedding entries into a different expert path (7 of 8 experts flipped by layer 28), and the exact path lands where layer 35's attention and gated norm give a 10× local backward gain per layer. The perturbed state gives a norm of 18.8 on the same data, and recomputing the two amplifying ops in fp32 and fp64 reproduces the bf16 values, so each op computes its true derivative; the forward state is what is unusual. The later spike at step 353060 (norm 20.84, 41.6× the median of 0.501 since step 340000) was the first to move the loss, 1.018 to 1.101 on the next step, recovering over 100–150 steps; it did not reproduce from the step-348000 weights, and the outlier sequences were long windows with no BOS token full of repetitive formatting (markdown table pipes, mojibake). #10783, merged Oct 30, clips the total gradient norm at 1.0 in the train step, outside the optimizer, so the opt_state layout and existing checkpoints are unchanged, and logs the pre-clip norm every step instead of every tenth. Two one-rack resume tests from step-354000 at batch 256 (hero-clip-rt1, hero-clip-rt2) saw a median pre-clip norm of 0.39, the clip fire at 1.041, 1.975 and 3.228, 13.2–13.4 s/step at 31.5–31.8% MFU, 123.21 GiB peak and no NaN; identical batches from identical weights spiked on different steps, which fits the ill-conditioned-point finding. He posted the PR to Discord Oct 30 with a plan to merge that morning. The relaunch as hero-gradclip-step356k is covered in #8506.

Matt Wittmann settled the 65K layout on Oct 31 in #10821. Two one-rack arms restored from step-300000 and run 500 steps each: context parallelism (CP) 4 × expert parallelism (EP) 16 at capacity factor 1.15, tok350-prec65k-cp4-01, took 14.699 s/step with drop fraction 2.41e-3 and 127.6 GiB peak GPU memory; CP1 × EP64 at capacity factor 1.5, tok350-prec65k-cp1cf15-01, took 15.397 s, 9.80e-4 and 128.4 GiB. CP1 won: 4.7% slower per step, about 3 h over the rest of the run, but 2.5× fewer dropped assignments, no launcher change, and a dropless eval that passes on main with 12.3 GiB free per GPU. CP4's eval crashes the next train step on main because the eval mesh puts devices into groups training never uses; XLA keeps an NCCL communicator per new group outside the allocator pool, about 7.4 GiB per GPU against 5.8 GiB of headroom, and the next step failed to allocate 90.49 GiB. #10709 folds the expert axis into data in place so eval reuses training's groups: non-pool memory rose 0.004 GiB across two evals, the weight move took 1 s instead of 5, and Paloma macro loss was 2.049 and 2.051, as before. The deploy PR #10820 (draft) sets max_seq_len 65,536, capacity factor 1.5 (at 1.0, CP1 drops 1.9% of assignments) and qk_mult 1.66, the 262K formula 1.3 × (0.1 ln(65536/4096) + 1), and keeps 4,194,304 tokens per rack per update, so the 11-rack batch is 704 sequences. Its one-rack test from step-360000, tok350-deploy65k-c1-01, ran 60 steps at a 15.571 s/step median (+1.1% on the earlier arm; this branch includes the clip), drop fraction 8.29e-4, 127.9 GiB peak, and three dropless evals at Paloma macro 1.995–1.998 with no OOM. Expected on the hero: about 16.3 s/step, −15% tokens/s against 16K, and about 50B tokens skipped at the switch (52.2B at the 16K switch). Proposed gates: CE below 1.25 over the first 10 steps, loss within a band of the parent over the 50-step control window, step time at most 1.03× the rehearsal, drop fraction below 2e-3, peak memory within ±1 GiB, no retries, and a completed first 65K eval. Open risks: the production launcher and capacity factor 1.5 have never run at 65K or on 11 racks, peak memory of 128.4 GiB sits against a 143.75 GiB allocator limit, and nothing evaluates the hero past 32K, per #7638, so the phase is judged on loss alone. CP work in #10618 and #10709 continues for 262K; on #10618, ravwojdyla-agent proposed keying the hero layout by context length so a deploy cannot forget to set CP. The handoff timeline is in #8506.

Two numerics audits ran against frozen checkpoints while the run trained. David Hall checked the hero's FP32 streaming cross-entropy for the softmax-collapse failure of Barron et al. in logbooks #10: on one training batch, checkpoints 270000, 300000 and 330000 rounded CE to zero on 2–7 of 4,194,048 tokens, 0.000048–0.000167%, some 600–2,100× below the 0.1% rate that prompted the check. The same sweep found a larger attention problem, logbooks #12: the FA4 backward forms D = rowsum(dO · O) from the BF16 forward output, and on peaked softmax rows the rounding cancels badly. Saving O in FP32 cut pooled Q/K gradient error at 330k from 0.9323%/0.8173% to 0.2767%/0.2487%, and global layer 15, which carried 65–76% of the squared error, from 4.48%/4.53% to 0.35%/0.32%; upcasting the already-rounded O changed nothing, and the FP32 path costs about 8% more attention time at 16K. He noted the deviation "is mattering at high context len" and put the step-time cost near 1%. Larry Dial argued that the hero's qk-norm bounds its attention logits and asked for a loss measurement, then ran one: a 50-step one-rack head-to-head from step-354000, #10785, with ha-base against ha-fp32o, gave a paired loss delta of +1.2e-5 mean with per-step SD 3.3e-5, the same as an identical-config rerun, at +0.4% step time (13.16 to 13.21 s), MFU 32.00% to 31.87%, and +1.50 GiB peak (123.21 to 124.71 GiB). He closed it as not a priority given no measurable loss impact, with the caveat that 50 steps at 1/11 batch cannot see a long-run effect near 0.001.

Eval design for the long-context phases also moved. Qirun Li warned that at 262K, with evals packed at the training length, Paloma's PTB and WikiText-103 sets are each shorter than one sequence and yield nothing: the Paloma group macro silently becomes a 14-subset mean, about +0.06 (3%), enough to trip eval_regressed, while the top-level macro averages in zeros, about −0.14. The 67B's 65K→262K switch showed exactly that, Paloma macro 2.1774→2.2150 while the other 14 subsets moved +0.0004. @mindscanlabs, whose preregistered final-loss forecast sits on the same thread, clarified that forecast B (1.935) was built on 4K-packed ladder rows and forecast A (1.979) on the hero's 16K evals, so the registered H1 and H4 favour the forecasts by about 0.03 from the packing change alone; the scoring plan now reports packing-corrected versions beside the registered ones, with the correction taken from a 4K eval of the final checkpoint if one is logged and otherwise from Qirun Li's 0.0306, and was fixed before the outcome. Both asked that the 262K phase keep all 16 subsets. David Hall answered with #10711: evaluate 4K and the training length at every scheduled eval, add fixed 65K and 262K series once training reaches those lengths, right-pad and mask each corpus's partial final sequence so short corpora stay in the aggregate, and exclude zero-token tags from the overall macros. It reuses #10709's device groups after a review flagged that its first design would repeat the NCCL memory growth. Larry Dial asked that the padding change be tested on existing checkpoints first, since a shift in the 4K and 16K macro loss would put the evals on different data from what the forecasts were scored on. The hero's own Paloma preregistration remains the August comment on this epic, which David Hall pointed to this week along with the forecast code; Qirun Li's preregistered forecast is #9977.

Kernel work stayed off the hero this week. boba shop cashier asked Larry Dial to review #10559, which runs Belay's ReLU² experts on the cudnn-frontend MXFP8 grouped GEMMs and targets the Belay fast-track branch rather than the hero; no rack A/B of #10415 was posted. On Discord, boba shop cashier asked why MFU was low when the hero started, and David Hall answered: about 20% MFU, with no prior access to NVL72 or Blackwell hardware to profile on, no fast open-source kernels, racks expected to drop, and a one-year allocation on which every idle day is wasted compute. Two older threads, the unapplied weight decay #7783 and the FP8 dispatch wire #7665, were auto-closed for inactivity.

Karan Sagar spent the week on why Snowball v0.1 calls itself ChatGPT or Nemotron, in #10695. Logit-lens and activation-patching runs in #10696 found no stored self-name: without a system prompt, late MLP neurons such as neuron 1072 of layer-20 expert 46 push “Chat” over “Marin”, and they fire about as hard on third-person questions about OpenAI, so they hold general knowledge about AI labs; with a system prompt, four attention heads in layers 22 and 23 copy the name into the answer (patching recovers 0.32, 0.26, 0.24 and 0.15 of the gap). A 16-update context-distillation SFT learned the creator but not the name (38 and 36 of 52 prompts say “Open Athena”, 2 and 6 say “Snowball”, thinking off and on) and cut Berkeley Function Calling Leaderboard (BFCL) tool calling from 27.6% to 7.3%; training only the late-attention slice kept BFCL but taught no name. On-policy distillation (OPD) from that checkpoint, #10732, reached BFCL 18.7% with zero self-naming and MATH500 −2.3, so the track was paused. The second path, a short SFT on Will Held's identity dataset plus v0.1 replay, produced the current lead, A13: with thinking on it names itself in 35 and its creator in 32 of 52 prompts and never claims another lab (v0.1: 46 of 52), BFCL is 28.3% against v0.1's 27.6% (+0.7 [−3.4, +4.4]), GSM8K is −1.1 [−1.9, −0.3] and MATH500 is unchanged. The mechanism behind the earlier tool-calling losses: identity rows rendered with thinking on teach the model to answer and stop, so arms with those rows fall to about 15% BFCL, and rendering most rows thinking-off with 10% reasoned thinking-on rows plus doubled tool-call replay recovers it. On Benjamin Feuer's 60-item identity and history panel, A13 names Marin on 35 and claims another lab on 1 (v0.1: 5 and 17) but scores 15 clean against a ship line of 20, since it recites a canned identity line instead of answering; Benjamin Feuer's own scoring of the context-distillation checkpoint gave 15 of 60. Mini840 on the candidates, #10804, is still running.

Getting Levanter to train the same model that vLLM serves took a numerics audit, recorded in four issues. #10697: the MoE ring dispatch at capacity factor 1.0 silently drops 8–23% of expert assignments per layer, and pad tokens take capacity; top-1 agreement with vLLM was 0.85–0.93 (limit 0.975) and rose to 0.990–0.996 once dropless. #10698: BF16 MoE combine, XLA's BF16 SiLU and sigmoid (30–36% of elements one step off), FlashAttention-4 rounding order, and an HF reload that drops the attention setting; with the fixes a zero-learning-rate run passed the precision gate with p99 log-prob error of 0.51–0.58 nats against vLLM and all 502 exported tensors byte-equal to v0.1. #10699: identical requests to one vLLM serve flip the chosen expert set at 5.55% of position-layers and move summed log-probs by up to 0.69 nat, which is most of the gate's 0.10 nat/token threshold; Xiaoyuan Wang reported bit-identical scores on a DP4 serve with batch-invariant mode, a fixed-root expert reduction, deterministic compile and chunked prefill off, tracing most batched variation to Grug's FP32 router going through cuBLAS, per #10760. #10700: merging v0.1 toward v0.1.1 passes the identity gate only at α = 0.875, where GSM8K loses 8.59 pp, and BF16 rounding erases 83.5% of the donor deltas at α = 0.5. The fixes are a draft stack: Levanter route observer and FP32 router dot #10684, capacity setting with drop counts #10685, FP32 combine #10686, FP32 activations #10687; the interpretability harness #10689, SFT recipe and audits #10690, precision gate #10691, LR0 probe #10693, diagnostics #10694, identity scorer #10701, 16-update spending gate #10702, and batch, export and launcher audits #10706, #10707, #10708. On the RL side, David Hall's #10760 adopts a serving profile that reproduced identical tokens, grades and outcomes across six evaluations and four engine startups on the fixed 100-task panel, and Ahmad Qamar drafted one entry point for Snowball RL runs #10740, Harbor evaluation of 30 tasks each from BFCL-Parity, SWE-bench Verified and OpenThoughts-TBLite inside full training #10746, local grading of the 910-task panel #10736, and comparison against calibrated controls #10747, where a saved 25-update Group Relative Policy Optimization (GRPO) run scored 0.3357 against 0.3420 for REINFORCE leave-one-out (RLOO) controls, within the 0.0353 tolerance.

Benjamin Feuer reproduced the cold-start SFT on CoreWeave H100s in #10799: import the Snowball base, stage the 216-source mixture, train on 128 then 256 GPUs, export verified checkpoints. A large agentic gap turned out to be a serving confound: the exported repositories lacked the original second stop token and did not reliably enable thinking, so one step-222 SWE trace issued 166 tool calls in batches of up to 33. With corrected configs, step 471 (9.9% of the 4,750-update schedule) scores a provisional 32.3% on SWE-bench (186 of 500 scored) and 23.9% on OpenThoughts-TBLite against the original release's 39.4% and 25.6%, so the working hypothesis is that about 10% of cold-start exposure recovers most of the performance; Benjamin Feuer posted it as a cheaper ablation baseline. The archive PRs #10789 and #10791 were closed in favor of a Hugging Face config archive. Two outside results arrived on Discord: Luke Lee took the 09-21 checkpoint from 12.8% on SWE-bench-100 and 7.2% on Terminal-Bench 2.1 to 50.5% and 23.9% with SFT on CalibForge and SWE-smith followed by RL on TMax and R2E-Gym, and Marianna Nezhurina reported RL data ablations on an SFT'd Snowball: TMax gives the most consistent gains, pass-rate filtering helps, and going from 260 to 750 tasks adds little. On the preregistered SWE-bench prediction #8933, Ricardo Olmedo added a follow-up: Snowball's best SFT learning rate is about 4e-4 against Grug's 8e-4, and a rerun with frozen router biases and capacity 1.5 reached 39.7% pass@1 against 38.4%, +1.3 pp paired (SE 0.85). In #10753 he proposes the next preregistration from a proper scaling ladder, since the earlier miss came mostly from a −3.5% test-loss error while the bpb-to-pass@1 link held within 2%, and asks for the overtrained d512/d768/d1024 ladder checkpoints to be released.

David Lundgren closed the mixed RL-with-verifiable-rewards (RLVR) study #10214 with a diagnosis of its step-12 Math500 drop: long answers start as normal reasoning and decay into multilingual word salad. In arm A seed 42, 93 of 500 step-12 answers are degenerate, 33 of the 42 lost problems among them; repeated 4-gram share is 0.02–0.05 against 0.17 for the SFT's long answers, so this is a flattening distribution (probability mass outside the top 20 tokens 0.07 to 0.35 over an answer) rather than a loop, and how far a run melts tracks its Math500 loss. The same melt appears in Antidoom-RLVR1 step 20 (37 degenerate answers) and not in the Snowball Smartie or math-RL runs. The planned Grug long-SWE PivotRL rerun #10585 was closed in favor of its successor #10801, which will recompute pivots from Marin's own student instead of NVIDIA's and train three matched arms, after #9558, #10699 and #10400 are fixed, as he summarized on Discord. On the stray think segments #10400, Xiaoyuan Wang found that the eval campaign's Grug configs let vLLM delete the think tokens, so stray reasoning reaches graders as answer text; keeping the markers changes almost nothing on the thinking-off benchmarks, and grading only after the last end-think raised Snowball v0.1's IFBench strict accuracy from 29.7% to 31.7% in one sample.

Smaller items: the two-expert transfer study #9984 closed, its ten public checkpoints join the Snowball family tree in #10599 (130 to 140 nodes); #10597 lets merges widen BF16 router biases to FP32 so the SFT-465 checkpoint can be a merge parent; the TPU merge-screening loop #10270 recorded a passing H100, v5p-8 and v6e-8 calibration (89 of 100 GSM8K on every arm); and Suraj Singh's #10786 reshards Snowball norms and router bias on load, fixing the SFT resume failure #9380. The post-training milestone, set for September as an ambitious goal, continues: v0.1 is public, the identity fix has a candidate under evaluation, and the RL and numerics tooling above is still in draft.

SkyRL correctness. Russell Power's ten transplant drafts (#10360, #10371, #10357, #10359, #10366, #10375, #10379, #10358, #10365, #10362) were closed in favor of Ahmad Qamar's rebased stacks: typed recipes #10642 and #10643, worker batch assembly #10666–#10668, exact vLLM-trainer numerics #10634–#10636, and a three-part correctness stack, all draft. #10680 lets Grug rollouts continue after a tool call, which raised on main. #10710 fixes four reward rules: unfinished Nemotron Ultra replies score 0, Group Relative Policy Optimization (GRPO) and REINFORCE leave-one-out (RLOO) baselines use only eligible rows, Harbor retries only masked failures, and GSM8K needs a #### <number> final line. #10681 makes Grug checkpoints state query scale, attention window, context limit and experts per token, because the pinned vLLM filled missing fields with its own defaults (1.0, full context, 4096, 2) where Snowball uses 1.57, 2048, 65536 and 4. #10682 computes trainer log-probabilities and entropy in fp32 from logit margins, following Barron et al.'s softmax fix: on a saved Qwen2.5-0.5B batch, 20% of active tokens had entropy exactly 0 on main and none after; Romain Yon approved it and it awaits merge.

SkyRL operations and RL experiments. Benjamin Feuer merged #10715, which lets Megatron policies with optimizer_cpu_offload resume at all (the Snowball on-policy distillation (OPD) recipe failed in the first 10 seconds of policy load), and #10742, which frees cached pinned memory after that load, after the first post-resume save of Snowball 67B was killed at 90–96 GB per save worker. Ahmad Qamar's #10402 raises rollout concurrency to the in-flight group cap and #10659 groups online EAGLE capture windows by prompt. Russell Power ported #10369 and #10368 (explicit NCCL channel limits: eight channels cut the 192-H100 hero learner update from 50.74 to 29.08 seconds), and #10825 makes HF export opt-in, since 1.3 TB intermediate 67B checkpoints had accumulated awaiting export, and moves cw-rno2a outputs to US-EAST-08A. In review: David Lundgren's stack #10624, #10625, #10626 and #10738 rejects or penalizes malformed thinking spans at 0.1 of trajectory reward; Karan Sagar's #10730 lets an OPD teacher see prompt insertions the student never sees, and #10731 bounds SkyRL Iris preemption retries; Ahmad Qamar's #10737 persists per-step metrics across resume; David Hall's #10714 honors explicit request sampling parameters. David Lundgren closed the PivotRL thread #9176 and summarized it on Discord: selected SWE and Terminal pivots gave lift just outside seed noise, random pivots hurt, and all arms lost MATH500 and MMLU late; the Qwen3-30B-A3B tracer #10546 raised SWE-bench Verified from 4.6% to 15.6% at step 10 with AIME24 unchanged. raspberryice's #10320 reports Nemotron RLVR1 (RL with verifiable rewards) instruction-following rows that reward all-caps and letterless answers.

Eval-policy corrections. Eval policy #10197's September-campaign corrections reached main. Benjamin Feuer merged GSM8K extraction #10808, which accepts $139 for 139; trial counting #10671; SOTOPIA scoring in Harbor with a versioned 13-model cohort #10670; judge defaults #10650; Tau3 observations as text blocks #10665; and Mini-SWE startup repairs #10655. Will Held merged inference startup retries #10656, the drop_params strip #10657, closing #10262, where SWE-bench had read 0 since every trial failed before its first model call, and Terminus tokenization errors #10630; Karan Sagar merged #10534. Metric selection through Evalchemy #10672 and NUPA Loose extraction #10733 are open. Benjamin Feuer closed #10804, the Mini840 and Micro210 subsets (21 of 26 benchmarks) for screening checkpoints during RL, and opened draft #10817 after an end-to-end check found the artifacts lacked the environment map SkyRL needs. On #9558, Xiaoyuan Wang's agent found both "identical" coding runs sampled at temperature 1.0 with no seed, because the pinned Evalchemy strips both. Benjamin Feuer's #10751 ran the chat-template gate on four untrained MoE bases and noted that Snowball's base often closes a </think> it never opened. Gonzalo Benegas's #10575 adds no-tools LAB-Bench subsets, with GPT-6 Luna results.

Shellbox and task curation. Russell Power merged #10640, persistent Shellbox tasks: a primary container plus runtime containers in bubblewrap, sharing private loopback with separate filesystems and PID namespaces. It prepares the end of Harbor's Docker-in-Docker: #10628 routes Harbor task and verifier execution through Shellbox and removes 37,167 lines of provider backends; two tau3-retail-2 trials scored 1.0 through bubblewrap. Benjamin Feuer's #10735 shares teardown across launch paths and restricts vendored Harbor to six agents. #10819 is next: remove storage and provider credentials from the orchestration process; Russell Power expects #10628 to resolve the agent audit #10488. Eval weight-prep #10277 is still open. Russell Power's #10757 curates all Evalchemy benchmarks as canonical TaskSpec rows (96 declarations over 31 task names); once it lands, vendor/evalchemy can go. Zachary Nichols's open-ended TaskSpec authoring #10590 is draft, Anirudha (Ani)'s #10750 was closed as superseded, and Baixin Guo's #10741 merged a screening of agent traces for same-history preference pairs and asked on Discord for a trace slice with several attempts per task.

Iris and canaries. The loom-oa-dev agent merged #10795, which rejects RECREATE for an active Kubernetes job instead of orphaning its pod, after #10712 found four copies of one controller job running at once; #10744, which shows scheduler reasons for pending replicas; and #10759, which makes JAX peers refresh task 0's current attempt after the Grug multislice canary #10755 stalled on unequal retries. Russell Power's #10538 adds an allowlist egress policy enforced with Cilium, default on cw-us-west-04a; after Rafal Wojdyla's agent review, multi-task GPU jobs on host-network clusters resolve to CLUSTER with a warning rather than running unfiltered. Drafts: native task containers #10822; Karan Sagar's #10770, #10771 and #10769 for #10768, where successful parents killed unfinished export and eval children; #10721, a 503 with Retry-After during serve replacement, after a GSM8K screen lost 112 of 1,319 requests to 502s in #10273; #10703, nested bubblewrap for non-root Kubernetes tasks for #10692. The Datakit canaries failed on scheduling capacity all week: Tier 1 #10717, #10743, #10752 had up to 106 of 106 store workers pending on insufficient memory, and Tier 2 #10728, #10754, #10802 waited on fuzzy-cluster workers. The SkyRL nightly #10776 failed on cw-rno2a storage, write-suspended in #10777 and stalling in #10787, which is why #10825 moved outputs. Ryan Williams opened #10718 (record an owner for every job output) and #10720 (audit compute use).

Dependencies and IAM. Russell Power's #10803 patched vulnerable dependencies across the Python environments, nine frontend lockfiles and three Rust workspaces; David Hall raised floors for OAuthLib #10749, Authlib, dnspython, GitPython, msgpack and NLTK #10748, virtualenv #10745, Brotli and Zod #10619. Dependabot's #10816, #10815 and #10814 are open. Marin Dev grants now go through Pulumi: Ryan Williams #10729, David Hall for two Stanford students #10775, and Rafal Wojdyla's agent #10781, declaring an existing grant the next pulumi up would otherwise have removed; #10772 adds the quota-project permission a new developer tripped over. The legacy-grant review #7715 closed.

Levanter, training and SFT infrastructure. Benjamin Feuer's agent found that TensorStore 0.1.84 restores a 2.5 MiB BF16 array as zeros when writes reference the staged source; #10792, merged, copies on write; the upstream cause is open. Drafts #10796 (FA4 singleton metadata) and #10793 (integer mixture allocations survive roundoff) follow. Karan Sagar's #10683 keeps router biases in float32 while computing in bfloat16 and #10688 adds a Grug attention path in torch eager's order, for Snowball torch parity. The loom-oa-dev agent's #10276 resumes interrupted HF mirrors by file size, closing #9230, where a GLM-5.2 retry re-downloaded 295 GB. David Hall's #10713 turns CPU test scripts into multi-device tests; #10774 makes Grug ranks agree on the completed step after SIGTERM and checkpoint before exit for #10761. Suraj Singh traced the resume failure #9380 to sharding and opened #10786. For SFT, Karan Sagar's #10705 counts supervised tokens per class from the loss mask and #10782 keeps one rolling recovery checkpoint plus the final one by default; Benjamin Feuer's #10794 bounds cache batches for very long replay documents, #10798 preserves generation_config.json on HF export, #10797 locks tokenizer staging across processes, and #10813 keeps only resolved Open-SWE trajectories. The unified export contract #10484 was closed instead at Russell Power's request. #10788 proposes cleaning the cold-start catalog with GLM against an unchanged control.

Datakit, Zephyr and Finelog. Luke Lee's #10027 registers pinned CalibForge relay SFT traces as a Datakit chat source. The loom-oa-dev agent's #10778 makes download_hf check each destination on rerun; Jason Chen's #10766 showed a rerun skipping the whole map because a metrics sidecar existed. It also closes #807, from 2024. Russell Power's #10553 records row-commit ranges in finestore shards so lookups skip superseded payloads. Open: Himanshu Kriplani's #10106 shuffles MinHash LSH buckets as uint64; zhaoyanthu's #10807 validates fast_transformer calibration knots after finding non-increasing x-values in the released nemotron_88k curve; #10639 hashes multipart S3 copies after CoreWeave returned different ETags for identical bytes, which Russell Power questioned as redundant with provider checksums; Louisa H's #10800 repoints the egress report at object-backed finelog logs, empty since September per #10716. In the Common Crawl sync #2351, Michael Ryan reported that decontamination was discarding useful code because license comments and import orders escaped the boilerplate rule, and added a StarCoderData frequency test. Observability tracking issues #7804, #7795, #5198, #6598 and #7714 closed as done.

Loom, Marina and Echo. Russell Power disabled MPTCP on the Loom host and Caddy #10823. #10430, where Open-Athena members got a GitHub 404 signing in to loom.oa.dev, came from GitHub's rule that a private organization App authorizes only that organization's members; #10805 documents the fix, two Apps: the Marin-owned one for repositories and a new Open-Athena-owned one for sign-in. Larry Dial's #10824 adds a Loom profile so Marin Belay sessions run there instead of on a laptop. Russell Power merged #10767, moving Marina sources to infra/marina/apps/ and retiring the applet distinction; Karan Sagar's nightly Snowball family-tree publisher #10719, for #10598, needs a rebase onto it. Rafal Wojdyla's agent added Echo work-log commands and an echo-log skill #10756.

Research: Delphi, Belay, vision-language models. Kaiyue Wen's #10790 adds a Delphi study of two-sided EMA gradient-Gram preconditioning at polar exponents 0, 1/8, 1/4, 1/3 and 1/2, each LR-tuned with three paired seeds; the agent running it in #10784 has finished the baseline pilot. Rohith Kuditipudi hit HF's public-storage quota on 20 more Delphi checkpoints (118.82 GB) in #10758. On the Belay ladder #9451, Larry Dial's agent closed two intake ideas: softplus cross-entropy tied the control at d512 (Paloma macro delta −0.0008, inside noise) and ReMoE ReLU routing lost; Kaiyue Wen proposed CPLM Plus and Jian (Tim) Shen proposed Mixture-of-Depths routing and a decaying MTP loss through intake #9514, flagged on Discord. #10260 adds TPU paths for Belay+CPLM and Ckpt 1. boba shop cashier's #10559, mentioned last week as a kernel port, runs Belay's ReLU² experts on cudnn-frontend MXFP8 grouped GEMMs: on one B200 at the hero's per-GPU expert chunk, forward plus backward fell from 91.0 ms (QuACK) to 61.0 ms, nearly all in the backward (70 to 39 ms), and peak memory from 40.5 to 34.8 GiB at the same error; it is open against Larry Dial's branch and has run on one GPU only. Amy Sartran and David Hall asked Ben Kolligs to split the vision-language model (VLM) stack #10076; image sources #10764 and a vision checkpoint converter #10763 are now separate, with a SigLIP PR to follow before the VLM experiments merge, per Ben Kolligs on Discord.

Inference and serving. Snowball on TPU is three stacked PRs by Karan Sagar: #10535 serves one model across a multi-host slice (four coscheduled replicas for a v4-32, Ray head on rank zero; Romain Yon and Russell Power asked why Ray rather than Iris primitives), #10606 adopts the tpu-inference commit that handles sliding-window and global attention correctly, and draft #10607 calibrates H100 against two TPU geometries on frozen GSM8K and BoolQ subsets: 89/100 and 81/100 on H100, 89/100 and 83/100 on v5p-8. The Levanter-native path #10543, paged JAX inference with expert parallelism (EP) 16, ran on v4-32 but 35.5× slower than vLLM and stays draft. Romain Yon's #10704 simplifies EAGLE3 draft training for speculative decoding; an eight-H100 fit produced a servable draft, but none of its 189 proposed tokens were accepted, so quality is unmeasured. Karan Sagar's #10818 proposes coordinating OpenRouter free-tier models from an Iris CPU task for pilots.

Stella Biderman asked where the scaling-law forecasting is written up and reopened a question from last year: whether to extrapolate along equal data-to-parameter ratios or equal token counts. On GitHub, Anirudha (Ani) ran golden and untouched controls over 100 TaskTrove tasks in Harbor to test whether the validation gate in #9083 is worth building. Haoyang Su offered an H100 model-size sweep for the synthetic parity tasks in #8539. Two first-time reports fixed the README: Jason Chen filed #10765 and Harshith Gupta opened #10779.

On Discord, David Hall said that after rereading the Delphi scaling-law post he could not say how the team should have known the first hero recipe would blow up; cole suggested that jagged isoflop curves may themselves signal an unstable recipe, and Kaiyue Wen expected outliers and attention-sink behavior in the original recipe. In the SFT teacher-trace thread, Charlie R argued the 32K context cap was timing out students who copied their teachers' long summaries, and Marianna Nezhurina explained that she splits each trajectory at every summarization into separate training rows. Franziska Weindel ran Bilibird's static rubric checker over all 73,592 published tasks; the one real API key it found was a free public NASA key, so the group will focus on a cluster-run judge calibrated on a ground-truth defect set.

Seven people introduced themselves: Tejus Peri, a high-school sophomore with NASA and APL projects and a small study of bias in models; metara, a UNSW undergraduate starting a thesis on in-context learning; Jonathan Su, a high-school senior in Hong Kong with a solo ICML 2026 paper, interested in pretraining and architecture; Mathieu, an optimization researcher between applied maths and CS; Ryan (human), a UIUC freshman interested in LLM efficiency through architecture; Leon (Helimere), an MPhil student at HKUST working on RL and agent memory, who brings context on the post-training side; and Aniket (valkyrie), who runs data ingestion, training clusters and inference at Causal Labs and brings context on the cluster operations the hero depends on. In the DNA channel, Mars Gao, a Stanford postdoc starting on DNA foundation models, said hello.

The 15 shared links clustered on predicting or collapsing training stages and scaling runs: single-stage training of foundation models, meta-models fit across accumulated runs, bf16 attention gradients blowing up late in training, Mixture-of-Depths routing, and PivotRL from the Nemotron-3 report. Pinak shared Finch 1, a geospatial foundation model trained with Iris.