Marin: Week of November 2nd summary

The 65K-context switch ran on the Sunday-night option rather than the 17:00Z Monday fallback. At 03:13Z on Nov 2 hero-gradclip-step356k was cancelled after it logged a 50-step control window through step 372,049; the deploy PR #10820 had merged at 02:55Z as 6e6924fedd. Matt Wittmann requested the launch at 03:24Z: hero-ctx65k-step372k, forked in W&B from step 371,999 and restored from the permanent step-372000 checkpoint (17.16T tokens), on the same 11 racks with context parallelism (CP) 1 and expert parallelism (EP) 64, capacity factor 1.5 and qk_mult 1.66. The first step ran at about 03:34Z, and by 04:03Z the gate record on #10821 showed all six gates passed at step 372,102 on the first attempt: median 16.05 s/step against a 17.3 s limit (2% under the one-rack projection), drop fraction 4.1e-4 against 2e-3, peak HBM 127.8 GiB against 130, loss over the control window 0.020 below the parent, no grad-norm spikes, no retries. MFU rose from 29.4% at 16K to 33.3% at 65K as step time went from 14.4 s to 16.1 s. The record projected a finish near 15:00Z on Nov 5. Matt Wittmann had announced the relaunch on Discord an hour earlier, expecting about a 15% throughput loss.

One planned restart followed. At 05:13Z on Nov 4 Larry Dial restarted the run in place to pick up #10896, which keeps a permanent checkpoint every 500 steps over the last 5,000 (385,500 through 390,000, plus the final step) instead of every 6,000. The old coordinator was cancelled at about 05:03Z; the run kept its ID, output root and W&B run and resumed from its own step-382854 resume checkpoint, written at 05:02Z, at 33189166cf, the running commit plus only that PR and none of main’s 47 later commits. Storage on marin-us-east-02a stood at 760.6 of 1000.6 TB with about nine more permanent checkpoints to come. Two other operational merges landed the same days: #10917 lets the hero train past its 390,251-step schedule through a checked-in extension, and #10861 adds a daily one-rack hero canary on GB200. The Oct 26 standup notes that the remaining IMEX pods were restarted onto the version without the memory-holding behaviour behind the Oct 29 stall.

hero-ctx65k-step372k finished training at about 15:00Z on Nov 5, on the projection; W&B logs step 390,175 at 14:50Z and the final eval at step 390,250 at 15:42Z. Larry Dial’s summary on #8435: 390,251 steps, 18.0T tokens; training loss averaged 0.965 over the last 1,000 steps at 65K; MFU ended near 33%; final dropless Paloma macro loss 1.960 at the 65K training context (step 390,250) and 1.982 at 4K on the step-390,000 checkpoint, against the scaling-ladder forecast of 2.039. @mindscanlabs scored their Oct 21 forecasts against the registered target of 1.9602: forecast A 1.979 (error 0.019), B 1.935 (0.025), a straight line through the hero’s 16K evals 1.970 (0.009) and the Aug 19 pre-registration 2.039 (0.079); the hero engineering section has the detail. An automated incident record filed at 16:02Z on Nov 5 asked a run owner to verify the final checkpoint metadata and W&B state; the check posted to #10821 on Nov 6 found every stored chunk finite, the first and final saves identical byte for byte, and a full GPU restore with optimizer state reading step 390,251. David Hall announced on Discord at 19:58Z that Mochi, the 535B-A23B, had finished its main pretraining phase: long context over the weekend, SFT over the following week, and checkpoints hoped for release the week after. Amy Sartran confirmed Mochi as the model’s name.

The 262K extension began the next evening. Its starting point is hero-ctx65k-merge-hb-avg11/checkpoints/step-390251, which Larry Dial posted at 04:19Z on Nov 6: an equal-weight average of the 11 dense end-of-run checkpoints, about 0.003 lower on every eval at 65K, saved with full optimizer state so training resumes from it directly (the hero engineering section covers the merge and the recipe choice). #11056 tracks the plan: two recipes trained one after the other from that checkpoint, 3,240 steps or 149.5B tokens each, to step 393,491, on 11 racks at CP4 × EP16, LR falling linearly from 5% to 1% of peak, with documents over 65,536 tokens at 4× sampling weight from #10980. Matt Wittmann requested the first launch at 21:35Z on Nov 6: hero-lce-global184-step390k at e0bc917ee5, the commit of #11041, forked in W&B from step 390,250. It restored the merged checkpoint and reached step 390,260 at 29.9 s/step on its first attempt; no rehearsal had run, so this was the first training run of the recipe and commit. GO came at 00:10Z on Nov 7 at step 390,526: median 29.06 s/step, drop fraction 3.0e-4, largest device memory 180.4 GiB after the first eval, one grad norm of 48.49 clipped to 1.0, no retries. It reached step 393,491 with its final checkpoint complete by 00:40Z on Nov 8, at MFU near 34% per W&B. Matt Wittmann described the two recipes on Discord.

The second finalist took over at once. At 00:39Z on Nov 8 Matt Wittmann launched hero-lce-log65536sq-step390k from the same merged checkpoint at 0454bdd609 on the branch of #11044, which merges only if this recipe wins; the launch record names the branch, not main, for recovery. GO came at 03:10Z at step 390,530: median 29.06 s/step, drop fraction 3.1e-4, largest grad norm 20.14 at step 2 as its 100-step arm predicted, no retries; only the stop path is untested. At the week’s close it was running toward step 393,491, due to finish early on Nov 9, and #11056 schedules the comparison of the two finalists for Monday Nov 9. Release preparation also moved: #10890, merged Nov 4, streams native hero checkpoints to BF16 safetensors on one CPU host and uploads them to checkpoint branches of a private Hugging Face repository.

The 65K phase ran clean against the gates Matt Wittmann set in #10821. The deploy PR #10820 merged on November 2 and hero-ctx65k-step372k launched at 03:25 UTC from permanent step-372000 on the context parallelism (CP) 1 × expert parallelism (EP) 64 mesh, with qk_mult 1.66 and capacity factor 1.5. By step 372,102 every gate had passed on the first attempt: median step 16.05 s against a 17.3 s ceiling (2% under the one-rack projection), drop fraction 4.1e-4 against 2e-3 (a quarter of the 16K parent's), peak HBM 127.8 GiB against 130, cross entropy at most 1.007 over the first ten steps, and loss 0.020 below the parent over the 50-step control window. MFU rose from 29.4% to 33.3% at the switch. After the phase ended on November 5, the final checkpoint was checked: all stored chunks finite, first and final saves byte-identical, and a full GPU restore with optimizer state read step 390251 with no dtype or tree mismatch. The relaunches and finish time are in the hero status section. Amy Sartran's #10861, merged November 3, adds a daily one-rack GB200 canary that restores step-372000, trains 35 updates and runs one dropless eval; its first run gave 15.52 s/step and 34.52% MFU against 15.57 s on Matt Wittmann's one-rack test, and it flags a step-time change above 1.5%.

Larry Dial posted the final evals on November 5: 390,251 steps, 18.0T tokens, dropless Paloma macro 1.982 at 4K context (a re-eval of step 390,000) and 1.960 at the 65K training context, with Uncheatable macro 1.503 and 1.478. MFU over the run rose from about 21% to 33% through six kernel cutovers and the two context switches. The August 19 preregistered ladder forecast was 2.039 at 4K, under the original mix with no context extension, so the hero came in 0.057 below it; the contaminated PTB reads from #9874 account for up to 0.019 of the macro. @mindscanlabs scored their October 21 forecasts against the registered target, the last 65K eval of 1.9602: forecast A (ladder plus the hero's own evals, 1.979) missed by 0.019 and the outcome fell 0.0045 below its 90% interval; forecast B (ladder only, 1.935) missed by 0.025 and contained the outcome; a straight line through the 16K evals missed by 0.009. Using Qirun Li's packing measurements, the 4K-to-65K offset at the end was 0.0227, so against a 4K-equivalent 1.983 forecast B was off by 0.048 and the August forecast by 0.056. The unsealed formulas match the SHA-256 digests posted before the outcome. Qirun Li's own preregistered median of 2.035 in #9977 has not been scored on its thread yet; Will Held noted on Discord that the forecast was for 4K loss and that the team will measure against it retrospectively.

Eval design moved on three fronts. #10849, opened November 2, lays out 15 eval-only runs of the step-372000 checkpoint across 4K, 16K and 65K, truncated versus right-padded final windows (#10711, still open), and five qk_mult settings; the token counts show truncation drops 1.17% of Paloma tokens and 3.79% of Uncheatable tokens at 65K, with PTB losing 24.8% and WikiText-103 21.3%, and the runs waited all week because every GB200 NVLink domain was allocated. Matt Wittmann's #11054, merged November 6, stops eval_dropless/macro_loss from averaging in a loss of 0 for ptb and wikitext_103, which are shorter than one 262K sequence; the macro rises by 16/14 to about 2.05 at 262K. On checkpoints, Larry Dial's #10896 (merged November 4) keeps a permanent checkpoint every 500 steps over the final 5,000, and he used them for a weight-merging study: an equal-weight average of the last 11 checkpoints lowers loss by about 0.003 on all 23 eval sets at 65K (clean Paloma 1.7515 → −0.0030, Uncheatable −0.0033), worth roughly 7% more compute on the ladder law; hyperball rescaling changed nothing because the plain average already sits at 0.9995 of the sphere radius. The merged weights were saved with step 390,251's optimizer state as hero-ctx65k-merge-hb-avg11/step-390251, reloaded and verified bit for bit. For release, David Hall's #10890 (merged November 4) streams native checkpoints to BF16 safetensors on one CPU host and uploads them to branches of a private HF repo: a step-378000 check converted 55 shards (70.8 GB) in 488 s at 95.6 GiB peak RSS with every SHA-256 matching the GPU export; #10898, still open, batches shards eight to a commit so a 193-shard checkpoint needs 25 commits under the Hub's 256-per-hour limit. #10556 (merged November 6) skips weight gradients for frozen short-convolution weights, and Will Held's #10856 adds sticker-drawing vibe checks sampled from step 386,001.

The CP and EP work for 262K landed through the week. #10618 lets the production launcher run CP, with each rack's 64 GPUs forming 64/CP expert shards; #10610 makes the FA4 zigzag sharding configurable, since zigzag pays at 262K but slows 65K. #10877 (merged November 3) reorders the compact mesh as data, expert, context so the dropless eval mesh is a reshape of the training mesh; the old order at 65K CP4 created NCCL communicators for groups training never used, 7.4 GiB per GPU outside the allocator pool, and crashed the eval. It supersedes #10709. That was not quite enough: #10994 (merged November 6) found that at 262K, CP4 × EP16 on one rack, the weight move to the eval mesh still ran a collective on a new 16-rank communicator, adding 5.63 GiB of buffers and leaving the next train step 1.62 GiB short. Renaming the PartitionSpecs keeps every shard on its device; an 11-rack proof across 704 devices added no non-pool memory, with 2.08 GiB free on the tightest device. #10923 (merged November 5) adds optional Inkling-style log scaling of global-layer queries, (1 + 0.1 ln(n/floor)) on top of qk_mult, and #10986 (open) makes the slope a required field and adds a power. #10917 (merged November 6) lets the hero train past its 390,251-step schedule through HERO_SCHEDULE_EXTENSION, moving the learning rate linearly from 5% of peak to a stated ratio while keeping rates through 390,251 bitwise identical. #10980 (merged November 6) reads length buckets from the existing Harrier token caches so documents above 65,536 tokens can be sampled at 4× weight, 43.9% of tokens against 22.2% at 1×, without a second store.

The 262K recipe was argued out in the open. Larry Dial listed the options on November 2 (SSMax-style scaling on global queries past 65K, global at 2.11 or 1.88, all layers at 1.88, or keep 1.66) and David Hall asked why touch the local layers at all. The research record is logbooks #37: seven 100-step arms on 11 racks did not separate the recipes on held-out text (clean macro spread 0.0004 against checkpoint jitter of 0.0002 to 0.0007), though some retrieval differences on MRCR survived correction, and the merged start's gain fell from 0.003 to 0.0017 after 100 steps. The decision, tracked in #11056: two finalists trained one after the other from the merged checkpoint, 3,240 steps and 149.5B tokens each to step 393,491, learning rate 5% to 1% of peak, CP4 × EP16 with 176 sequences per step at about 29.9 s. First, Global 1.84 (#11041, merged November 6): qk_mult 1.66 with long scale 1.108 on global layers only, chosen because the no-training sweep put it 0.0024 below the all-layers value at 262K, though no training arm or rehearsal ever ran it. Second, log floor 65536 squared (#11044, draft): its 100-step arm had retrieval loss 0.0014 below global 1.88, but its checkpoints cannot be exported or served until vLLM and SkyRL support a position-dependent scale. The branch merges only if it wins the comparison on offline Paloma, Uncheatable and MRCR, planned for Monday November 9; the winner feeds SFT. Run progress for both finalists is in the hero status section.

No identity-fixed release shipped this week, and none was chosen. Karan Sagar's #10695 still names A13, the short SFT on Will Held's identity data plus v0.1 replay, as the lead candidate and A3 as the fallback. The research record moved to marin-logbooks on Nov 4, with a 15-page report, a reproduction guide and archive branches for the 11 experiment-only drafts that were closed the same day; they include the interpretability harness #10689, the identity SFT recipe #10690, the precision gate #10691, the LR0 probe #10693, the MoE-drop diagnostics #10694, the scorer #10701, the spending-gate eval #10702, the 16-update milestone #10708 and the paused identity on-policy distillation (OPD) #10732. The four most promising checkpoints joined the Snowball family tree in #10922, merged Nov 4, after #10719 and #10599 made reviewed PRs to graph-data.json the only route into the tree and closed #10598. The interim eval-policy tables posted Nov 5 put A13 beside v0.1 on all 26 benchmarks: A13 is ahead on SWE-bench Verified (28.1 vs 25.2), BFCL-parity (Berkeley Function Calling Leaderboard; 26.8 vs 23.7) and FinanceBench (62.0 vs 46.0), behind on HumanEval+ (84.8 vs 90.9), MBPP+ (72.2 vs 77.0), GSM8K (90.6 vs 92.8) and the TBLite recovery cells (12.7 vs 22.5); several agentic cells carry coverage flags. Karan Sagar closed the merge-ladder record #10700 and the TPU weight-merging loop #10270 on Nov 8, since the identity work now runs through SFT with replay, and summarized the interpretability finding from #10696 on Discord: ChatGPT comes from late MLPs, Marin (with a system prompt) from late attention, and a steering vector either did nothing or said Marin everywhere.

The Levanter numerics stack behind the identity recipe is still in review; nothing from it merged. On Nov 6 Karan Sagar collapsed three settings into one, TrainerConfig.fp32_fields, in #10683 (router biases stay float32 from HF load through export) and ported the rest of the stack onto it: #10684 (float32 router dot plus a route observer), #10685 (MoE capacity as a checked setting, with drop counts) and #10687 (float32 SiLU and sigmoid) are open for review, while #10686 (float32 combine) and #10688 (torch-order attention) stay draft as uncertain layers. The audits #10706 and #10707 and the identity scorer #10884 were rebased onto the stack on Nov 8, also as drafts. A new draft, #11061, adds a cluster check that scores the training forward Levanter actually runs against vLLM at every position of 64 prompts, since the serving-parity check only looked at the last position and missed the capacity drops, bf16 router bias and bf16 combine this stack fixes. Benjamin Feuer's #10492, which honors packed segment IDs in Snowball attention and keeps sharding through context-parallel SFT, merged Nov 3, as did #10597, which lets merges widen BF16 router biases to FP32.

Snowball on TPU now has a merged, nightly-checked serving path. #10854 (Nov 4) pins a tpu-inference commit whose native layer schedule keeps seven of Snowball's 26 layers global instead of windowing them all at 2048 tokens; on a teacher-forced probe the pooled mean absolute log-prob difference from the H100 oracle was 0.01598, and a MATH500 run with the schedule fixed moved from 0.878–0.882 to 0.940 against H100's 0.944. #10535 (Nov 4) serves one model across a v4-32 slice through Ray, with four coscheduled replicas and the endpoint published from rank zero. #10607 (Nov 5) compares frozen next-token probabilities on H100 and v4-32 at TP16 rather than GSM8K scores, a change Karan Sagar made after Romain Yon asked whether the existing Levanter/vLLM log-prob tests would be a stronger check; the live run passed with maximum probability errors of 0.074 on H100 and 0.059 on v4-32 against a 0.125 bound. #10860 (Nov 6) runs the parity arms nightly across CoreWeave H100 and Marin v4 from one workflow.

Ahmad Qamar's Snowball RL stack remains draft. #11069 makes snowball_rl the launcher: learning-medium trains the 09.21 SFT checkpoint for 25 synchronous REINFORCE leave-one-out (RLOO-N) updates on 40 H100 at LR 1e-6, scores a 1,500-task locally graded test panel greedily before and after, per #11032 and #10926, and judges the run against pinned control replicates through #10747. #11047 adds 100-update learning-long and data-long presets, #10946 adds timings and asynchronous throughput, and #11033 and David Hall's #10863 publish runs to the RL progress dashboard. The #10946 audit runs give the first numbers from the gated presets: speed-short held engine throughput at 6,026.6 tokens/s/GPU against a control mean of 6,009.4, and the 25-update learning-medium run passed its quality guard but was labelled regression, because trainer-versus-engine log-prob gaps grew past the control tolerances. #10681 tightens that trainer–sampler match for Grug: SkyRL's config now requires the four values vLLM otherwise fills with defaults (query/key scale 1.57, window 2048, context 65536, four experts per token), and a tiny-checkpoint test holds the mean sampler–trainer log-prob gap to 0.000421. Xiaoyuan Wang found that the data-short gate fails its guard on 6–8% of random control-pool selections against a 2.5% target, because replicates share the task seed, and that a --resume-from run that changes --eval compares best-checkpoint scores across different task sets. David Lundgren's #10626 (Nov 4) turns on the reasoning-span penalty in the quick RL loop by default; the double-think rate fell on both panels but neither paired interval excluded zero. #10907 (Nov 6) registers the thoughtcrime evaluation set of double-think prefixes, and #11050 (Nov 7) strips the trailing <|eot_id|> that Snowball leaves on every SkyRL reply.

Three pieces of analysis. David Lundgren's preregistered replay #10850 ran and closed on Nov 2: re-rendering the first 50 steps of the 10.05 SFT in the #10519 span format left the teacher-forced re-close hazard unchanged (it climbs from near zero to 0.45 between steps 7 and 20 in both arms), so the renderer is a format fix only and #10851 closed unmerged. His follow-up on #10400 finds the stray <|end_think|> is the model closing its visible prose before a tool call, a habit SFT picks up in its first 20 steps and slowly trains back out. Benjamin Feuer corrected the v0.1 eval-policy release: the checkpoint shipped no sampling defaults, so v0.1 had run at temperature 1.0; re-run at T=0.6/top-p 0.95, TriviaQA rose 0.127 and MRCR 0.069, no benchmark fell, and v0.1 stays first FLOP-controlled. Ricardo Olmedo's scaling-ladder study is preregistered: the Nov 1 design fits seven SFT budgets on the three Snowflake rungs to predict Snowball's SWE-bench pass@1 at 7.55B tokens (39.71%), David Hall released the three pre-SFT Snowflake base checkpoints on Nov 3, and a Nov 5 amendment adds the 2.25B cell before any budget cell trained. Karan Sagar wrapped up the agentic SFT shootout #10213, archiving five experiment-only drafts and keeping the recording relay proxy #10887 and recovery rows #10888 open as reusable parts, and told the #sft channel that the trace source decided the result and no SFT method moved agentic scores much. On Discord Atula Tejaswi asked whether Snowball's reinforcement learning with verifiable rewards (RLVR) trajectories are stored anywhere.

SkyRL, Harbor and agent evals. The Harbor-through-Shellbox arc closed: Russell Power's #10628 merged on November 5, routing every Harbor task command and grading script through Shellbox adapters for bubblewrap, Daytona, QEMU and ShellSim and deleting 42,984 lines of Harbor's own environment providers. Benjamin Feuer stacked follow-ups on it: #10735, still open, stops agent process trees before verification, and #11080 proposes cutting another 27,493 lines of unsupported Harbor agents so only Pi, Mini-SWE-Agent, Codex, OpenCode, Terminus-2 and Terminus-Kira remain. Pi sessions now archive as ATIF trajectories through #10972, with reasoning and tool-error flags preserved by #11072, so Berkeley Function Calling Leaderboard (BFCL) and BixBench Pi runs get an episode viewer in EvalDash. #11003 retries Daytona snapshot reads after a single 502 cost one grade in a fixed 100-task evaluation. On the trainer side, David Hall extracted pieces of the Hero RL branch into main: #10958 validates Megatron dense and expert grids separately, #10960 writes unbounded FineStore archives for route captures, and #10376 offloads recompute inputs to pinned CPU memory, which bought 32.8% more valid tokens per second per GPU on 192 H100s. #10844 lets installed agents recover from context overflow with an HTTP 400 instead of a fatal SSE error. David Lundgren's malformed-reasoning grading and reward penalty in #10624 and #10625 merged, closing #10520; his --max-staleness-steps draft #10996 reports 79 s per update at staleness 2 against 170 to 203 s synchronous. Still draft: Romain Yon's score-centering stack #11034 and #11066, Will Held's FP8 expert-block weight sync #10893, and the reproducible Grug prefix-caching profile #10760, whose fixed100 qualification is recorded in #11007. The SkyRL nightly failed again on November 7 and 8 under #10910, and #11077 tracks cw-rno2a tasks that stall after their final output.

Eval policy, harness and EvalDash. Russell Power's #10757 merged on November 4, curating 96 Evalchemy benchmark declarations for 31 benchmarks as canonical TaskSpec rows; its successor #10957, which removes vendor/evalchemy entirely and has passed a 32-of-32 Qwen3-8B smoke, is open. Grading corrections under the eval policy #10197 kept landing. Benjamin Feuer's CruxEval fix #10970 grades the first span that holds a real answer, after 78 Snowball items had extracted the literal and; #10733 and #10876 make NUPA preserve thinking markers and reject concatenated repeats; #10672 applies the publication metric policy in the shared reader; #10901 stops Qwen3.5 base at <|im_end|>; and #10965 refuses policy launches whose records land where EvalDash cannot see them, after 26 Snowball runs went to the wrong bucket. Xiaoyuan Wang's IFBench checker fixes in #10423 merged after the repeat_span bug had failed every correct answer; her canonical-grading follow-up #10927 and verifyit IFEval checks #10928 are open and tracked by #10828, and she asked on Discord that #10927 be reviewed first for the next policy version. Chris McConnell, who introduced himself as the new owner of eval policy, opened SimpleQA Verified in #11063; Tim O'Donnell's PDBThink #11059 and Gonzalo Benegas's GenomeQA #10984 are open, while LAB-Bench v1 was closed as not planned in #10575 and #10582. Open bugs: a mid-eval timeout discards every SkyRL rollout, #10837; a stalled Harbor trial blocked six benchmarks behind it, #10962, which the open #10963 bounds with a 90-minute attempt deadline; and Jeff Hammerbacher found five defects in Levanter's BPB computation, #10968. Benjamin Feuer's write-ups #10912 and #10935 found Qwen3.5-35B-A3B-Base jumps from 0.05 to 0.55 on SWE-bench Verified once <|im_end|> is a stop token.

Taskforge, Forge and task curation. Mark Muchane's Taskforge stack landed its first four layers on November 2: sandbox factories and the Iris image builder in #10293, TaskSpec assembly and lowering in #10294, typed JSON proposals in #10295 and deterministic triage in #10296, with #10866 moving image-job credentials into Iris secrets. The build, validate, driver and policy PRs #10297, #10298, #10300, #10301 and #10847 are still open, and #10302 was re-cut into smaller pieces. In parallel he started an effect-driven rewrite called Forge: the workspace package #10998, effects and replay #11000 and workpiece contracts #10999 are open; three small Forge PRs merged, #11005, #11006 and #11008; and #11036 is a read-only review aid. The November 2 n=25 curriculum run wrote 49,975 proposals across 1,999 capabilities, browsable in the Marina viewer of #10868, open. #10921 lets RolloutEngine run without turn or output caps for these sessions. Community task sources arrived. Robert Joseph opened a Lean v4.34.1 proof grader with ProofNet-Verified in #10879 the day after Benjamin Feuer filed #10842; Matt Wittmann offered to coordinate the Lean effort and laid out the path from source PR to curation run, and Robert Joseph reported a 7.6k-row NuminaMath-LEAN source ready behind it. Siddhu (flufcloud)'s QASPER tasks in #11016 drew a review note that 206 of 208 wh-questions are labeled unanswerable. Zachary Nichols's draft BioTasks pilot #10983 ran 305 combined tasks through Taskforge. #10919 lets Atlas publication tolerate up to 2% unavailable judgments; #10924, which surfaces declared counts, is open.

Belay. Marin Belay moved off a maintainer's laptop. Larry Dial's Marina Belay app, #10829 at 5,775 lines, gives the idea queue public pages, a Status checklist and an admin view where maintainers approve or decline ideas. A Loom belay profile from #10824 runs the sessions; #10833 scopes a dedicated CoreWeave user to Belay's data and run prefixes, and #10931 adds explicit bucket-policy denies after a live check showed the key could still read hero checkpoints; #10934 switched the watch on. Admin follow-ups merged the same day: #10937 syncs every watch pass, #10939 declines with a reason, #10943 attaches approval instructions the session must follow, and #10940 removes the old GitHub intake workflow, with #10941 marking its token for removal. The first production runs all failed at startup because the key could not read the tokenizer mirror or compilation caches; #10955 fixes that and is open, as are re-queueing escalated ideas in #10951 and domain-wide approvers in #10954. Four ladder ideas were posted to #9514 on November 5, including CPLM Plus, whose author shared a faster head on Discord. David Hall suggested Belay should become self-hostable, and ravwojdyla-agent's open Data Belay #11019 extends the ladder to test whether a new HF dataset improves held-out evals.

Iris, cluster ops and canaries. A hard week for scheduled runs. #10774, which saves completed Grug steps on preemption, added a required IRIS_TERMINATION_GRACE_MS read that deployed workers did not supply, so on November 4 the TPU, Datakit, Grug multislice and hero GB200 canaries and the SkyRL nightly all crashed at startup: #10900, #10902, #10904, #10915 and #10916. #10911 restored a 10-second default and #10913 was closed as superseded. The hero GB200 canary had separately waited 340 minutes behind Kueue gating because the monitor treated a RUNNING job as started, #10897; Amy Sartran's #10909 keeps the queue deadline until a task runs, noting 67 of 200 RUNNING jobs on the controller had no running task, and #10975 and #10977 report capacity-skipped canaries as skipped in Slack and Grafana. From November 6 at 21:59 UTC one pod whose exec failed made every cw-rno2a controller tick raise, leaving 80 tasks across 8 users assigned with no pods, #11075; #11076 was rolled out on November 8, and Russell Power's follow-up #11079 moves gVisor logs off the root RAM disk after 9,970 orphan log directories filled it. The Grug multislice smoke still fails on missing us-east5 dataset pins in #11042, #11074 and #11084; Matt Van Horn's #11070 is the open fix. Karan Sagar closed #10768 after #10769 and #10771 merged and #10770 was closed without merging. @HRL9000's Finelog rows for controller jobs and attempts, #11037, and job tags, #11038, are open. Louisa H's egress report #10800 reads logs again after two months of empty scans.

Datakit, Zephyr and Finelog. The canary failures produced Zephyr fixes. #10976 keeps one idle worker so a shard requeued after a heartbeat expiry can run; the November 5 smoke had stopped at 105 of 106 shards, #10974, and tier 2 repeated it in #11045. ravwojdyla-agent's #11012 retries a shared pool's failed start, replacing both #10853 and #11010. Benjamin Feuer's #10899 gets Polars Parquet scans working inside Iris pods, where storage keys arrive as files. Amy Sartran retired legacy exact dedup including the paragraph mode in #10886, which Rafal Wojdyla voted to remove. Himanshu Kriplani's Arrow-based centroid sampling, #10862, open, cuts a shard that was OOM-killed at 8 GiB to 1.35 GiB. Russell Power's Finelog work merged and deployed: on-demand index opening in #10936 and bounded reads without cache-visibility contention in #10929; Karan Sagar's #11087 and #11086 propose raising the 10-second query deadline after a 60-day log pull lost 58% of its windows. Decontamination: Ricardo Olmedo reported in #10832 that 404 of CalibForge's 5,431 tasks are Terminal-Bench 2.1 variants, #10839 drops the 229 matching relay episodes, and Ricardo Olmedo argued on Discord for the distinction between sampling a benchmark's distribution and leaking its test items. AAII v5 and eval-policy evals in the decontam corpus, #10938, and an item-level marking rule, #10836, are open. #10639 merged November 8 to verify multipart S3 copies on CoreWeave after three model mirrors failed that day, #11081; #11082 proposes the simpler direct-write design. Michael Ryan posted the weekly sync for Common Crawl curation in #2351.

Levanter and training infra. David Hall's #11028 fixes fused cross entropy under tensor parallelism: with a 16,384-token vocabulary on four H100s, every target at or above the local width of 4,096 had produced an infinite loss. Benjamin Feuer fixed FA4 singleton-batch metadata in #10796 and a roundoff that moved one document between mixture components in #10793, and #10798 preserves the reference generation_config.json on HF export; his open #10906 would write a complete one with sampling defaults for every export. #10713 replaced embedded CPU test scripts with marked multi-device tests. Amy Sartran's NFS checkpoint locking, #10969, was closed after she found transferring to CoreWeave faster than writing to the AMD clusters' NFS. An explicit-array Llama, #11029 and #11030, is draft; Ben Kolligs's vision converter split and image sources, #10763 and #10764, and Karan Sagar's final-checkpoint-only SFT default #10782 are open. SFT data hygiene got a community audit: Min Li reported on Discord, and #10865 records, that 32.1% of Open-SWE's 49,925 successful traces quote a prompt section NVIDIA's export dropped; #10881 restores it and drops 8,614 traces, while #10813 keeps only resolved trajectories, with Will Held asking for an SFT ablation to settle the question. #10988 defines a versioned candidate mix keeping 99,453 of 268,164 Nemotron science rows. #10519 closed with exactly one reasoning span per /think turn, and Zoey Li's report in #10320 that 1,142 of 11,817 RLVR1 IF rows reward letterless answers drew RL runs showing SkyRL-reward policies learned the checker defects.

AMD kernels and Grug MFU. Amy Sartran's MI350X/MI355X work toward #9812 merged three step-time cuts for June Snowball. #10345 takes the ring_filtered routing-weight gradient from the down projection, saving 16.5 GiB per expert parallelism (EP) 8 rank and 2.9% step time, so the production batch of 128 per node fits where main ran out of memory; #10343 replaces a slow scatter-add with a gather, 3.0% together with the next item; and #10341 uses a 4,096 vocab block for the cross-entropy backward, 0.7 to 2.1%. Opt-in collective-overlap XLA flags in #10673 cut the step 13% across two MI350X nodes, and #10217 lets the Pallas flash kernel read precomputed FA4 bounds. The current best is 16.61% MFU on MI355X. Chao noted the Pallas-Triton backend is abandoned upstream, and the drafts #11053, #11052 and #11046 move to jax_triton on JAX 0.12; #11051 already swaps pl.dot for plgpu.dot. A JAX-AITER flash attention, #10859, is draft, and TheRock container scripts with RDMA, #10858, which made a cross-node all-to-all 4.5x faster, are open. Larry Dial posted heuristics for tuning architecture to AMD hardware.

Inference, serving and Knotmeter. #11031 restored the serve dashboard's tools by moving ShellSim to its public catalog after the old host returned 502s. Open: Rafal Wojdyla's Parallel-backed web search tool #11068, Chris McConnell's Report-issue button that drafts a GitHub issue from a chat transcript, #11055, and HF weight pre-mirroring before GPU reservation, #10277, which Karan Sagar's Finelog analysis in #9233 motivates with 60 days of cold-start timings. Romain Yon's native DP session routing #10956 sent 38 conversations across 16 GB200 ranks with every later turn returning to its rank; it, the JAX-vs-vLLM serving score comparison #11021, and Knotmeter, an InferenceX port for saved Snowball and Hero results in #10646, #11020 and #10953, are drafts, with Russell Power asking whether it should be a Marina app. #10818 closed with the finding that OpenRouter's free tier caps at 1,000 requests a day; #10237 was a duplicate of a fixed bug. zoe110501's Delphi EMA-Gram experiment #10784 reported its seed-0 campaign with the tuned comparisons still missing.

Loom, Marina, Echo and Filer. Rafal Wojdyla merged a Google Doc profile for Loom sessions in #10918 and then proposed replacing it with a Loom-native artifact doc in #11060, open and waiting on a Loom release; the gdoc change-watcher #11015 was closed for the same reason. #10889 fixed macOS activation after the November 3 Loom rollout, #10892 made the Loom infrastructure tests run in CI for the first time, and #10823 disabled MPTCP on the host, which Matthieu Baerts said gives a false sense of protection since the bug is in the TCP stack. Benjamin Feuer unblocked Marina: #10964 serves applet routes with dotted names, which had hidden EvalDash pages for 21 of the 37 eval-policy models, and #10966 added rolloutengine to the image after the production deploy failed. The draft #10973 would release maintained apps on merge, and #10869 asks for named public apps. #11009 gave Filer history, favorites, grid views and text previews; #10756 added work-log commands and an echo-log skill to Echo; #11011 tightens holistic review of design boundaries.

CI, dependencies and security. Amy Sartran made the "Marin - Unit" check faster: a blobless clone in #11004 cut the selector's download from about 1 GB to 75 MB, Romain Yon did the same for six more workflows in #11018, round-robin sharding in #10979 evened out runners that had taken 360 to 380 s against 190 to 250 s, and #10995 stops non-Python edits under experiments/ from running all 282 marin test files, which had happened on 1 in 9 commits. Time-balanced sharding, #11002, was closed after it failed to beat round-robin, and parallel Evalchemy tests, #10997, were closed because #10957 deletes the suite. A test audit fixed twelve tests that passed regardless of the code, two of them now expected failures over real bugs, in #10883; deleted 3,482 lines of tests for dead code in #10880 and duplicate Iris tests in #10992; and #10993 is open. Louisa H's #11013 skipped a 7-to-9-minute font install that was timing out the Iris smoke test. Security bumps merged: OAuthLib and Transformers in #11040, pyasn1 and the PyPI publish action in #10949, npm minimatch in #10816; Tau3 sandbox npm alerts are now accepted risk under #10947. Russell Power's shared OpenFGA grants, #10950, remain open, and two IAM grants for new developers merged, #10878 and #11088.

Housekeeping. Thirty-five old issues were closed as stale, among them the PDF and DOCX pipelines #7616 and #7736, the Marin 8B safety-evaluation epic #7931 and the ablation-catalog proposals under #7966; Isaac Hodes posted the week-of-October-26 standup in #10944.

Two newcomers turned Discord threads into pull requests. Min Li audited all 49,925 successful Open-SWE traces in the 10-05 SFT mix and found that NVIDIA’s released prompts omit a “Solution Originality” section the teacher saw. Benjamin Feuer tracks it in #10865; Min Li’s #10881, under review by Benjamin Feuer and Will Held, restores the section and drops 8,614 traces that guess at hidden tests. Robert Joseph opened #10879, a Lean v4.34.1 proof grader with ProofNet-Verified, after Matt Wittmann laid out how a task source moves from PR to the quick RL loop; by Sunday he had 7.6k NuminaMath-LEAN rows ready and had found Nemotron solves all 555 Lean statements in RLVR1. Xiaoyuan Wang’s #10423, seven IFBench checker fixes, merged Nov 3, with #10927 and #10928 queued behind it.

Contamination and base models ran through the week’s debates. After Bilibird asked whether CalibForge’s train-fasttext variant counts as contamination, Ricardo Olmedo argued that sampling from a benchmark’s distribution is fair while perturbing its test items is not, citing #10838, and questioned holding Terminal-Bench Science and GDPval out of training. Benjamin Feuer’s #10935 showed Qwen 3.5 35B-A3B base gives SWE-bench and Terminal-Bench signal after a stop-token change alone. Yoav Artzi asked for the pretraining corpus and mixture weights; David Hall said Marin cannot rehost the data and is building a metadata-only version with Will Held.

Twelve people introduced themselves: PhD students Alex (Harvard, optimization and pretraining), Robert Joseph (Caltech, formal verification) and Kang (Rice, Hyperball-related work, offering help on hyperparameter schedules); Ousema (École polytechnique, mixed-integer programs and network verification); Jaagat (Stanford undergraduate, drug discovery and self-driving vision-language-action models); Arsalan Zafar (LLMs for local agentic work); Nico B (Rhizome, agent-to-agent messaging); August (full-stack engineer, wants an open pool of SFT and RL tasks); Cheryl (Elasticity Institute, GovAI and Yale) and JS Denain (METR), who are studying Marin’s research process through its public W&B and GitHub to measure how AI speeds up AI research; Chris McConnell (Open Athena, formerly Meta’s Llama team, now on evals and eval policy); and Allen Hart (Warwick, terminal-bench-science), who is curating a 200-task lite set and wants RL tasks that need the same skills without copying the items. Tim O'Donnell set the first weekly Marin for Science meeting for Nov 10.

Reading leaned toward frontier tech reports, Mistral Large 4, Inkling 2 Small (whose 13T-token pretraining Larry Dial said moved his prior toward missing something) and Reflection’s Beam, plus Greg Lindahl’s Common Crawl paper comparing content quality with CCBot’s crawl-budget ranking.