Marin: Week of September 7th summary

The week opened with the pooled-wave hero hero-wd-gate-router-p02-step58k at step 70,333 and 3.25T tokens on September 7, train loss about 1.23, 21.5% MFU and 2.55M tokens/s at 19.8 s per step. On September 9 Matt Wittmann posted a plan in #8506 for a second attempt at the ragged all-to-all expert parallelism (EP) transport: merge #8959 (QuACK 0.6.4) and #9043 (ragged transport and fp32 weights on device, reverting the #8884 rollback), force a handoff checkpoint on the live run, let the old run train 200 steps past it as a paired control, then launch the new run from the same checkpoint and gate on loss within bf16 noise of the control after the ~0.006 nat dropless offset, drop fraction below 1e-3, MFU at or above the control, and no hang. The checkpoint step-81716 (5.36 TB) was forced at 18:26Z; the old run was cancelled at step 81919 at 19:33Z and hero-ragged_a2a-ep-step81k launched from main, also carrying the one-executable-per-rank fix #8911, the balanced ragged device kernel #8976, and the cross-entropy tile and FA4 padding fixes #8960 and #8961. Matt Wittmann noted on Discord that the step counter would rewind 200 steps to 81716 for the replay.

The first attempt matched the control for 43 steps (loss −0.0061 nats, MFU p50 23.5%) and then hung at step 81759 in the #8870 pattern: one rack at half CPU, every cross-rack AllReduce with one member exactly three operations behind, no error from any rank. Three more attempts on the same commit hung before completing a step, on rack indices 9, 1 and 2, with py-spy showing every rank inside the jitted train step. Attempt 6 changed one thing, swapping the PJRT wheel for one whose device kernels were compiled against NCCL 2.30.7 headers to match the 2.30.7 runtime, and completed the 200-step window at 22:15Z: loss −0.0064 nats against the control, capacity overflow 1e-4 vs 3.4e-2, MFU 23.3% vs 21.5%, peak HBM 103.6 vs 136.3 GiB. #9062 merged the released build of that wheel and the hero was redeployed from main at 22:50Z as hero-ragged_a2a-nccl2307-ep-step81k, restoring step-81716 a third time so the lineage stays on main commits; it passed step 81916 at 23:57Z, reproducing the trial to within 0.0005 nats, and became the hero from there. On #8870 Wittmann put his confidence that the header change fixes the hang at about 60%, noting that only two of the four clean eleven-rack windows tested it. The deploy-hero-change playbook #8873, which encodes this handoff-checkpoint-and-paired-control procedure, merged September 12.

September 10 brought two restarts. After 1,590 clean steps the run hung at step 83308 on rack index 0 with the same signature; the progress watchdog killed the gang at 07:10Z, Iris restored step-83305, and the run was past the hang step by 07:16Z, 21 minutes and three replayed steps after the stall. An hour later rack index 9 logged CUDA_ERROR_NVLINK_UNCORRECTABLE and took the gang down again; the retry replayed 83305 to 83487, about 55 minutes, because the previous attempt had not reached its hourly save. The faulted GPU's NVLink link 14 had shown 505 lifetime link-recovery events in a hand-run sweep five days earlier, and #9086 asks for per-link NVLink counters, since the DCGM exporter on the cluster emits none. Earlier that morning a one-rack diagnostic had verified the restore path for the run's own master-less checkpoints, which no restore had yet exercised. From there the run stepped without interruption for almost three days; W&B history shows one more gap of about an hour starting 00:54Z on September 13 with some 30 steps replayed, which has no entry in the status thread, though Wittmann posted an updated incident diagnosis to #8870 at 01:26Z.

Alongside the restarts, a Discord thread on the Paloma aggregate rising after the weight-decay restart ended with Larry Dial judging the train loss healthy and no intervention needed; the subset analysis and the later gradient-norm thread are covered in the 535B-A23B tracker section above. He also posted step-78,000 prompt completions #8827, including one that solved a bowling word problem and then repeated the answer until the 4,096-token limit. The week ended with the run at step 98,362 on September 13 at 13:56Z: 4.54T tokens, train loss about 1.21 as a six-hour mean, 23.1% MFU and 2.73M tokens/s at roughly 4,670 steps a day, about 28,000 steps and 1.29T tokens for the week.

The hero went back onto the ragged all-to-all expert parallelism (EP) transport this week, after last week's rollback. Matt Wittmann landed the three fixes the rollback had queued: #8911 initializes the step counter on the training mesh, so each rank compiles one train-step executable and registers one NCCL symmetric window instead of two (64 rather than 128 across a rack), closing #8861; #8976 moved the PJRT wheel to a balanced ragged device kernel with 128-thread CTAs; and #8959 upgraded QuACK from 0.6.1 to 0.6.4 to pick up the grouped-GEMM scheduler fix whose corruption was reproduced on GB200s. #9043 then restored ragged as the hero's backend with fp32 master weights on device, relaunching from step 81,716 (3.77T tokens) as a 200-step trial against the pooled-wave run on the same batches; Matt Wittmann announced the rewind on Discord. The first wheel hung four of four attempts on one rack's EP domain after 43, 0, 0 and 0 steps. A wheel rebuilt against NCCL 2.30.7 headers, matching the 2.30.7 runtime, in #9062 ran 225 clean steps, tracking the control at −0.0064 nats with drops of 1e-4 against 3.4e-2 and MFU of 23.3% against 21.5%, and the hero has trained on it since. The hang is not gone: the release stalled once after about 1,590 steps and recovered in 21 minutes, and Wittmann's audit on #8870 puts his confidence that the header change materially reduces it at about 60%. A source audit of XLA's ragged kernel found two deadlocks in the remote branch, neither on the path the hero executes, so the next step is a capture of the first blocked GPU operation. #9082 added a projected finish tile to the Grafana strip, which read November 18 at step 85,956, and Rafal Wojdyla tightened the hero chart ranges in #9120.

The NVLink story resolved into a narrower ask. #8934 had named GPU 0, link 14 on node s9jvxs64 as the one NVLink counter moving anywhere on five racks; that exact link failed with Xid 149 on September 10, five days later, taking down the rack. Wittmann closed it into #9086 and walked back the claim that fabric faults were the leading explanation for the hangs, since the two failure modes have now separated. The fleet rate, two rack-scale NVLink episodes in 28 days across 12 domains, sits at or below Meta's published Llama 3 figure scaled to this fleet, with 96% training availability over 338 hours. The remaining work is to collect per-link recovery counters ourselves, because DCGM emits none on driver 595.71.05, and to ask CoreWeave whether its L12 qualification reads them and whether the affected racks run NVSwitch firmware older than the version NVIDIA documents as fixing Xid 149. Kernel work continued alongside. #9077 tracks replacing the vendored QuACK expert GEMMs with jax.lax.ragged_dot lowered through cuDNN in a fork of XLA: on one rack from step 54,000 it reaches 23.77% MFU against QuACK's 24.84% with per-step losses agreeing to 5e-5, but two of four runs hung and it needs cuDNN 9.25 and cuBLASLt 13.5 minimums before it can land. #8733 closed after its portable pieces were retested on ragged: fused shared experts were neutral (23.48 versus 23.50), and noncollective command buffers with small-parameter Fully Sharded Data Parallel (FSDP) cost 0.4 points and 7.5 GiB. Looking toward the long-context phases in the run plan, #9116, #9117, #9118 and #9119 add a context-parallel (CP) axis to the hero mesh, sharding queries and the short convolution by sequence; a 262K-token configuration at EP16 × CP4 completed 40 updates on 64 GB200s at 10.03% median MFU with 1.42% dropped assignments, a throughput number only.

On the science side the week's decision is a data-mixture swap planned for Monday, September 14. Will Held opened #9126 to compare an H100 ladder that switches from the Harrier 2026.08.18 mixture to the new one at 25% of training. The completed d768 rung finishes at 1.07933 Paloma macro bits per byte against the baseline's 1.08372, 0.41% lower, improving 11 of 16 subsets; RedPajama (+1.28%), Penn Treebank (+0.77%) and Wikipedia (+0.46%) regress, and the d1024 and d1536 rungs are still training. That comparison leans on the same Paloma aggregate that a Discord thread questioned. markhart0034 flagged the eval curves after the weight-decay restart, and Larry Dial traced it to three subsets rising over the last 10,000 steps, among them a Twitter set of ten-token documents with emojis; the scaling ladder's preregistered end-of-run prediction of about 2.04 dropless Paloma macro loss is stated on this aggregate, so its floor matters. David Hall proposed tracking uncheatable or Paloma micro instead, and Larry Dial preferred keeping the old aggregate as the reference for the projection while building a new one for intervention decisions. Kaiyue Wen noted that adding weight decay resembles raising the learning rate for the affected modules, which could shift the effective temperature; by September 9 Percy Liang observed the Paloma loss had resumed falling.

Two training-dynamics threads opened late in the week. Larry Dial reported the gradient norm rising since the switch to near-dropless EP, dominated by the lm_head through the learnable gain on the final RMS norm; his read is that the model can make more confident predictions without 3% random dropping, the 1e-4 logit z-loss bounds it, and the trigger for action would be a norm past 5 with a diverging loss, at which point he would resume an earlier checkpoint and raise the z-loss or add weight decay to the final norm. A Discord member, @szha, suggested unit-norm router rows, routing noise or sampling, and a centered bias as stabilizers from trillion-parameter training; Larry Dial replied that the hero's router z-loss is logging-only under sigmoid routing and that the sender-drop increase was a product of the earlier EP implementation. Dial also posted #8827 prompt completions from the step-78,000 checkpoint. The attention-gate issue #8818 and the MuonH expert-norm question #8621 stay open with no new findings this week.

The cold-start SFT stage moved onto the long-context base this week. Will Held posted a roundup of the five 262K-context Snowball checkpoints in #8977: QK-norm scale 1.57, 1.75, and 1.75 with 2×, 4×, and 8× long-document upsampling, each trained from step 156,000 to 157,000 and now on Hugging Face under open-athena. Larry Dial proposed a qk_mult=2 with 8× skew arm and checkpoint averaging as v4-2048 time allows. Held's production SFT run #8954 (80% cleaned Datakit chat mixture, 20% pretraining replay, 2,000 steps on a non-preemptible v4-2048 from step 157,000) stopped before TPU dispatch on its first attempt because the coordinator lacked a W&B key; by September 11 Will Held had published a 262K SFT checkpoint with evals running. Will Held later noted it does much better in harnesses that leave tool calling at the OpenAI-API level than in Terminus, which parses tool calls itself. The data path under it merged: #9056 normalizes every SFT source into OpenAI's Harmony format and enforces a conversation contract (every tool call must name a defined tool; Terminus traces get a terminal tool definition added rather than being dropped), and #9067 makes string, mapping, and list-of-parts content render identical token prefixes, closing #9103, a template quirk Benjamin Feuer traced after MarinSkyRL's migration smoke discarded 68 of 69 Snowball rollouts whose vLLM-rendered prefix carried an extra newline after assistant turns. Feuer's own SFT of the 262K checkpoints on H100s surfaced a run of trainer bugs: the Snowball model ignores packed-example attention boundaries #9029 and omits per-block rematerialization #9030 (a 32K-sequence smoke planned 283 GiB per device); AdamH divides by a zero norm and poisoned a 67B checkpoint on its first warmup step #9054, which the Grug loop then committed as a successful checkpoint because a NaN loss exits with status 0 #9051; and multi-host XLA compilation hung when ranks chose different fused cross-entropy block sizes #9038, #9042, in part because the dispatcher drops LEVANTER_ environment variables #9044. Separately, @mrinal-bespoke reproduced the three-stage recipe from #8225 on TACC Vista's 64 GH200s #9026: Chat and Thinking finished 0.05–0.07 higher in loss than the reference (0.701 vs 0.648; 1.067 vs 0.998) at about 11% MFU, and a Stage 3 on the Nemotron Terminal Corpus ran 1,888 steps to loss 0.501; evaluations are queued.

On the RL stage, Luke Lee reported that 20–30 steps of RL on R2E-Gym doubled the SFT'd 64K Snowball's validation pass@1 from 8.7% to 16.6%, with the gain coming from ending turns earlier rather than better debugging; 70–91% of pre-RL attempts die from context overflow, and RL brings that to 50–75%. His ablation #9027 tested whether trimming the reasoning history Terminus-2 re-feeds (31% of input tokens) would help: pass@8 across five variants landed between .301 and .327, within noise, trimmed variants spent 35 turns instead of 24, and a quick RL overfit under the trimmed contract drove reasoning to zero in ten updates, so RL runs stay on the full history. Luke Lee also found that MarinSkyRL's token-in/token-out (TITO) path silently fell back on 40 of 64 sampled trajectories MarinSkyRL#520, filed #9099 documenting that with TITO off Snowball sees a history format it was never trained on while with it on Qwen does. In the evals channel Luke Lee asked for a policy on when the eval protocol should update; a pinned comparability lane plus a training-aligned lane for checkpoint selection was proposed in reply. Because about 70% of a 740-second RL step is the trainer waiting on decode, Lee trained a one-layer EAGLE-3 speculative-decoding draft on Snowball's own outputs in 25 minutes #9079: 1.50× per-stream decode on held-out 64K R2E-Gym turns with outputs unchanged. Adapting it on 2,177 R2E-Gym rollouts raised that to 1.61× and brought three 5-node RL steps in 31 minutes instead of 44, with acceptance holding through 36 RL steps without a refresh #9114; the base and adapted drafts are on Hugging Face. Russell Power suggested making the draft tuning automatic by fine-tuning it on the large model's rollouts as they arrive.

Ahmad Qamar closed the first async-RL epic #8936 on September 8 and restarted it as a structured program in #8955, with a frozen answer contract, hashed prompt templates, and a preregistered 2 pp non-inferiority margin on completed answers (correct and self-terminated), after the first epic showed raw GSM8K reward hiding nontermination. Closing out the old runs, a third seed of the Snowball cadence comparison showed that syncing every four updates cuts core time 29.3%, but completed-score differences of +26, −31, and +2.7 pp across seeds establish nothing about quality. The engineering lane then went after weight sync, which had a median cost of 20.2 s per Snowball sync: an exact expert-block publication path cut transfer-and-install to 0.51 s from 19.9 s with all receiver bytes matching, and moving diagnostics outside the generation pause brought the paused interval to 1.99 s. Even so, sync plus generation-buffer wait is still about 74% of core time. Other results from the thread: 16 inference GPUs save 4.2% of core time at 15.8% more GPU-hours, so 32 learner and 8 inference GPUs stay; Fully Sharded Data Parallel 2 (FSDP2) fits Snowball on 32 learner GPUs at a 38.6 GiB PPO peak against roughly 57 GiB for Megatron but fails the exact probability-ratio gate at every update; and BF16 gradient reduction saves 6 GiB at the PPO peak, though 20 updates do not select a precision. Qamar's draft #9123 renames the Grafana view "RL Post-training" and lists recent RL runs on Home. On the JAX side, David Hall opened draft PRs #8978 and #8981 that expose the June Snowball model as pipeline stages and add Group Relative Policy Optimization (GRPO) replay and an offline Levanter learner; the combined run measured zero maximum absolute difference between recomputed and training-forward logprobs across three 67B updates on 32 H100s with pipeline parallelism (PP) 4 and expert parallelism (EP) 8. #8953 asks the MoE dispatch to skip padded tokens after 62% padding in the first microbatch tripped the no-drop guard.

Evaluation and data plumbing for the launch also moved. Feuer's open #9113 checks in the Snowball evaluation policies (Evalchemy MMLU-Pro, GPQA Diamond, CruxEval, FinanceBench, IFBench, and MRCR, plus Harbor SWE-bench, OT-TBLite, Terminal-Bench 2.0, SimpleQA, and DS-1000) and routes judge credentials through Marin secret resolution to fix #9070. Russell Power merged the TaskTrove conversion pipeline #9061 (1,739,326 rows in, 1,449,686 Harbor tasks retained across 43 sources, every rejection logged in a ledger), a publisher to open-athena/task-trove on Hugging Face #9124, and direct SkyRL reads from TaskTrove Clean by source and tag #9125. Feuer published a reconstruction of Nemotron-3 Ultra's post-training pipeline #9107 (35 production stages, no complete recipes, post-training about 5.9% of pretraining FLOPs) as a reference for what a full agentic-MoE recipe needs. On serving, the GPU vLLM fork refresh that #8828 blocked last week now passes its Snowball parity gate on Romain Yon's draft #9102 with the September 12 release, and Yon's #9127 shows sparse weight updates stay faster than dense through 40% changed values on two H100 nodes. Ricardo Dominguez's preregistration #8933 was closed on September 8 with its 33% SWE-bench pass@1 prediction for the 9.97T Snowball after 2B tokens of SWE-smith distillation on record; the distillation run that will resolve it has not been reported yet. Neha Hulkund asked what SFT work would best support the RL experiments and proposed an SFT capabilities ladder once the RL stack settles, and Marianna Nezhurina suggested keeping per-epoch SFT checkpoints, since heavy SFT can erode the exploration RL depends on. Work continues on the 262K SFT evals, a validated RL stack on R2E-Gym, and the launch eval policy.

Pipeline-parallel MoE and GRPO in Levanter. David Hall merged automatic DualPipeV pipeline training for Grug MoE #8739, closing the JaxPP tracker #7024. Parameters and Adam state initialize on their owning stages, so no process holds the full 67B training state; Snowball 67B-A2B at batch 256 reaches 603,327 tokens/s and 19.46% MFU on 64 H100s. The extra pins NCCL 2.31.2 because 2.30.7 hangs JaxPP's grouped transfers. #8974 then saves pipeline checkpoints as one unsplit tree so Fully Sharded Data Parallel (FSDP) and pipeline-parallel (PP) runs can resume from each other's state #8972. Behind it sits a Group Relative Policy Optimization (GRPO) stack split into reviewable pieces, from objectives #9046 and oracle replay #9047 through packed batches #9049 and staged JaxPP execution #9050, with #8981 as the combined branch. One finding along the way: the first RL microbatch was 62% padding, and padding counts against expert capacity #8953.

Kernels and the GPU runtime. Two kernel fixes came out of the same work. #8961 drains pending FA4 copies before a fully padded query tile reuses shared storage. #8960 stops the XLA cross-entropy kernel shrinking its batch tile to the largest divisor of an irregular row count: at 8,519 rows a 256-row tile collapsed to seven rows and 1,217 GEMMs per pass. Padding instead cut a packed Snowball scoring-and-update iteration from 402.7 to 83.2 seconds. Romain Yon's #9102 moves training and serving to a CUDA 13.2 vLLM release and fixes an FSDP2 initialization path that lost tied policy/reference weights, which had shown up as KL near 0.37; it also unblocks the GPU fork refresh #8828. Ethan Liu made Levanter report a missing accelerator before the mesh-shape error that used to mask it #8973, and @Dominator-Index proposed RMNP, a Muon variant that replaces Newton-Schulz with input-dimension normalization #8950. marin-ops-agent removed a checkpoint-discovery glob that listed every tensorstore chunk under a root at each job start #8952. Benjamin Feuer's Snowball SFT runs surfaced a cluster of trainer bugs: packed segments attend across conversations #9029, per-rank cross-entropy autotuning deadlocks multi-host compilation #9038, and a NaN loss exits with status 0 and publishes the checkpoint #9051.

Evals, inference and RL monitoring. Karan Sagar worked through the EvalDash backlog: Harbor's error taxonomy now drives trial classification #9109; agent context limits derive from the served model, after a DeepSeek-V4-Flash run served 1,048,576 tokens while Terminus-2 compacted at Harbor's 32,768 default #9105; benchmark variants share one family column #9112. Liu pointed GSM8K at openai/gsm8k so it loads under datasets 4.x #8979. Feuer counted verified agent timeouts as scored samples, after 30 of 100 OT-TBLite trials were being discarded #8947, and decoded file:// cache paths before vLLM sees them #8946. Ahmad Qamar made the Grafana RL view work on finished runs #9123. Luke Lee documented that whether a thinking model sees its own earlier reasoning in Terminus-2 evals depends on the serving path, not on training #9099; Mrinal Kumar proposed one pinned eval lane for cross-model comparison and one aligned with the training stack for checkpoint selection.

TaskTrove tooling. Russell Power merged the TaskTrove conversion pipeline #9061, which turns the raw dataset into Harbor tasks with an explicit grader contract; release 2026.09.10.9 kept 1,449,686 of 1,739,326 rows across 43 sources in one 3.8 GB Parquet file. #9124 publishes releases to open-athena/task-trove, and #9125 lets SkyRL experiments select rows by source and tag straight from the Parquet instead of staging thousands of objects, a cost quantified in #9122: 1,273 tasks became 8,265 files and held two H100 nodes idle for 58 minutes. Hall's exact-instruction dedupe #9058 stalled on Hugging Face rate limits, as David Hall reported, before a source-streaming rerun.

Iris. Rafal Wojdyla fixed attempt durations that kept growing after the controller cancelled or preempted a Kubernetes attempt; one 58-minute attempt displayed nine extra hours #9078. Power's #8914 records a runtime-release timestamp only once the backend has seen the container or Pod gone, and shrinks ControllerServiceImpl from 3,629 lines to 326. Sagar's Zephyr stage views on the job page #9088 drew pushback from Power, who would rather Iris not depend on Zephyr. Michael Ryan found that agents can edit user budgets #9060, and Matt Wittmann asked for per-link NVLink counters after a link he had flagged five days earlier failed #9086.

Zephyr, Finelog and data pipelines. Power's #8945 makes Finelog's regional relays durable across object recovery: settlement advances the downstream cursor and retires whole segments in one catalog transaction, and an empty-cache restart rebuilds a table from the selected HEAD. Zephyr now writes Parquet with zstd and page indexes #8401; a canary's 27 tokenize shards shrank 34%. Tinuade Adeleke opened a shared Common Crawl discovery, planning and range-fetch layer #7917, and Max Freedom Pollard stopped the markdown converter emitting empty titles on every link #8938. The datakit-nemotron canary timed out waiting for all 1,000 verify workers #8948.

Marina, Loom and agent tooling. Power exposed Marina application APIs as searchable MCP tools #8943, registered that endpoint in Loom with short-lived IAP tokens #8971, and added an embedded "Ask Marina" panel that opens a Loom session with page context #9055; Plantt planning charts are the first checked-in app #8980. #8944 adds marina validate so applet backends fail before upload rather than after. loom-oa-dev opened an agent-cost tracker #8982 with roughly forty sub-issues; the first PR #9023 pins lint models, rejects unpinned agent launches, and trims the always-loaded AGENTS.md files from 483 to 369 lines.

Docs, CI and dependencies. David Hall noted that external PRs were languishing and pointed new contributors at the code-review channel #9111; he also split Ruff configuration by package after pre-commit stopped passing on two machines #9028. Wojdyla made CI select tests under experiments/, which the selector had never searched #9091. Among the daily runtime bumps, #9069 pinned the canonical MRCR evaluator, and #9108 carried Lee's Harbor fixes for token-in/token-out (TITO) continuation.

@Dominator-Index proposed RMNP in #8950, a Muon variant that swaps Newton-Schulz orthogonalization for input-dimension normalization; David Hall asked to run it through the Agent MoE pipeline first, and the author agreed. @mrinal-bespoke completed all three stages of the Snowball SFT recipe on TACC Vista in #9026. Yizheng Jiao's #8931 answers the Grace-Hopper abort Neha Hulkund reported in #8852; Matt Wittmann asked both to retest on the #8976 wheel, which now has Hopper kernels. Tinuade Adeleke's Common Crawl fetch layer #7917 got its first review, and @jenna-russell asked in #9073 how the third of recent uncheatable-eval data that is AI-written, mostly code, shifts the loss numbers.

External PRs were languishing, David Hall said in the code-review channel, so #9111 now sends new contributors there and asks them to tag a reviewer. In the DNA channel Jean du Terrail introduced BOTANIC-1, a long-context plant genomic model whose scaling and data-curation findings overlap MarinDNA's; Eric Czech asked about data-mix ablations and whether their variant-effect benchmark would ship, and du Terrail offered Marin early access. Data-curation became an intake for TaskTrove: @likelytobelaura turned a text-to-SQL proposal into a MarinSkyRL pull request, and Fluf took on a QASPER conversion after David Hall argued its unanswerable questions make it more useful, since they penalize hallucination. Benjamin Feuer stepped back from running the OpenThoughts-Next meetings, handing RL to Marianna Nezhurina and SFT and data to Neha Hulkund, who opened the meeting to members who asked.

Seven people introduced themselves: a UT Austin master's student on inference efficiency and tabular foundation models, a Lawrence Berkeley postdoc in scientific and atomistic machine learning, a UChicago undergraduate on post-training and scientific agents, a Tübingen researcher on RL post-training, alignment and evals, and an NLP scientist with post-training experience. Two intersect with open threads: @qz, a Cambridge statistics professor, asked what it would take to train a mathematics model useful to research mathematicians, and Jeff Hammerbacher replied that Open Athena is already talking with maths collaborators; @Raghu, a Microsoft engineer on inference for trillion-parameter models, asked where to start and was pointed at post-training. Accounts named Ben Recht, whose test-set-reuse work bears on the uncheatable-eval question above, and Lintang Sutawika, a maintainer of the lm-evaluation-harness that #8979 repairs, joined without posting. The week's reading was mostly about how others build post-training data: the DeepSeek V4.1-Flash report's data-first recipe, Feuer's reconstruction of Nemotron 3 Ultra's pipeline, and a run of papers on generating terminal-agent tasks.