Marin: Week of September 28th summary

The run stayed on build hero-fa4sm100-nomask-step146k all week. From the W&B history, it opened September 28 near step 168,400 and closed October 4 at step 205,288, about 36,900 steps and 1.71T tokens in seven days. It passed 9T tokens on October 2 and stood at 9.47T by the end of the week, with train loss down from about 1.185 to 1.176. Daily median MFU drifted from 26.6% to 26.4%. Paloma macro loss fell from 2.185 at step 168,000 to 2.157 at step 204,000, and UncheatableEval macro loss from 1.623 to 1.608. A measurement posted on #9205 on September 29, at step 177,860, put the median train step at 14.80 s and the full iteration at 15.67 s, for 3.12M tokens/s. Matt Wittmann's agent then showed that the 0.77 s gap is a watch-step artifact: over all updates the gap averages 0.335 s, 2.2% of a 15.21 s iteration, and the slow stretches that covered 40% of intervals on September 16 are gone (1.4% of intervals over steps 171,469–179,469). On #9180 the hourly saves were costed at about 0.55% of training time on the live run; a one-rack test of deferred jemalloc purging did not reduce that cost.

Three stops cost about 11 hours. On September 30 the temporary checkpoint at step 184,731 started at 21:34Z and never completed; the next save hit a BarrierTimeoutError at 22:35Z, JAX shut down the trainers, and Iris retried the 176-task gang at 22:37Z. loom-oa-dev reported that the logs do not show why some trainers missed the barrier, and no storage, worker or GPU error preceded it. Training was back by 00:32Z on October 1; W&B shows no progress between step 184,950 at 22:30Z and step 185,035 at 01:23Z. The incident exposed an alerting gap: Grafana cleared its pending alert when the new attempt began restoring, and the retry event aged out before training resumed, so #9620 would announce retries during restore. On October 2 the gang retried four times between 14:46Z and 15:28Z. Attempt 10 timed out in XLA RegisterTask, attempt 11 aborted in multihost tracker setup, and attempt 12 reached step 193,732 at 15:54Z; the cause is unconfirmed, and W&B shows about 1 h 40 m without progress. The same day #9700 scoped hero alerts to the production Iris user after a page for a similarly named one-rack job of David Hall's. The longest stop came on October 3. Attempt 12 reached step 197,999, then one rank (global 483) failed the step-198,000 save at 10:06Z: TensorStore got HTTP 400 reading an object from the earlier temporary step-197,873 checkpoint, and the other 703 ranks timed out at the barrier. Attempt 13 restored from step 197,637, but at 11:24Z all 176 pods were Running and training had not resumed. loom-oa-dev's read-only triage found ranks 68–71, on node s2brxs64, stuck in XLA compilation; that node's storage proxy had logged TCP timeouts and HTTP 502 for the hero-checkpoints bucket at 10:09–10:11Z. It asked for a node-authorized operator to check the host and for the 400/502 evidence to go to CoreWeave storage support. W&B shows no progress from step 197,995 at 10:04Z until step 198,092 at 17:00Z, just under seven hours, plus a replay of steps 197,638–197,999.

The next planned handoff is a context-length switch. One-rack tests on #9615, restoring the step-180,000 checkpoint, priced 16K at about 3.1% of throughput with more dropped expert assignments. On October 3 the team settled the data-stream question: at the switch, each source skips the rest of its current shuffle window, at a worst-case cost of about 50B unseen tokens. #9689 implements the switch and was still open at week's end, so the run stays at 4K.

Monitoring changes landed around the run. #9415 merged September 28: the eval-regression check, which had never received hero input, now reads the three newest Paloma macro-loss points from W&B. Paloma proved noisy, so #9729 would switch the alert to UncheatableEval loss. #9486 merged the same day and #9732 followed, aligning the stall and loss runbooks with the telemetry the Grafana bridge uses; #9731 notes the rules cannot yet see a hero run on GCE. #9637 merged October 2 so hero W&B runs record their launch commit, which had been blank because the 65 MB checkout exceeded the code-capture limit. On #9479, Romain Yon reported that his hero RL job, meant for the spare rack DH1-392, landed on training rack DH1-394 with group_by alone; Russell Power replied that a job should not be able to land on a hero rack while the hero is running, and that he is fine adding a named-rack constraint, though priorities already push lower jobs aside. On Discord, Percy Liang announced the September 29 community meeting on how the hero's architecture and kernels were co-designed for the hardware. The thread posted no new finish date; what follows is this page's own estimate from W&B. With 184,963 steps left after 205,288, at this week's pace of about 5,300 steps a day, outages included, the run would finish near November 8; at the clean 15.0 s step time it would finish near November 6.

Matt Wittmann's draft #9708 shortens the hero training step on one GB200 NVL72 rack from 13.90 to 12.61 s, taking MFU from 28.24% to 31.13%; a profile-guided latency estimation (PGLE) profile built from the program's own trace brings it to 12.52 s and 31.37%. Each change was measured as a 100-step single-rack run restored from the hero's step-180,000 checkpoint. The largest gains: computing the routing-weight gradient on the expert side (+3.14%), leaving the ragged transport buffers unfilled (+1.44%), letting rematerialization discount the host-offloaded carry (+1.18%), a token-major dispatch scatter (+1.14%), and fusing the SwiGLU backward into the GEMM epilogue (+0.89%). The second half of the list came out of #9636, which David Hall opened after AI2 released OLMo-core 3; on Discord David Hall suggested benchmarking it for low-hanging kernel fruit. The verdict: Marin's MoE path already matches or beats OLMo-core's BF16 kernels. PyTorch's grouped GEMM took 40.91 ms per median expert chunk, forward and backward, against 30.31 ms for QuACK. What transferred was the dispatch scatter, a Triton router top-k (0.835 to 0.102 ms per call, bitwise), and running the shared expert's gate/up projection beside the first ragged dispatch. The PR supersedes #9481: this dispatch overlap measured 12.666 s per step against 12.783 s for last week's two-stage chunk pipelining, each with its own PGLE profile. Loss over 100 steps differs from main by at most 3.8e-4, against 3.3e-4 between two runs of main. Peak HBM rises from 103.1 to 123.2 GiB of 143.75. The 11-rack production program needs its own PGLE profile, and none of this is on the hero yet; the live run measured 26.47% median MFU on September 30 #9615.

The run had planned to lengthen its context at about step 195,000, a point it passed this week still at 4K, and the week measured what the change costs. In #9615, a Claude agent supervised by Matt Wittmann restored the step-180,000 checkpoint on one rack at a fixed 4.19M tokens per step, three seeds per arm. 8K context trains at 1.6% fewer tokens per second than 4K and 16K at 3.1% fewer. Dropped expert assignments rise 4.5× at 8K (1.9e-4 to 8.3e-4) and 16× at 16K (2.9e-3), though each arm covers only 100 steps after the switch. Larry Dial read the small speed difference as a reason to skip 8K and go straight to 16K, then 65K at the first cooldown and 262K at the end of training. He wants 100-step trials of the query-key scale multiplier at 1.48, 1.58 and 1.68, each ending in an eval. The harder problem is the data stream. Matt Wittmann found that a context change reshuffles it in four ways: Levanter sizes shuffle blocks in sequences, permutes sequences within 512-block windows, and aligns its mixture blocks and the stage boundary at step 108,000 to step counts that change with the batch size. #9689, which replaces #9639, holds 4,194,304 tokens per rack per update so the 390,251-step schedule, learning rates and weight decay are unchanged, and shuffles blocks of 2^20 tokens regardless of context; with the old sequence-sized blocks a 16K relaunch would have re-read about 2.85T of the remaining 9T tokens. For the within-window permutation, Will Held and Rafal Wojdyla settled on the lowest-risk option: skip the rest of each source's current shuffle window at the switch, about 50B unseen tokens in the worst case, which Larry Dial's thread put at about 0.6% of the remaining run. The same procedure will apply again at 65K. No GPU rehearsal of the switch has run.

Smaller pieces landed. #9600 merged on September 30 and starts every device-to-host copy before reading any W&B metric; converting the hero's 828 router-metric values took 175 ms before and 58 ms after, and the 3,592-value watch payload fell from 740 to 269 ms. The investigation on #9205 behind it put the trainer's time outside the timed train step at 2.2% of a 15.21 s iteration. #9353 aligned the pipeline-parallel FA4 stack with Levanter's pins (flash-attn-4 4.0.0b28, CUTLASS DSL 4.6.2, QuACK 0.6.4). Draft #9662 adds a GB200 pipeline parallelism (PP) launcher that consolidates #9279 and #9460; a two-layer shape at PP2 with four-way expert parallelism (EP) ran on eight GB200s, but the full 48-layer shape is still unvalidated. #9709 moves small hero-shape ablations onto the hero's own data store and Harrier 2026.08.18 mixture. Romain Yon's #9584 exports a pinned native hero checkpoint to BF16 split-expert Hugging Face files and applies the checkpoint's pending router-bias update first, the gap last week's router study found.

Will Held published the mixture-swarm runs for both Snowball and the hero as a Hugging Face dataset, and Xiaosen Zheng reported that models mapping mixture to downstream bits per byte reach an out-of-fold R² of about 0.93 on LAMBADA and HumanEval but predict other tasks less well. Percy Liang announced the September 29 community meeting on how the architecture and kernels were co-designed for the hardware, and Rafal Wojdyla posted the slides, recording and a blog post on expert parallelism. Larry Dial said engram-style memory would have gone into the current run had the kernels been ready, and that the next run's MFU will start from a much better place.

Benjamin Feuer closed the broad-domain RL with verifiable rewards (RLVR) campaign on Snowball in #9735. Every lineage gained for a few steps and then became unstable. The six-arm loop and length penalty sweep #9599 ran 16 optimizer steps per arm on 128 GPUs from the September 21 Datakit SFT: a length penalty cut token-limit stops in every matched pair (48% to 15%, 22% to 6%, 52% to 23%) but did not raise pass@1, and the best arm led the 100-prompt holdout by only one to four answers. The turn-budget sweep #9625 found a 48K per-turn cap cut first-batch length stops to 9.7%, against 12.7% at 32K and 18.4% at 16K. The unpenalized Antidoom run peaked at 34/100 holdout pass@1 at step 12 and fell to 2/100 by step 20; a restart with a frozen reference and KL loss reached 42/100 at step 20, though that checkpoint was picked on the same holdout. The clearest win came from a different objective. In #9691 Feuer adapted Liquid AI's Antidoom method, online final-token preference optimization (FTPO), to the September 21 checkpoint. On 512 matched math prompts at a 16K budget, exact loops fell from 139 (27.15%) to 12 (2.34%) while correct answers rose from 232 to 283. On a 100-prompt cross-domain set at 32K, loops fell from 43 to 19 and verifier passes rose from 19 to 26, short of the per-domain target. He published the result as Dr. Doom, and Benjamin Feuer recommended that anyone running RL on Snowball start there. He also showed how MarinSkyRL's process-reward callback penalizes looping: Benjamin Feuer saw 24% fewer looped responses after four steps, with mean tokens per trace down from about 14.5K to 12.5K.

Agentic SFT moved the terminal benchmarks. In #9582 Harshit Varma fine-tuned the September 21 checkpoint on 5,324 MiniSweAgent-format teacher traces from allenai/tmax-sft and raised OT-TB-Lite from 0.118 to 0.255 pass@1 over 68 tasks and three seeds (+0.137, 95% CI [+0.083, +0.196]); Terminal-Bench 2.0 moved from 0.091 to 0.120 and 2.1 from 0.083 to 0.090, neither outside noise. Adding the model's own verified rollouts helped only a little. Harshit Varma noted that MiniSweAgent suits this model better than Terminus-2, and Will Held suggested class-conditional training over plain rejection sampling, which he expects collapses entropy. Luke Lee submitted two SFT checkpoints trained on CalibForge traces generated by relaying between a teacher and the student, plus 4,392 Kimi SWE-smith trials. Against the base's 6.5% TB, 11.7% TB-Lite and 13.8% SWE-bench, model 1 scores 20.8%, 28.2% and 45.3%. Benjamin Feuer reran it under the draft policy and read 27.2 on OT-TBLite and 16.7 on TB2.1, so he expects the gains to hold; he asked that the data be wired into Datakit's SFT sources for decontamination and conversion to the mini-swe-agent harness, since SFT knowledge does not transfer across harnesses. Expert SFT fared worse. #9719 records three negative results: a 5,000-conversation GLM-5.3 rollout continuation dropped HumanEval+ from 71.3% to 50.0% and GSM8K from 83.0% to 66.0%; three raw-science mixes on the Step38 RLVR checkpoint left GPQA Diamond flat to 3 points lower; and a 3.48M-conversation synthetic science continuation cut GPQA Diamond from 53.87% to 28.62%. #9718 preserves those recipes. Rafal Wojdyla's five-day coding-expert spike #9528 also closed as a negative result after its four-update pilot failed its gate; it also found that identical one-sample eval runs differed by 9.76 points on HumanEval+ #9558.

Long context is the newest gap. David Hall wrote that the September 21 SFT is close to Qwen3.6-35B-A3B on most evals but that its MRCR errors look like retrieval failure; Benjamin Feuer put the post-fix MRCR scores at 18.3 for Snowball against 92.7 for Qwen. Hall reran the four-condition probe from #8701 in #9519: on the same 299 two-shot examples, needle recovery amid distractors fell from 91.18% at the pre-SFT step 157,000 to 81.61% at SFT step 158,000 (drop 8.16 to 11.08 points), while isolated needle gain was unchanged. David Hall called this good news because SFT damage is easier to repair than long context itself; Will Held suggested the pre-SFT checkpoint could serve as the MRCR expert. GraphWalks is also weak: #9562 traced Grug's F1 of 0.060 to 859 of 920 responses hitting the roughly 4,100-token output cap mid-reasoning. Doubling the budget raised Grug to 0.145 and Qwen from 0.516 to 0.723 on a matched 750-example set, but 636 Grug responses still ended on length.

Benjamin Feuer posted the draft Eval Policy 09-29 #9566, which he expects to be the last major revision until November: Terminal-Bench 2.1 at 65K context with three trials per task, SWE-bench Verified on all 500 tasks under mini-swe-agent instead of Terminus-2, AIME24 at ten trials with a standardized grader, stratified NUPA with a loose grader, enforced thinking-off for MRCR, and MiniMax-M3 replacing GPT-OSS-120B as judge. Will Held is checking it line by line: he caught six Harbor benchmarks listed at 32,768 tokens while the launcher passes the model's served limit, and asked how to score τ³ tasks where the user simulator, not the model, fails. #9569 checks in the September campaign with model-bound resume and Mini-SWE context enforcement, and #9461 gates launches against a chosen policy and makes the September 24 cohort the Evaldash default. Will Held opened a plan to test whether the policy correlates with AA-II; Percy Liang cautioned that hill-climbing it is an intervention, not a correlation. On RL infrastructure, Ahmad Qamar merged #9507, moving Iceball, Qwen curriculum RL and TaskTrove to Megatron; #9605 brings Snowball and async training along. Toward the release, Feuer's #9737 adds a public Snowball family tree of 118 checkpoints, and the step-92 RLVR checkpoint chosen in #9412 is being rescored under the draft policy. Work toward the first official post-train continues; David Hall said the long-context results make him reluctant to freeze the current SFT.

David Hall's 28-PR draft stack that turns Levanter into an inference server for Snowball and Hero appears under hero engineering; what matters here is what it is for. The upper layers exist so SkyRL can train against native serving: exact token IDs over HTTP #9664, streaming before the batch finishes #9669, cancelling one request without aborting its peers #9668, and staging remote weight broadcasts before installing them under a monotonic model version #9670. The opt-in GPU paged-decode kernel #9656 rebuilds FP32 attention probabilities as three BF16 pieces so tensor cores can do the work; for local Hero attention at batch 8, H100 median host latency fell from 2,140 to 162 microseconds. Four drafts add EAGLE speculative decoding, batching requests with independent budgets #9683 and refreshing drafts through the remote update protocol #9685. None of the stack has merged, and accelerator throughput is unmeasured throughout.

TaskCompendium, David Hall's common format for RL tasks, settled its schema. Schema 0.20 #9646 defines single-result tasks with environment requirements, named tool-provider contracts, resources grouped by visibility, and one private verifier; the September 17 stack of direct-chat lowering, TaskTrove MCQA imports, and NeMo function-call submissions merged beneath it #9214, #9220. The grading package was renamed verifyit and the standalone repository's logic and tests came back into lib/verifyit #9697, after the numeric example had graded 12.0 wrong against the string 12 #9540. On October 2 Hall closed about fifteen importer and runtime drafts as deferred, from containerized tool providers #9607 to the Reasoning Gym import #9596, keeping each branch, so that the narrow path can finish first: asynchronous grading of typed submissions #9608, file submissions through Harbor Docker #9556, and an optional eject_button that earns reward only on tasks privately marked unsolvable #9614. On Discord David Hall asked how to handle stateful tool environments such as NVIDIA's workplace-assistant database, where grading needs the final state; Russell Power argued for running tools inside the agent's container for a first version and revising once a model can cheat, and Hall took that shortcut while keeping the schema open to either. Power's open #9642 curates task sources with Zephyr artifacts and GLM review.

Rafal Wojdyla's rollout engine #9612 would replace SkyRL's four rollout paths with one that reads TaskSpecs, runs them in a sandbox, grades them, and writes the rollout buffer. On October 1 his agent worked through twelve GPU smokes before a Qwen3-4B run on 16 H100s passed the eight-task ShellSim gate in 21 minutes with mean reward 1.0. The Marin side is draft #9623, paired with MarinSkyRL #879; its Shellbox prerequisites merged #9641, and a stub recursive self-improvement (RSI) loop #9626 builds on it. Ahmad Qamar landed frozen-token RL mismatch probes stored in FineStore with route-agreement reports #9506 and a CatCount GPU canary that trains pretrained Qwen with Megatron and stops at reward 0.65 #9553. Benjamin Feuer's RL Data Atlas merged #9518, a Marina applet that tracks every RL source with quality reviews, solve-rate bars for difficulty, and dated reports; Russell Power proposed it as the start of a common ingest, validate, and cleanup pipeline. Gonzalo Benegas registered NVIDIA's 7,496-task Skill2Env from a pinned Harbor release #9632, and Tim O'Donnell registered 100,000 PDBThink protein-coordinate tasks #9695.

The Datakit ferries now run the reference DAG #9727. Each ferry had copied the dedup pipeline, including verify_fuzzy_dups and a memory store on 1,000 workers of 160 GB each, and the tier-3 canary had failed every week since August 17 because that pool never became ready #9494. Pinning tiers 1 and 2 to one region at the same hour then let tier 1 preempt tier 2's workers 148 times, so tier 2 moved to production priority #9736; #9741 fixes a shard that stayed in flight forever after Iris replaced a lost worker attempt under the same ID. Rafal Wojdyla's cluster verification from August merged #8725, and its v11 output, 2.38 billion markers across all 292 sources, is registered #9508. A new contributor, Himanshu Kriplani, profiled the decontamination mark pass: about 90% of its time was Python string work, and the 270 MB bloom filter was read from object storage 162,535 times per run. The byte-identical fix #9690 took eight workers from 112 to 91 seconds, and the stdlib pickler in place of cloudpickle on shuffles #9699 trimmed a 3.6M-row shuffle from 34–36 to 30–31 seconds. In Finelog, Russell Power pruned Parquet segments by timestamp before loading substring indexes, taking an impossible query from 22.8 seconds to 29 milliseconds #9618.

Snowball trains on AMD GPUs. Amy Sartran ran the June 67B-A2B recipe end to end on one 8× MI300X node under Slurm #9462: 0.76 seconds per step at 6.5% MFU with expert parallelism (EP), against 2.27 seconds and 2.2% with Fully Sharded Data Parallel (FSDP) alone. On MI350X, XLA turns the bf16 expert matmul into a dense product over all 256 experts because hipBLASLt's grouped matrix multiply covers only fp16; switching that one op to haliax's Triton ragged_dot gives 0.73 seconds and 3.86%. Supporting changes merged: AMD peaks in the FLOPs table #9568, #9688, corrected L4 and A6000 entries after L4 MFU had read half its true value #9574, a synthetic-token Snowball trainer #9613, and xla_flash #9692, Levanter's pure-JAX flash attention exposed to Grug for backends without a CUDA or TPU kernel, which runs within 4% of the reference and cuts peak memory from 81 to 32 GiB per GPU at 16,384 tokens. Percy Liang opened an #amd channel for the work. On TPU, the d1280 investigation covered under training runs produced #9561, which selects the fast XLA cross-entropy backward for a 3.25% gain, and a canary MFU drop from 0.661% to 0.609% was bisected to #8960 with the procedure written up #9703.

A cgroup leak took down CoreWeave nodes. On October 2 pods failed to start on one H100 node with "no space left on device": the host had 65,535 cgroups, 65,412 of them empty under /silo, the nested runtime inside Mark Muchane's privileged pipeline jobs #9696. Rafal Wojdyla's agent found the leak on every Silo node in the cluster, ten of them at 80–87% of the limit, and Muchane traced it to nerdctl rm -f under gVisor exiting zero while leaving the sandbox, its shim, and its cgroup behind; all Silo hosts are stopped until a fixed host ships. Seven nodes cordoned for kernel soft lockups, about 22% of the cluster's H100s, could not reboot while Iris pods remained #9726; once their workloads were cancelled or moved, all 32 were schedulable by October 4. Iris cleanup had deleted a retry's ConfigMap by a hash shared across attempts, leaving ten hero tasks waiting out a 30-minute JAX timeout #9705. Russell Power's open PRs forward preemption to the peer that runs a federated task #9720 and quarantine nodes whose S3 requests stall while Ready #9721. Shellbox fixes from the Snowball coding-expert smoke strip Harbor's hosted_vllm/ prefix that drew HTTP 400 #9551 and keep trials alive after an unsupported tool call #9555.

Larry Dial opened Marin Belay Intake #9514 so anyone can propose ideas for the agent-run ladder climb #9451, which is locked to collaborators: a workflow copies each suggestion a maintainer approves into the main thread #9515, posting from a dedicated bot account because bot tokens cannot comment on a locked issue #9565. @RavnaBergsndot filed the first suggestions, among them semi-orthogonal latent-MoE projections and a per-layer query embedding for attention residuals. The serve dashboard gained a Compare tab that sends each prompt to two models and records a preference #9712; Will Held's #9713 launches both jobs from marin-serve compare, reports output cutoffs, and raises the default budget from 1,024 to 16,384 tokens. Marina got a read-only Hugging Face token so the RL Data Atlas stops hitting rate limits #9501, and Levanter's TPU CI lane now runs on root lock changes only when the resolved TPU graph changes, after a verifier-only dependency change spent the lane's 25 minutes #9580. Docs now cover CI for new libraries #9500 and Hugging Face uploads under OpenMDW #9604; Romain Yon would adopt the shared stale policy #9744. Daily runtime bumps and four Dependabot updates round out the week.

Himanshu Kriplani, whose merged decontamination speedups appear under Other Changes, has three more PRs awaiting review: tokenizing ahead of the quality model #9617, scoring quality from the token ids the tokenize stage already wrote #9694, and larger Arrow batches in the Parquet writer #9711; each waited on a maintainer to approve workflow runs from the fork. On Discord Himanshu Kriplani asked how anyone scores quality on 15T tokens when a 110M-parameter head would need about 6,000 H100-hours; Mark Muchane described labeling with GLM-5.3 and distilling, and noted that tokenization and disk reads, not FLOPs, set the pace. Amy Sartran followed the MI300X result with a setup recipe for JAX on AMD Instinct GPUs #9462 and opened #9633 after finding the shared FLOPs estimator counted six of Snowball's seven full-attention layers. @RavnaBergsndot filed a suggestion a day to Belay Intake #9514, from per-element attention gates to logging the stable rank of each projection matrix, and Larry Dial approved each into the ladder climb #9451. Ben Kolligs opened #9524 to attach a vision encoder to Delphi, starting from the earlier Qwen-based VLM recipe #2383.

Tim O'Donnell opened a #marin-for-science channel for benchmarks and post-training data in biology and other sciences, where Steven Dillmann of the Terminal-Bench-Science team proposed a TB-Science Lite set built from submitted tasks that proved too easy for frontier models. In #architecture, Wenhao Chai reported that sampled softmax on the muonh baseline matched final performance in 1.1 to 1.3× less wall-clock time, the gain shrinking with model size. Eric Czech shared a preprint in #mechinterp arguing that hybrids of sliding-window RoPE and global NoPE change how models retrieve from context, which bears on the long-context gap described above.

Three people introduced themselves: Vivek Ramakrishnan, finishing an AI master's at Northeastern after work on agentic systems, continuous agent evaluation, harness engineering and some post-training; aquajet (Anish), who worked on evals, datasets and fine-tuning at EleutherAI and Georgia Tech and on agent and coding harnesses; and André Martins, professor at IST Lisbon, who leads the SARDINE lab and the EuroLLM initiative (open models up to 22B for European languages) and works on multilingual and multimodal LLMs, dynamic sparsity and long-context adaptation. The first two intersect with the TaskCompendium and harness work; Martins, who introduced sparsemax and the adaptively sparse transformer, brings context on long-context adaptation and sparse attention. About thirty others joined without an introduction.

The links shared in #news and #architecture leaned toward the machinery of training rather than new models: tokenizer design, how much filtered web text is now machine-written, and other labs' released training code and RL environments.