Marin: Week of August 3rd summary

The 67B-A2B hero run on TPU v4-2048 advanced from 6.35T to 7.65T tokens this week — 76% of its 10.07T-token horizon — holding MFU at 18.6% with train cross-entropy at 1.305. Paloma macro loss fell from 2.3000 to 2.2702, putting the uncooled run just 0.0012 above the preregistered 2.269 stage-1 target at 8T with 0.35T tokens still to go before that mark. On its current trajectory the run should meet or beat the preregistered number once it reaches 8T and the learning rate decays; Benjamin Feuer shared the loss curve in Discord, calling the run “pretty on.”

Larry Dial launched a long-context cooldown branching from step 102,000 of the main run (~6.84T pretrain tokens) and extending the sequence length from 8,192 to 65,536 tokens with learning-rate decay. The cooldown finished in 32.8 hours on the same v4-2048 slice, landing at Paloma macro loss 2.2242 — solidly below the preregistered 2.269 target and 0.053 better than the Snowball 2T cooldown’s 2.2772 from #6811. This is the second intermediate checkpoint from the hero run and the first with long-context capability. In the Discord questions channel, Ricardo Olmedo asked about releasing raw intermediate checkpoints at roughly 70B, 220B, 660B, and 2T tokens (no cooldown needed) for joint pretraining/fine-tuning scaling law research — echoing a similar request from the epic’s own thread last week.

Quiet week. The full Hugging Face release plan #7495 filed by Romain Yon last week — covering repo ID, chat-template disposition, upload sequence, license, and data-lineage paragraph — is still waiting for team sign-off, with no new comments on either #6811 or the release plan.

No new activity this week. The reinforcement learning with verifiable rewards (RLVR) plan #7786, the corrected eval tracker #7505, the agentic supervised fine-tuning (SFT) diagnostic #7743, and the MarinSkyRL Fully Sharded Data Parallel 2 (FSDP2) performance comparison #7878 all went without comment.

The #6689 epic (preparing the ~120B-A8B hero run on B200s) had no direct sub-issue or PR activity this week. The architecture and data-mix decisions it depends on are being driven through #7073 and the FSDP-vs-EP ablation ladder in #8062.

No issues, pull requests, or comments were filed against this epic during the week. The 67B-A2B pre-training run that must finish before post-training can begin is still underway on TPU v4-2048 under #6704, and Benjamin Feuer noted in Discord that the main 10T run is looking good. The intermediate-cut post-training epic #7170 and the approximate-token-budget tracker #7074 were also silent. Post-training work continues to wait on the pre-training run to reach its cooldown checkpoints.

No new activity landed against this epic or its JaxPP tracker #7024, which had been the focus of an intense push the prior week when UB-X symmetric-memory transport cleared the 20% MFU gate at 20.41 mean across four H100 nodes. The extra issue #8073 (GPU numerical instability debugging) was root-caused and closed: Larry Dial traced an output_proj collapse in isoFLOP sweeps to the MuonH hyperball projection computing per-parameter Frobenius norms with jnp.linalg.norm under Single Program Multiple Data (SPMD) partitioning, where the norm was silently computed on a shard rather than globally, producing a spurious rescaling. The fix landed and the issue was closed on August 9.

The central architecture decision for the hero run crystallized into a structured ablation. #8062 specified the experimental design: the fully sharded data parallelism (FSDP) option is d6144 top-4-of-128 at 359.6B total / 22.6B active, and the expert parallelism (EP) option adds LatentMoE with 2x latent compression at capacity factor (CF) 1.3, both at sequence length 4096 on one GB200 rack at EP degree 64. Small-scale runs at d768 through d2048 mirror hero-scale dropping dynamics. Larry Dial launched the ladder on H100 #8046 and reported early results: at CF 2.5 (which roughly matches FSDP MFU), EP gets +7.3% effective-compute slowdown on training cross-entropy loss, within noise of run-to-run variation. But Paloma macro evaluation loss is 35% worse under EP because quantile-balanced bucketed routing drops tokens unevenly across evaluation domains.

The critical insight that shifted the framing: at inference time the model can swap to dropless FSDP routing, so the evaluation gap may not matter for the deployed model. Larry Dial's d768 dropless re-evaluation confirmed this — reloading EP-trained checkpoints under dropless sonic_cute recovers most of the Paloma gap. LatentMoE with CF 2.5 pulls drop fraction to 0.25%, and histogram quantile balancing (QB) is neutral on loss but gives slightly lower drops and should behave predictably cross-rack. Rafal Wojdyla ran the EP64 hero shape through the LatentMoE launcher #8013, #7981 and posted two candidate EP configurations with 192 and 256 routed experts. The GB200 ladder is now running at d768/d1024/d1536/d2048 across EP and FSDP flavors; results will determine the hero architecture.

On the data side, the domain-cluster quality classifier #6855 went through a 16-comment campaign of label collection, model training, and release-gate testing, with the latest rank-safe GLM-projection candidate passing private and speed gates. The PDF-to-document pipeline for the focus crawl #7616 was validated end-to-end on a 10% sample.

The 20% B200 MFU commitment #6706 was met last week with the corrected single-rack FSDP figure of 22.71%. This week the gate continued to rise: Russell Power's sixteen-wave configuration sweep #8070 — FSDP-sharding the router, attention gate, and GatedNorm factors plus selecting NVLink SHARP — added +3.31% pooled MFU on one rack. The FSDP control at the hero shape on the EP64 mesh #7981 measured 19.40% MFU vs. 26.28% for EP64 with the same model, confirming the EP throughput advantage that motivates the ongoing FSDP-vs-EP ablation. Ragged all-to-all correctness work #8077 found that XLA's one-shot ragged path deterministically corrupts the first backward pass under latency hiding; disabling latency hiding and reducing collective overlap to 1 recovered 31.6% clean throughput on a two-node EP8 proxy, with the fix now scoped to the ragged EP backend only #8081.

No new activity landed against the post-training budget this week. #7074's thread was silent, and the standing estimate — roughly one to ten billion post-training tokens, each costing about 700× the hardware FLOPs of a pretraining token — remains unrevised.

The silent multi-rack collective wedge that has blocked 8+ rack GB200 training since last week got a root cause and a fix. Matt Wittmann ran a systematic delta-debugging campaign on #7344, starting from a minimal reproducer that shrank the model from 48 layers to 1 (wedging in 7 minutes instead of 48) and reproduced inside NVIDIA's own JAX container for an actionable upstream artifact. The bisection isolated the cause to NVIDIA Collective Communications Library (NCCL) 2.28.9's proxy-op slot pool on aarch64 (Grace): ncclProxyGetPostedOps publishes freed slots with a weak compare-exchange whose retry path checks the observed value instead of the compare-and-swap (CAS) result, so a spurious LL/SC failure silently orphans the batch. Once a rank's 2048-slot partition runs dry its next collective blocks forever. The fix is a pin to nvidia-nccl-cu13 >=2.29.3, which ships as 2.30.7; #8038 carried the standalone bump and #8040 delivered it into the hero launcher alongside hang-detection instrumentation. That PR also switches hero runs to one JAX process per GB200 GPU, following the N3 positive from the ablation ladder (1,710 clean steps at 10 racks with per-GPU processes vs. repeated wedges at the old 1-process-per-node layout). A durable incident record was published to Echo wiki/93.

On the operational side, Russell Power landed #7929 (checkpoint-based warm restart for stalled gangs), #8031 (GPU hang detection with XLA deadman and heartbeat supervisor), #7951 (NCCL RAS telemetry persisted every two minutes), and #7959 (RAS collection reduced to fit the 256 KiB runner cap at 512 ranks). A separate failure mode was tracked in #7956: a single-rank CUDA_ERROR_ILLEGAL_INSTRUCTION destroying the CUDA context and taking down the gang, which reproduces independently of the NCCL wedge and is now the binding constraint on multi-rack run length. CW Grafana Xid alerts surfaced during wedge investigations may connect the two failure modes, but attribution is still open.

Compilation and startup time also improved. Russell Power landed CuTeDSL kernel caching #8045, #8065, #8078 so that QuACK and FA4 kernels survive across runs without recompilation, and a sixteen-wave FSDP configuration sweep #8070 sharded the router and gate factors plus selected NVLink SHARP, netting +3.31% MFU on one rack. #8052 made the FSDP hero profilable and #8057 vectorized per-layer router metrics across the stacked scan axis.

Larry Dial opened #8037 with the post-sweep hero configuration: MuonH learning-rate coefficients refit to the August LR sweep results from #7856 at sequence length 8192, and the hero attention cadence changed from sliding window 512 / global every 6 to sliding window 2048 / global every 4. The sweep itself #7856 concluded that the loss bug seen in earlier Liger cross-entropy runs is gone on B200, and the new data mix causes only a 7% LR shift at the target model size. #8017 measured inference cost for the attention window choices: global every 4 instead of every 6 costs 5–6.5% throughput and 45.65% more KV-cache memory, while window 1024 vs. 512 is under 1% throughput and 97% more cache. The recommendation is to keep every-6 globals and prefer window 512 for inference, though the hero recipe chose window 2048 / global-4 to prioritize training quality.

A new iso-FLOP sweep #8003 is running d512 through d2048 across 1e18–3e20 FLOPs to refine the compute-optimal frontier at the refit LR. Early results show clean U-curves in Paloma macro loss vs. tokens at each budget. #8032 explored cross-layer expert tying (sharing one expert bank across groups of layers): at d512 the cost is +0.038 Paloma, at d768 it narrows to +0.029, and the d1280 full comparison was launched and running at week's end. The experiment tests whether the architecture can preserve parameter count while reducing active compute, a relevant lever for both expert parallelism (EP) and fully sharded data parallelism (FSDP) configurations.

Two JAX coordinator bugs surfaced on the GB200 cluster this week, both triggered by production preemptions. Russell Power scoped supervised coordinator publication and lookup to the current Iris task attempt in #8079 after a preemption of an ra2a-s02-ep16 run on cw-us-east-08a: attempt 3 ranks 1–15 resolved the prior attempt’s address (10.186.213.145:27055) before rank 0 published its new one, silently joining two generations. Earlier in the week, Rafal Wojdyla moved coordinator port selection to global rank 0 in #7994, eliminating fixed-port reuse between retry generations — the root cause of the different incarnation aborts documented in the incident record. Together the two changes mean a retried multi-node gang on the GB200 cluster now resolves a fresh coordinator address on a fresh port, instead of racing against the prior attempt’s stale lease. Supporting that cluster further, #8055 cross-compiles native arm64 Iris wheels via the Debian GNU toolchain, cutting build time from over 30 minutes under emulation; #8075 lets retryable coordinator pods drain during node maintenance by setting maxUnavailable: 1 on their PodDisruptionBudget (Pod Disruption Budget), keeping minAvailable: 1 for admin-only production jobs; #7980 admitted the GitHub CI service account to the GB200 federation peer so validation jobs can reach it; and Mark Muchane documented the gb200x1 and gb200x4 device targets in the tiny-model tutorial #7858. The topology-aware scheduling fix in #7928 collapsed the two-flavor CoreWeave queue model into a single cw-tas ResourceFlavor so Kueue can preempt accelerator-free batch pods that previously blocked GPU gang admission through topology fit — the issue that held five interactive RL gangs on cw-rno2a #7916.

Finelog’s telemetry path, rebuilt last week after 24.5 M rows/hour overwhelmed the hub, got the indexing and admission controls it was missing. #8069 added trigram indexes on log.key: a job-scoped key LIKE '%<job>%' query previously decoded 660 M rows in 16 s against a 10 s deadline; the trigram sidecar now prunes to the matching row groups before any decode. #7978 did the same for telemetry_v1.name, where an NVIDIA Collective Communications Library (NCCL) (NVIDIA Collective Communications Library) substring query cost 28.5 s over 1.38 B rows — each trigram sidecar adds 430 KB per 80 MB segment. #8010 bounded analytic queries to a 10-second ceiling with two concurrent slots, after nine production level bumps with no decoded data overlapped slow RPCs at a 20.986 s median. #8012 added planner-facing segment indexes beside each L1+ compacted segment, and #8066 made projection schema disagreements degrade rather than wedge ingest — a redefined projection previously returned 503 on every /v1/telemetry POST fleet-wide until someone restarted with the matching binary. On the producer side, #8035 limited Levanter tracker telemetry to process zero, cutting the fourfold replication from 15 stale MoE (Mixture of Experts) runs emitting roughly 39,300 rows/s; #8034 drained forwarding backlogs without dropping rows, after the USE08 hub skipped 28.9 M sequence positions in its first 15 minutes; and #8008 compressed client uploads with zstd level 1. #8067 added a shadow serve mode that boots read-only over a real store copy, blocking any mutation path at startup.

Monitoring was reorganized around the question operators actually ask. #8001 replaced ten data-source-organized dashboards with seven question-organized ones and surfaced fleet GPU power in kW for the first time, attributing watts to training runs by joining node_name across node-agent DCGM (Data Center GPU Manager) telemetry and Levanter streams — 81 kW of 532 kW showed as idle or unattributed. #7952 shipped a Finelog SQL workbench with time bounds, variables, and stat panels. Alert routing was consolidated: #8050 has the bridge post each critical alert to Slack and hand the thread to Loom for automated triage, so operators steer in the same conversation the alert appeared in; #8056 routed all alerts through the bridge into #marin-alerts; and #8074 retired the old persisted Slack receiver that had kept posting to marin-eng alongside the bridge. On the IaC (infrastructure as code) side, #7910 moved the hai-gcp-models Identity and Access Management (IAM) declaration from Python constructors to schema-versioned YAML, preserving 189 role grants and 70 encrypted principals while declaring each ciphertext once; Will Moss declared the 11 project-scoped custom roles in #7926 and granted Key Management Service (KMS) encrypt/decrypt to marindev in #7962. #8061 unified three compiled-artifact caches behind a shared rigging.cache.PersistentKvCache, content-addressed with atomic-rename writes, and #8059 made a missing or unwritable autotune mount degrade to recompilation rather than aborting JAX init. #7842 replaced random chaos tests with 49 deterministic product journeys over the real controller, and #8068 added a generic refresh-fork skill with per-fork migration descriptors as the first step toward automated upstream tracking #8048.

The Zephyr and Datakit runtimes received a round of performance and reliability work feeding the hero-run pipeline. #7888 kept PyArrow RecordBatch values through the map and write stages of MinHash and tokenization, eliminating Python row materialization for schema-stable streams. The measured gains were a 1.76x speedup on a 1M-row local benchmark, 3.38x on a 16.2M-document MinHash root job, and 28.72x on the 100B through-MinHash span (from 2h57m to 6m09s), the last driven mainly by shared pooling and higher concurrency rather than per-core code changes; generated outputs were bit-identical up to 100B in controlled comparison. #7941 fixed out-of-memory (OOM) kills in global fuzzy dedup on 16 GiB workers by dropping the scatter flush threshold to 20% of task memory and capping external-sort merge passes at 32 runs. #8011 bounded per-document embedding memory to 32 KiB so sources with megabyte-scale documents cannot exhaust a worker, and #7944 made the concurrent-pipeline limit configurable after a hard 16-pipeline cap was failing jobs outright rather than applying backpressure. The shard-preserving memory stores merged in #7873, redesigned as worker-resident tables that load data on entry and keep lookups local, replacing the cross-actor network design.

Mark Muchane built a complete PDF-to-document pipeline for the Common Crawl focus crawl in #8023. The pipeline fetches the crawl's 3.16 million PDF records by coalesced Web ARChive (WARC) ranges and routes each between Docling and a vision-language model (VLM) using a classifier trained on measured extraction agreement: at a 50% VLM budget it catches 79% of the documents Docling reads badly, against 42% for the FinePDFs rule. The output passes through exact dedup, decontamination, quality scoring, fuzzy dedup, and language labeling into NormalizedData. Validation on a 10% range sample (316,297 PDFs yielding 275,296 labeled documents, 60.5% English) and a token-matched head-to-head against FinePDFs #7621 show the pipeline roughly 0.3 loss better on arxiv, GitHub, and S2ORC subsets and roughly 0.4 worse on general web, the expected shape for a science-steered crawl. Separately, Tinuade Adeleke opened a shared Common Crawl WARC layer #7917 and a DOCX-to-text pipeline built on it #7919; manual review of 15 extracted records across four languages confirmed correct text and language identification #7736. Will Held noted he expects unfilled form templates in the DOCX corpus to receive low quality scores downstream.

The domain-embedding rework on #6855 logged 16 agent-driven iterations this week as Rafal Wojdyla’s agent worked through student architectures and label sets for the GLM-5.2 taxonomy. A rank-preserving projection passed all semantic and vector-health gates on 10,000 held-out documents, lifting parent macro-F1 from 0.45 to 0.49. A 50,000-document labeling run completed after recovering from a Unicode line-separator edge case, but the resulting projection did not pass release gates at rank fraction 0.31. The current candidate is a 0.7 mix of the Arctic base and the GLM-trained FastTransformer, which passes private and speed gates with mean semantic gain 0.086; CPU inference was switched to float32 after the bfloat16 path failed the speed gate. On the embedding side, Will Held opened #7998 to embed canonical documents from the pinned global fuzzy-dedup artifact with microsoft/harrier-oss-v1-0.6B, running up to 256 H100s across the CoreWeave federation.

The global fuzzy dedup over the full 292-source registry completed this week. Will Held posted the results: of 18.710 billion documents, 14.397 billion were retained (76.95%), bringing estimated tokens from 25.615T pre-dedup to 19.710T. The post-dedup total fits inside the end-of-year token budget, confirming the shift noted last week from mandatory source reweighting to quality filtering that can drop the lowest-quality tokens outright. How much of the existing swarm evidence carries across the new deduplicated data and the pending architecture decision — expert parallelism (EP) versus Fully Sharded Data Parallel (FSDP) — remains open; the team expects to need roughly a week to finalize and run the mix pipeline once the architecture is pinned.

Michael Ryan’s weekly sync on the LLM extraction recipe #2351 reported the first mixing results. The v1.1 extraction spec scores 90/102 on the project’s unit-test suite (13/13 code, 10/10 math, 9/9 tables, 12/12 attribution), with the weakest category being negative-case removal at 13/24. Against external benchmarks the spec achieves higher recall but lower precision, though he notes those benchmarks penalize keeping titles, authors, and discussion sections that the spec deliberately retains. The Olmix-style mixing experiments, bucketed by quality times domain, show that optimizing on one eval suite does not transfer to another: a mix tuned for the OLMo development set does not move DataComp-LM (DCLM) Core, and a mix tuned for DCLM Core fully inverts the OLMo scores. He flagged DCLM Core as noisy at this model scale, with many sub-tasks indistinguishable from random. The high-coverage extraction spec is still running, so the mixing exploration is preliminary.

Three older issues closed this week. #4148 (synthetic reasoning bootstrap corpus) and #5094 (public diagnostic-log sourcing) were auto-closed after inactivity, and #6643 resolved after Benjamin Feuer published the Delphi midtraining dossier documenting the procedure behind the p*m* checkpoints.

The biggest structural change this week is Finestore, a new append-only columnar archive for per-sample eval output. #7976 introduced the library: a DataStore writes typed Parquet shards and opaque blobs under a run's URL prefix as immutable objects, a CompositeReader unifies schemas and deduplicates by primary key, and the format needs only object creation with no manifest or writer coordination. Evalchemy and Harbor now write EvalSample rows directly into a run's archive, and Harbor stores each raw agent trajectory as a blob the sample references by a finestore:// URI. #8026 published the wheel to PyPI alongside the other pure-Python libraries and moved the eval-sample contract (EvalSample, StepRecord, EvaluationStore, and the lm-eval and trajectory mappers) from marin.evaluation.samples into finestore.eval, so evaluation forks can import the contract directly rather than copying it into their images. Two follow-ups hardened the archive: #8044 fixed a Unicode line separator (U+2028) that was splitting JSONL records mid-string and causing completed GSM8K evaluations to lose their Finestore archive during sample export, and #8049 keyed samples by extraction filter so that the strict-match and flexible-extract rows lm-eval writes for each document no longer collide on the same primary key. #8005 migrated legacy evaluation runs from the record indexes, validating normalized samples and available trajectory bytes before retiring source Parquets.

The shared evaluation launcher gained three file-backed entry points. #7997 lets launches accept repeatable Evalchemy and lm-eval configuration YAML, moving the standard definitions out of Python and into checked-in files. #8021 adds file-backed model configs so a catalog-schema YAML or JSON file can be used at launch without modifying the checkout. #7993 allows GPU cluster and Iris priority-band overrides, so a launch can target a chosen cluster rather than the fleet-selected default. Harbor's production resilience also improved: #7914 writes trial results straight to durable object storage as each trial finishes so a killed driver no longer discards completed work, and #7960 pauses Harbor when a managed Iris inference task leaves RUNNING, waits for the replacement endpoint, and restarts the deterministic Harbor job against its durable output directory. A live Qwen3-0.6B Terminal-Bench 2 run validated the full preemption cycle. Will Held fixed a stale fsspec directory cache in evaldash that had frozen record ingest at 251 entries despite new record.json objects being written #7975.

On the policy side, Benjamin Feuer published a formal Marin Eval Policy #7958 defining what constitutes reportable, canonical eval results for Marin models and baselines, and shared it in Discord #evals. Separately, Gustavo Sandoval ran allenai/safety-eval on marin-8b-instruct with Olmo-3-7B-Instruct as the reference, reproducing Olmo 3's published safety table to within plus-or-minus 3 percentage points on all 13 rows and filing the results as an epic #7931. The two models score about the same overall, but the child issues surface specific gaps: misinformation is marin-8b-instruct's largest deficit versus Olmo 3 #7940, and refusal is removed by roughly ten fine-tuning steps #7943. #7937 proposes adding safety benchmarks to the eval stack so safety numbers regenerate automatically rather than requiring someone outside the project to run them. Five evaldash feature requests were filed (#8083–#8087), including a 90% infrastructure-success gate for agentic evals so that a batch with one or two trial timeouts records its aggregate metric instead of failing the entire run.

Benjamin Feuer closed out the full Snowball 67B-A2B non-agentic reinforcement learning with verifiable rewards (RLVR) experiment matrix in #7786 this week, posting verdicts for every arm alongside an experiment index and synthesis document. The newest cohort, E10, was a four-arm frozen-router ablation at 32k context. Its AdamW control produced the campaign's strongest learning curve: reward climbed from −0.743 to +0.530 and pass@16 from 0.688 to 0.918 over nine steps, with stable gradient norms and rising advantages — direct evidence that the router-bias corruption tracked in MarinSkyRL #309 had been the binding constraint on earlier arms. Every 32k arm, however, died around step 10–11 from the same failure mode: a stochastic max-length rollout VRAM-spikes one rank, which silently drops, and the remaining 63 ranks hang a 1-element ALLREDUCE for the full 1,800-second watchdog. The failure is optimizer-independent and sequence-length-driven; MuonH OOM'd at step 5 on-device, and expert-parallel was chronically gradient-unstable with raw gradient norm spiking to 204 at step 9.

hero-v1, the campaign's first full-budget continuation of E6, confirmed the RLVR-MATH dataset effect at 32k and scale — reward from −0.744 to −0.334 and pass@16 reaching 0.621 through step 8 — but terminated at step 31 of the same rank-drop, and its resume died on a transient S3 checkpoint read timeout. The hero-v2 proposal supersedes it with four changes: 24k context to cap the worst-case sequence at roughly 23k tokens instead of 31k, MuonH as the optimizer to match pretraining, truncated importance sampling (TIS) plus training-inference token ordering (TITO), and frozen router. Four MarinSkyRL fixes merged in support: rank-drop fast-fail #331, resume read-amplification reduction #332, launcher retry budget #333, and TITO alignment-metric inheritance #336. The proposal is posted but not yet authorized.

On the data side, the normalized post-training pool specification #7799 moved to design-doc stage, with Russell Power noting that the datasets need first-hand inspection before requirements settle. #8060 was filed to rerun E6 with MuonH, framed as an onboarding exercise for learning the RL framework and Iris. Separately, Romain Yon closed out the Grug eager-versus-grouped divergence investigation #7903: the first mismatch was localized to BF16 accumulation order at the final index_add_ combine, and an FP32 accumulation correction was merged into MarinSkyRL #276. The correction is 3.84% slower at the combine boundary with 252 MB additional peak allocation, accepted for grouped-reduction stability; the live-routing route-aware residual — one action log probability exceeding tolerance by 0.07% — remains formally unresolved.

Russell Power's SkyRL artifact PR #7883 merged on August 7 after a week open for review. Pretraining, SFT, group-relative policy optimization (GRPO), and evaluation are now expressible as a single typed Marin artifact graph, with SkyRL kept as an isolated external runtime pinned by commit and profile. The artifact owns immutable model and data locators, a semantic role topology, checkpoint roots, and a validated terminal policy; the iceball-micro experiment validates the full graph on a random-init Qwen3-0.6B-size model, with resilience evidence from a planned-preemption resume and a stalled-worker recovery. The PR closes #7797.

The supporting framework changes that #7883 consumes landed in sequence. MarinSkyRL #284 consolidated the distribution into one root package and frozen lock on August 3. #296 then wired Iris training tasks to install from that frozen distribution at the exact pinned commit, replacing the custom image build, on August 5. On the Marin side, #8063 externalized the TPU vLLM stack into an isolated uvx environment and deleted the legacy in-process marin.rl engine entirely on August 8 — the engine had no live consumers, and its in-process vLLM path was the reason the TPU inference forks had to co-resolve in the workspace lock. Dropping them decouples the workspace torch version from the TPU stack.

#7920 was filed to retire the remaining SkyRL custom image paths now that the image-free routes are live: the maintained path publishes immutable H100 and GB200 vLLM wheels with source provenance, and the frozen environment passed a 30-step dense FSDP gate on H100 and a Grug rollout-to-FSDP2 checkpoint/resume gate on GB200. Three campaign retrospective issues — #6654, #6655, and #6656 — plus the Iris launch workflow #6629 were closed as informational records of the campaigns they document.

The vLLM and TPU-inference forks no longer co-resolve in the root uv workspace. #8063 moved TPU serving to an isolated uvx environment pinned by SHA in marin.external_dependencies, matching how Evalchemy, Harbor, and MarinSkyRL are already pinned through config/update-external.py. Co-resolving the forks had held the workspace at torch==2.11.0; dropping them decouples the workspace torch version from the TPU stack. The same PR deleted the in-process marin.rl engine, which had no live consumers beyond its own tests and was the reason the forks had to live in the workspace in the first place. The out-of-process SkyRL launcher from #7883 stays. On the GPU side, #8027 pins the isolated CUDA vLLM install to promoted GitHub Release wheels referenced by SHA-256 digest, storing the release metadata, cu129 backend, architecture, SM compatibility, and asset digests in config/external/vllm/gpu-release.toml. This closes #8022, where the source pin did not pin the precompiled wheel.

Last week's multinode vLLM parity work also landed: Romain Yon's #7758 runs GrugMoE pipeline-parallel (PP) inference across Iris tasks through native vLLM, without Ray. The leader Iris task starts vLLM's API, scheduler, and local data-parallel (DP) and expert-parallel (EP) workers; each follower starts a bare MultiprocExecutor for its pipeline stage. The parity matrix covers PP1/TP1/DP8/EP8 on one 8xH100 node and PP2/TP1/DP8/EP8 across two gang-scheduled nodes, both scoring the same 64 GPU goldens, requiring exact same-rank recomputation, and logging cross-rank spread as a diagnostic rather than a hard assertion. TPU-side parity for the June 67B checkpoint is still in draft #7556. Russell Power filed #7954 on reducing chat-template confusion between pre-training and post-training, and Mark Muchane proposed sharing broker session bring-up between the GPU serve path and the converter pool #8024 to remove roughly 40 lines of duplicated Iris lifecycle code.

The original SFT data curation epic is closed, but the post-training SFT line continues under a new scope. Benjamin Feuer filed #8089, an epic for establishing a versioned cold-start supervised fine-tuning (SFT) recipe that runs immediately after pretraining cool-down. The recipe targets ten competencies: chat-template adherence and basic instruction following, think blocks, cross-domain reasoning in science, mathematics, legal, and medical contexts, adaptive thinking effort, model-name responses, basic agentic tool use, answer-format hardening, task summarization, and task resumption from summaries. Six research sub-issues were filed alongside it: cross-domain thinking and tool use #8090, task resumption #8091, answer-format hardening #8092, adaptive thinking effort #8093, model-name response behavior #8094, and task summarization #8095. Each sub-issue has explicit acceptance criteria requiring candidate data, evaluation cases, and a recorded recipe entry. The epic builds on prior work in #7321 and #7743 but does not yet have data, configurations, or results attached.

The Harbor-trace-to-SFT conversion PR #7830, which adds a Datakit transform and reproduction recipe for rebuilding agentic SFT records from durable trace datasets, remains open in draft; Benjamin Feuer asked Will Held for review. The SFT launcher GPU-path bugs in #7417 — unsized host RAM causing an OOM during initialize_from_hf and tokenizer staging rejecting S3/GCS paths — also remain open.

The epic itself saw zero pull requests, issues, or comments this week. Three related issues saw only automated maintenance: #5416 (aggregate downstream-task measure for the group-regularized perplexity (GRP) surrogate) was auto-closed for inactivity, #5521 (proportional perturbation scale-transfer experiment) was auto-closed with a managed-experiment summary added by David Hall's agent, and #5618 (synthetic surface-form perplexity probes) was marked stale and kept open with an archival summary.

The upstream signal that shapes the next round of data-selection work arrived in #data-mixing: Will Held reported that the 292-source registry has been globally deduplicated, retaining 14.397B of 18.710B documents (76.95%) for an estimated 19.710T tokens against the end-of-year budget of at most 20T. With the pool now close to the budget ceiling, any future mixing experiments will operate on a meaningfully tighter corpus than the 25T raw figure from prior weeks.

No pull requests, issues, or comments landed against this epic, continuing the pattern from last week when planning moved upstream into the serving-topology discussion for the ~360B candidate.

Benjamin Feuer landed three PRs making the shared evaluation launcher more flexible: #7990 accepts Evalchemy configs at launch, #7991 lets launches override GPU serve cluster and priority band, and #7992 accepts file-backed model configs. He also fixed a bug where Finestore exports split JSONL records at Unicode line separator U+2028, causing scored GSM8K evaluations to be recorded as infrastructure failures #8042. Separately, he drafted a formal Marin Eval Policy document, opened an Evaldash request tracker with proposals for infrastructure gating, aggregate-score de-emphasis, cross-cohort result merging, and benchmark-coverage filtering #8084 #8085 #8086 #8087, and scoped a new cold-start SFT recipe epic covering cross-domain thinking, tool use, answer-format hardening, summarization, and model-name behavior.

Gustavo Sandoval filed a set of safety-evaluation issues for Marin 8B #7931. Key findings: misinformation accounts for most of the 6.6-point HarmBench gap versus Olmo 3 #7940, and refusal behavior can be removed in roughly ten Low-Rank Adaptation (LoRA) fine-tuning steps #7943. A companion issue notes the model card incorrectly claims no safety evaluation exists #7933.

David Hall opened a highly active investigation (34 comments) into cross-layer expert tying for Grug: whether routed expert weights can be shared across layers from initialization, and whether an already-trained untied checkpoint can be converted post hoc into a shared-bank model #8032.

Will Moss moved custom Identity and Access Management (IAM) roles like marindev into Pulumi #7924 and granted Key Management Service (KMS) encode/decode permissions to IAM-grant operators #7922. A follow-up issue tracks a spurious-replace bug triggered by shared etags during import #7961.

Russell Power restored unattended verified vLLM release publication #7995, proposed unifying four compiled-artifact caches behind a single adapter #8051, and opened a policy discussion on keeping forks (vLLM, SkyRL, Harbor) closer to upstream #8048. Benjamin Feuer fixed a related bug where the vLLM source pin did not pin the CUDA wheel #8022.

Jeff Hammerbacher filed a series of issues to make ablation sweeps catalogable and queryable: emitting artifact records per sweep cell #7967, joining W&B runs with fingerprints #7968, exporting records to a catalog #7969, and recording gate and effective-speedup decision fields in metadata #7970.

Outside contribution ran heavily toward infrastructure this week. Matt Wittmann was the largest single source, authoring the NVIDIA Collective Communications Library (NCCL) 2.30.7 bump #8038 and the expert-parallel sharding carry-over #7932, and filing detailed root-cause analysis across dozens of comments on the silent collective wedge #7344 and the single-rank illegal-instruction failure #7956. Will Moss worked the infrastructure-as-code layer, moving custom Identity and Access Management (IAM) roles into Pulumi #7924 and landing Key Management Service (KMS) and logging grants #7923. Tinuade Adeleke landed the shared Common Crawl Web ARChive (WARC) retrieval layer #7917 and hand-reviewed 15 extracted DOCX records across four languages. Gustavo Sandoval filed five safety-evaluation issues in a single day #7931, running allenai/safety-eval on marin-8b-instruct, identifying misinformation as the largest gap versus Olmo 3, and demonstrating that refusal is removable in roughly ten fine-tuning steps.

Two Discord threads carried work that GitHub did not. In #evals, boreas opened a first PR to marin-community/evalchemy adding chat-mode GSM8K under perturbation and was paired into a reciprocal review with another new contributor on the SciCode proposal #7658. In #architecture, Mayank posted a linear-attention implementation benchmarking at ~29B tokens/day on a 1B model against 32B for softmax on 4x TPU v6e, dropping to 17B/day once causal convolution is applied; markhart0034 flagged the inference-time state cost that the training throughput number hides. In #questions, Ricardo Olmedo asked whether intermediate Snowball checkpoints at 70B, 220B, 660B, and 2T tokens could be released for joint pre-training and fine-tuning scaling-law research.

Four people introduced themselves: a software engineer previously at Datadog who found Marin through inference lectures, Thom from Common Crawl offering to connect on crawling, standards, and datasets, a Yale statistics PhD student interested in open-source post-training, and one more looking to contribute. The Common Crawl arrival intersects directly with the focus-crawl PDF pipeline #8023 and the WARC layer that shipped this week. Jungwoo Park, first author of Monet which pushes fine-grained expert counts to 262,144 per layer, joined the server without posting an introduction. Reading shared in the server stayed close to active work: Gonzalo Benegas posted the MarinDNA blog write-up, and papers on verification as a scaling axis, capability shaping via token-level data filtering, and variance-controlled off-policy RL circulated across the RL and safety channels.