Marin: Week of May 18th summary

The controller was sluggish in three separate windows this week and the response shapes the bulk of the epic. Rafal Wojdyla diagnosed the worst of them — Friday's wedge — as ConcurrencyLimitInterceptor._acquire calling blocking threading.Semaphore.acquire() from its async path; once the lone production cap ({"FetchLogs": 4}) saturated, the 5th request parked the uvicorn loop and dashboard / health / sibling RPCs all hung. #5925 rewrote the path to await an asyncio.Semaphore with a cancellation-safe release, fixing #5922, and #5927 lifted the canonical interceptor into rigging.rpc after discovering iris and finelog each carried a drifted copy. The headline write-side change is #5912, which dropped ControllerService.SetTaskStatusText — at ~125/s, 73% of inbound controller RPCs per the tcpdump in #5909 — and routes worker status writes directly into finelog namespace iris.zephyr_task_status with the dashboard reading via the mounted finelog stats proxy. #5901 wrapped the active-state predicate in compute_user_spend and _enforce_execution_timeouts with a likelihood(..., 0.005) planner hint so SQLite drives off the small active set instead of full-scanning ~773k tasks rows every tick, fixing #5900; #5889 replaced a continue with break in _preempt_solo so the descending-priority victim scan stops at the band gate instead of walking the unpreemptible tail of ~9k PRODUCTION tasks per scheduling tick, fixing #5888. #5920 stamped finished_at_ms on heartbeat updates for already-terminal attempts — preempted attempts were leaving the field NULL and reads.resource_usage_by_worker was counting those rows as in-flight, pinning ghost CPU / memory commitments per worker #5918.

The week's two largest landed PRs are both Russell Power's. #5636 — the reconcile-only worker protocol previewed last week — merged at +2666/-259, unifying dispatch, polling, and status reporting into a single Reconcile RPC plus a pure-compute reconcile_workers(ReconcileInputs) -> list[plan] split, with legacy StartTasks+PollTasks gated by IRIS_RECONCILE_RPC_ENABLED for a flag-flip rollback. #5862 then introduced a controller-minted 16-hex attempt_uid as the routing key for task attempts on both the Reconcile and legacy wires, with workers falling back to the composite key for mixed-fleet convergence and labeling containers iris.attempt_uid; the same change also fixes the worker same-name-resubmit duplicate-rejection bug. #5848 closed a separate class of leak: once a slice went READY the autoscaler trusted its in-memory view forever, and marin-dev had 5 idle e2-highmem-2 VMs orphaned for up to 15 days because workers had died without re-registering. #5848 added a probe_health tick that pings each READY worker's /health endpoint and terminates the slice after PING_FAILURE_THRESHOLD consecutive failures, and flipped GCE bootstrap to --restart=unless-stopped so transient container failures self-heal; #5856 followed up by replacing the bare internal_address+port pair carried by each RemoteWorkerHandle with a single worker_url, removing the per-consumer host+port stitching that had been producing malformed probe URLs in LOCAL mode. #5892 zeroed buffer_slices on the v4-reserved 2048 group so the autoscaler target tracks demand exactly on reserved capacity instead of holding a warm slice, with a config validator rejecting buffer_slices>0 on any reserved-capacity group #5891.

Observability picked up two substantial additions on top of the dashboard work. Rafal Wojdyla's #5911 stood up a synthetic infra canary daemon at infra/probes/ with three v1 probes — ControllerPing, IrisJobSubmit/<zone>, and FinelogWrite — dual-writing samples to a local sqlite canonical store and best-effort finelog marin.canary, with a 30s scheduler-tick heartbeat under marin.canary.meta for external absent() alerting; a failed finelog push is recorded as a synthetic finelog-push sample in sqlite so finelog outages stay visible. Romain Yon's #5928 prototyped first-class node lifecycle events with a new append-only node_lifecycle_events table and a GCP PREEMPTED mapping into CloudSliceState.PREEMPTED; the PR description itself flags the open question of whether TASK_STATE_WORKER_FAILED already approximates GCP preemptions closely enough to make the work worth doing. Romain Yon's #5830 widened the infra status dashboard so charts get more horizontal room and downsampled the /health latency summaries to 1-minute spacing while keeping the underlying 30s probes. The fallout from the prior week's coscheduled-sibling story continued resolving: #5747 and #5753 closed with the broadened TPU_INIT_FAILURE_PATTERNS for JAX-distributed-RPC peer-loss / SIGABRT signatures, and #5872 remained open as the larger redesign proposing a worker preempt-watcher plus a controller atomic slice-preempt path. #5258 — the longstanding /dev/vfio busy on preempt-then-reuse — also closed this week. The Iris VM bootstrap got a robustness fix from Will Held's #5885: gcloud ships as a snap on the TPU base image and snapd mounts it asynchronously during boot, so worker startup could reach the Artifact Registry auth step before gcloud was on PATH, skip auth, and fail the image pull; the bootstrap now blocks on snap wait system seed.loaded right after the Docker daemon starts.

Zephyr-side hardening tracked the same workload pressure. Rafal Wojdyla's #5890 introduced a PullStatus(StrEnum) with NO_WORK_BACKOFF / STAGE_COMPLETED / SHUTDOWN so _stage_manager can finally distinguish end-of-stage from end-of-pipeline — the old "SHUTDOWN"-string-plus-None overload was causing worker hot-spin and constant re-registration of slots. #5907 changed the heartbeat-giveup path to fail the actor task instead of clean-exiting, so iris sees FAILED and ActorConfig(max_task_retries=10) reschedules the replica rather than silently disappearing. #5906 and its follow-up #5930 applied double-checked locking to IrisActorHandle._resolve and ActorClient.rpc_client after multiple threads on a freshly unpickled handle were racing the None check, each cold-starting a TCP connection and logging paired duplicate Resolving name ... lines at worker startup. Michael Ryan's #5875 promoted heartbeat_timeout, max_shard_failures, and max_shard_infra_failures from hardcoded constants and module-level values to per-context fields on ZephyrContext, threaded through _CoordinatorJobConfig; the open follow-up #5877 tracks rationalizing the now-~12-field ZephyrContext surface. #5910 finally split lib/zephyr/src/zephyr/execution.py — five concerns and 2k+ lines, with runtime-only cycle-breaker imports — into a 4-module package per #5850. On the supply-chain side, Romain Yon's #5914 pinned Iris to kubernetes<36 after the v36 release flipped DynamicClient bearer-auth lookup from authorization to BearerToken while the v36 config loaders still populate the old key, breaking CoreWeave in-cluster auth; #5913 and the exploratory #5917 mirror the in-cluster token into both keys for forward compatibility, and #5924 tracks the eventual upgrade past v35. #5867 — uv add marin-zephyr failing because marin-finelog 0.99 was never pushed to PyPI even though marin-iris==0.99 pins it — remained open at week's end. The design-side prep included Will Held's #5832, a 565-line per-user GCS prefix design that adds a users:// fsspec filesystem so training checkpoints land in gs://marin-{region}/users/{user}/... by default with reads probing personal → glob other users → shared root for transparent cross-user reuse, and Rafal Wojdyla's #5869, a dry-run-by-default Zephyr pipeline that found 76,555 GCS objects whose basename is .artifact or artifact.json across six buckets and renames each to .artifact.json via gcsfs.mv; the live run is gated on a separate update to lib/marin/src/marin/execution/artifact.py, blocked by the executor-sidecar JSONL mis-detection bug #5864 that closed earlier in the week.

One signal loss worth flagging: the Tier 2 datakit canary (marin-canary-datakit-tier2.yaml) ran seven times this week and produced zero green checks — three runs failed outright and four were cancelled. The failure mode is not a data-pipeline regression: the runs blew up in the autoscaler with Quota exceeded for tpu_v5p-preemptible_8-us-central1-a and Quota exceeded for tpu_v6e-preemptible_4-us-east1-d, before any data processing started. Net effect is that the daily Tier 2 green-check signal was effectively absent this week, so a real datakit regression could land without the canary catching it. Tier 1 stayed mostly healthy (5/7 success, 2 cancelled) and the weekly Tier 3 passed.

Romain Yon opened #5887, the next concrete step after last week's vllm-tpu==0.19.0 bump, and it lands the two pieces the prior-week notes flagged as still ahead: an OpenAI-compatible proxy in front of vLLM and a preemption-resilient eval pathway. The shape is a brokered serving path for running HumanEval against Qwen3 across one or more Iris TPU workers. A CPU parent job owns the eval client, the Starlette/uvicorn VllmHttpProxy serving /v1/models, /v1/completions, and /v1/chat/completions, and a broker actor exposing a leased FIFO request queue through the sync WorkloadBroker interface. TPU workers claim requests with leases, forward them to a local vllm serve, and submit responses back through the broker; the proxy matches responses to pending requests by id and drops late duplicates. The same code path runs in-process via LocalWorkloadBroker for local-mode coverage.

The reliability story matches the epic's definition of done. The parent eval/proxy job and broker actor run on non-preemptible CPU, TPU workers are preemptible and recover through broker leases, and timeouts are explicitly ordered worker HTTP timeout < broker lease timeout < proxy request timeout so a slow request fails once from the worker before the broker treats the lease as lost and another worker retries it. Startup readiness sends /v1/models through proxy → broker → worker → vLLM before the eval client starts, and vLLM subprocess cleanup uses a process group so stale EngineCore children do not keep the TPU claimed after parent exit. Parent, broker, and worker jobs are constrained to one Iris region, with split-region runtime configs failing fast. Romain Yon notes the broker is in-memory and parent-owned and scoped to one eval run — this is the eval-serving path, not a durable serving service.

Validation covered the broker unit tests, Iris integration tests, brokered smoke runs on Iris, the duplicate-response drop path, and worker-crash lease recovery; the same-region full four-worker Iris validation is still pending. In review the @claude agent flagged a late-response-from-an-expired-lease race against the retry; Romain Yon appended a commit closing it on the grounds that the worker's vLLM timeout is asserted to fire before the lease expires and the proxy's id-based dedup would catch the survivor anyway, but preferred the tighter lease semantics. The PR also plumbs Iris priority bands through Fray jobs and actors. It remains open pending review from Russell Power, who is out for a few days; Romain Yon is stacking follow-up work on top.

Elsewhere the epic was quiet. #2495, the long-standing Evalchemy-integration ask for reasoning evals like LiveCodeBench and AIME, was closed as stale — the brokered HumanEval path in #5887 is the more direct route to reasoning-eval coverage now. A separate distributed-inference thread in #infra from @Michael-Ryan — the Zephyr-backed multi-region fleet for the WARC-download workload — continues in parallel and is not the same code path as the Iris eval-serving epic, though both share the broker/worker shape sketched in last week's RFC.

This week the gap suite was reorganized from a flat pool of surface-form slices into a competency-organized portfolio with explicit benchmark-leak discipline. David Hall opened #5819 as an umbrella to find pretraining proxies for Artificial Analysis-style competencies, drafted an AA-task-to-PPL-surrogate matrix, then audited it for benchmark leaks and tightened the rule: exact AA, HLE, GPQA, AIME, MATH-500, MMLU-Pro, IFBench, LiveCodeBench, SciCode, HumanEval, MBPP, AA-LCR, AA-Omniscience, and similar held-out items should not enter regular tracking, even where a train split exists. Truthfulness, hallucination robustness, and abstention are explicitly deferred as post-training behaviors. From the reframed taxonomy, seven competency subissues were filed: generated-math validation #5823, instruction-following format #5824, long-context reading and retrieval #5825, factuality and abstention surrogates #5826, professional-document competence #5827, MCQA train/dev coverage #5828, and scientific/technical reasoning #5829, plus a FEVER evidence-conditioned factuality slice in #5863.

On the implementation side the suite was consolidated into one durable entrypoint. #5836 added experiments/evals/model_perplexity_gap_suite.py as the Marin 32B vs Qwen3 32B model-perplexity gap suite, split the previous long-tail provider into semantic submodules, and wired in diagnostic-log coverage with a report retry key. #5897 added an HF-backed synthetic reasoning provider using canonical arrow-ICL task names only so legacy ICL aliases stop double-counting byte-identical slices, with HF dataset revisions pinned for reproducible scoring. #5898 followed up with v2 generators for the four surface-form probe families landed last week (delimiter, identifier/encoding, machine-record, patch/diff), switching to raw continuation prompts with longer delimiter targets and target-only supervised metadata to fix the prompt-format artifacts the v18 dashboard exposed; David Hall caught and fixed malformed timestamp values like 14:70:02 in the machine-record generator and a wrong unified-diff hunk header before the PR merged. #5923 opened a 32K long-context bundle with raw PG19, GovReport, and QuALITY plus supervised target-only QASPER, NarrativeQA, and BookSum, with BookSum using a clean Chapter/Summary prompt and QASPER keeping the question adjacent to the target. The workflow itself was finally written down: #5803 opened a document covering provider registration, the merged model_perplexity_gap_suite entrypoint, Iris scoring, report retry, dashboard artifact copy, heatmap generation, local validation, and website publication.

Two of last week's tracking issues closed cleanly. #5247 on identifying a useful eval set was marked done by Calvin Xu, and David Hall closed #5254 on programming-ecosystem coverage as done pending an IID validation set from Marin's own code. The first correlation-style read against the original P<0.05 target also landed inside #5005: Calvin Xu ran the current priority raw-PPL bundle on the historical 300M/6B mixture panel, adding 40 raw-PPL datasets with 242 signal rows and ten fixed and ten variable-subset noise baselines, then reported the SNR summary against the variable-subset baseline; David Hall wondered aloud how much of the low SNR is simply insufficient training data on those slices. David Hall also added a follow-up note on #5614 proposing that the next synthetic numeric/JSON tracking cut use PR #5723's segment-labeled losses to separate JSON syntax, schema-key awareness, whitespace, and numeric payload rather than continuing to rely on post-hoc lexical buckets. The mcqa_train_core bundle from #5828 was locally implemented with MMLU auxiliary_train, ARC-Challenge/Easy, OpenBookQA, CommonsenseQA, SciQ, QASC, and RACE train/dev staged as supervised whole-answer-string targets, with bridge and renderer extensions and bundle registration in place.

The tokenizer thread continued in parallel. @Percy Liang asked in #evals whether Marin should be using the Qwen tokenizer; David Hall replied that staying on Llama 3 looks wrong, that Qwen is unclear if the target is monolingual, and that a recent toksuite paper has him interested in TokenMonster's vocab-size cascade. He was less sure after reading the appendix and noted that TokenMonster's lack of byte fallback seems easily fixable. @Pranshu Chaturvedi began benchmarking 32K and 64K Marin tokenizers built by dropping less-frequent merges from the 128K vocabulary, getting roughly 800K tokens per second on a single CPU. @Bilibird filed #5837 on vocab-size compression via the TokenMonster technique. On #5079, Yiyuan Li posted a whitespace-token comparison across Qwen3-32B, Marin-32B, Gemma-2-9B, ByT5, and o200k_base, showing Marin and Qwen3 within a few percent on most whitespace categories with about 3,000 more lowercase-prefixed tokens on Marin.

@Benjamin Feuer published an interactive Marin/Qwen BPB comparator on the dashboard and used it to point out that the largest "Qwen-better" files cluster on standard web data and eval-shaped corpora like lm_eval/gsm8k_train, paloma/falcon-refinedweb, and paloma/redpajama. David Hall attributed most of that to uncontrolled Qwen contamination plus rule-based-extraction whitespace artifacts of the kind that drove the Marin 8B WSD-S cooldown-depth fix, and argued that a tokenizer trained on Marin's own corpus is unlikely to have weak spots on in-domain data even if it does out of domain. The @willheld exchange on MCQA scoring concluded that choice_logprob correlates best with accuracy while plain bpb over the whole answer string is good enough for tracking, which is why mcqa_train_core in #5828 stays on supervised target-only BPB rather than adding a new framework path.

The decontamination work that landed last week closed out as #5519: Rafal Wojdyla shipped the v0 pipeline end-to-end across 8 Artificial Analysis evals plus 849 lm-eval-harness leaves (90% of task_configs.py; the 94 misses are datasets 4.x script-loader removals and tasks outside the pinned lm-eval commit), built a 21.78M-unique-13-gram combined bloom at FPR=1e-9, and ran decon over all 113 sources. 15,077,568,955 records scanned, 5,696,170 flagged at 0.038% overall. The headline finding came from the ad-hoc Claude-judge precision spot-checks: bloom decon is trustworthy for prose (biodiversity precision 0.31 strict / 0.69 lenient) but breaks down badly on code (coderforge at 0.00, code-idiom collision on shared OSS function signatures) and on synthetic-eval-format sources (nemotron_specialized_v1_1/formal_logic at 0.00 and 35.28% flag rate, template collision on shared question stems rather than actual content). Synthetic-injection recall: 54% verbatim and 100% on prefix/suffix variants; the 46% miss is the structural floor for eval items under ~13 words. v1 followups now have separate issues: bloom from eval answers + reasoning chains only to kill template collision, an exact-match index for short items to fix the recall floor, and AST/function-level code decon to fix the idiom problem.

The two quality-tagging deliverables both got their first end-to-end distribution analyses. On #5812, the 104-source Dolma3 fasttext quality pass ran in 1h 50m at production priority — the hplt_v3 long tail at 6330 shards finished a single Zephyr execution in 8m 36s with max_workers=1024 — and a follow-up streaming distribution-stats sweep (36m 27s wall) revealed the classifier's actual behaviour. Top-of-distribution is uniformly synthetic / code / SFT content: nemotron_specialized/scientific_coding and nemotron_code_v2/synthetic_question_answering sit at mean > 0.999 — essentially every row scored as high-quality. Bottom is curated historical text: cp/foodista, cp/regulations, cp/uk_hansard, cp/uspto, cp/caselaw all under 0.025 mean, with uspto's p99 still under 0.30. Within nemotron_cc_v2, the synthetic-rewritten variants score dramatically higher than raw-CC at the same tier label (high_quality_synthetic mean 0.763 vs high_quality 0.265). The read Rafal Wojdyla landed on: this is a chat/instruction-likeness detector — matches its OpenHermes / ultrachat / WildChat training data — not a "good prose" detector, and downstream filtering on high_score should be reasoned about that way. #5811 Dolma3 WebOrganizer topic ran 104/104 sources in 10h 45m with zero failures, writing ~970 GiB to gs://marin-eu-west4/datakit/weborganizer/ with the full 24-class distribution per document for map-side join with the other attributes.

To get a quality signal that isn't just chat-likeness, Rafal Wojdyla kicked off the LLM-oracle classifier tracked in #5810. Run 1 (sonnet46-thr05) labelled 7,000 stratified docs with claude-sonnet-4-6 against a 1-5 pretraining-utility rubric — 5,613 usable after rate-limit losses — then trained a binary fastText at threshold 0.5 (62/38 balance) and fanned out across all 104 sources. Held-out (n=961, fresh seed): AUC 0.846, Spearman 0.641 vs LLM, accuracy/F1 0.78/0.72 at the 0.5 cut. On the same n=961, dolma3-fasttext-quality vs LLM only hits Spearman 0.168, and ours vs dolma3 is 0.21 — the two largely disagree, so this is adding signal rather than rediscovering dolma3. Total spend $32.73 under the $50 cap, 4.5h wall. The per-source spot check confirmed the classifier transfers across languages (Chinese math textbooks top the ranking) but flagged two artefacts to watch: it scores starcoder2/ir_* and massive_function_calling at literal 0.000 mean — the rubric reads LLVM IR and tool-JSON-schemas as noise — and the top nemotron-terminal docs all share an identical instruction-template prefix, so the wrapper may be getting rewarded rather than the trace. @willheld's intruder-test sanity check in #data-curation agreed with the new tagging more often than with Dolma's — quality-v0 8/20 vs quality-dolma3 2/20, domain-v0 10/20 vs domain-dolma3 7/20, underpowered but enough to confirm the new tagging isn't worse. On the embeddings side, #5809 closed with Luxical-One int8-quantized embeddings co-partitioned across all 113 sources, and #5808 got the embed→sample→train→assign DAG running end-to-end (FAISS spherical K-means at K=5000, agglomerative merge for K=1000 / K=40, ~10.5M stratified samples); the summarize step keeps preempting before pass-2 completes and needs checkpointing before the next attempt.

Underneath the science, Will Moss's Zephyr stack made the dedup hot path materially faster. #5814 introduced sub-workers within a single Zephyr actor — a pool of threads acting as distinct worker slots above InlineRunner and SubprocessRunner, with separate map and reduce sub-worker counts — closing #5813 and dropping the fineweb_exact wall from 16m to 11m on two threads. #5859 added Parquet row-group splitting bounded by size, optional filename-on-load, and a load_parquet_batch path that returns RecordBatches into Dataset, closing the #4587 request to stop treating one parquet file as one shard. #5860 used both to take dedup_exact_paragraph in fineweb_10bt_exact.py from ~16m to ~6m. The architectural followups are queued: #5855 proposes moving Zephyr to a model where workers just receive tasks and resource constraints and spawn threads, and #5850 tracks splitting up execution.py as it has grown circular-import-prone. Rafal Wojdyla's #5879 reverted the row-level partition_id stamp on normalize output — decon now synthesizes it from shard.shard_idx at read time, making the v1-fallback the only path and leaving cached normalize outputs valid without rewrite — and #5931 bought more headroom on the datakit canaries while they're still flaky between runs. Will Held's #5874 registered AlienKevin/SWE-ZERO-12M-trajectories (106.91B measured tokens) and, more importantly, renamed the executor sidecars from artifact.json / provenance.json to .artifact.json / .provenance.json; without the dot prefix normalize._discover_files was matching them by extension and either crashing normalize or silently injecting phantom records.

The source-onboarding pipeline absorbed the week's data-mixing requests from #data-mixing. @Benjamin Feuer flagged tokyotech-llm/swallow-math-v2 (32B math tokens), openbmb/UltraData-Math-L2 (~34B), and several arXiv/HAL source archives; David Hall filed #5844, #5845, and #5846 for the text/chat-compatible splits, and David Hall's agent put up #5847 the same day with pinned HF downloads and tokenized steps for both Swallow Math v2 (stage3 QA + textbook) and UltraData L2/L3 (Conversation-Synthetic, QA-Synthetic, Textbook-Exercise-Synthetic). Tai Vu registered AI-MO/NuminaMath-1.5 via #5841 with row-transform and registry tests, and kept the parallel NuminaMath-TIR PR #4997 open after dropping standalone NuminaMath-CoT to avoid overlap with the existing gpt-oss-rollouts path. The IID-validation question — should every source carve out a held-out slice at normalize time or post-dedup? — surfaced in #data-mixing when David Hall asked whether the new buckets need IID validation sets; @willheld argued for an IID sample of all datakit sources, and Rafal Wojdyla tentatively pegged normalization time as the right place, with treating it as decontam as the obvious alternative. In #2351, Michael Ryan posted week-2 progress on the small-model raw-web-to-tokens experiment: the DCLM-400m URL list is sorted by date so the prior 3k WARC subset only covered 2013-2017 — a real distribution-shift confound flagged as blocking for an arXiv push but not for peer review — and the new high-quality extraction prompt now beats DCLM on Paloma and continues descending past the point where DCLM bends up. The followup levers — fewer input tokens via more aggressive HTML filtering, distillation into a smaller extractor, multi-spec extraction across internet eras — are now scoped against a flop-budget calculator dashboard.

The gating dependency cleared on schedule: David Hall announced on May 20 that the compute-optimal 1e23 MoE landed at a Paloma macro-average loss of 2.234 against a preregistered 2.252, a prediction extrapolated from runs 333x smaller — closing out #4697 on the day Will Held had targeted for swarm liftoff. The production swarm itself did not launch as a separate job on May 20, though; instead the swarm code landed as #5849 (opened May 19, still open at week's end), which wires Grug-MoE to the 39-domain Dolma 3 + Dolmino top-level mixture across five compute-optimal scales — d512/2.19e17, d768/1.70e18, d1024/9.00e18, d1280/2.83e19, d1536/9.00e19 — with the d=1280 / 2.83e19 point matching the spec from #5364. The PR also adds eval_logprob.py, a minimal lm-eval-harness path that runs the JIT'd forward pass inside loglikelihood so few-shot RNG state stays consistent with request scoring, bumping capacity_factor to 8.0 at eval time so the routed MoE doesn't silently drop tokens. Every cache path is region-relative to gs://marin-us-east5/, so off-region launches hard-fail rather than triggering a copy.

On #5362, Will Held wrote up the next move on the over-optimization problem flagged last week. The naive mean aggregate over-rewards math/code; a hard min over per-factor scores is noise-dominated by whichever factor happens to be unluckiest on a given step. The proposal is a softmin (log-sum-exp) relaxation, A(s; β) = -1/β · log Σ_k exp(-β · s_k), where β→0 recovers the mean and β→∞ recovers hard min, plus per-capability priors π_k so product priorities and laggard-weighting are orthogonal knobs. To make β legible, runs report an effective-number-of-factors ENF = exp(H(w_softmin)) rather than nats of inverse temperature; acceptance is a Pareto sweep of worst-factor Δ vs mean Δ across β, committing to a point where no factor regresses meaningfully and the laggards close a non-trivial fraction of the gap. There's also a planned ablation of factor-aggregate against raw benchmarks at matched ENF: if the FA layer's contribution is small, the relaxation is doing all the work and FA can be dropped. #5367's lower-noise PPL signals slot in as drop-in replacements once they land.

The bucket question on #5363 moved into operational territory: David Hall asked in #data-mixing whether IID validation splits should be carved per-bucket; Will Held clarified the ask was an IID sample of all datakit sources, and Rafal Wojdyla proposed splitting at normalization time, with David Hall floating immediately-after-dedupe or as decontam instead. @willheld also posted intruder-test results comparing the new domain and quality taggers against Dolma 3's: domain-v0 at 50% / domain-dolma3 at 35%, quality-v0 at 40% / quality-dolma3 at 10% — underpowered but enough to argue the new tagging isn't worse than Dolma's. #5364 and #5365 drew no new comments; the spec is now expressed in code on #5849's will/moe_mixing branch.

On candidate sources, Jeff Hammerbacher's #5770 (OEIS integer sequences) cycled: David Hall initially asked for an eval-only wire-up via the perplexity-gap framework, the agent opened PR #5852 streaming stripped.gz + names.gz capped at 50k sequences, and then David Hall reversed course — "no reasonable model is going to do well on these without thinking so they're not a great pretraining target" — and #5852 was closed. #data-mixing also picked up a batch of math/arXiv corpora from Benjamin Feuer (Swallow-Math-V2 at 32B tokens, UltraData-Math-L2 at ~34B, HAL Open Science, several arXiv dumps); David Hall asked the ingestion agent to file issues and PRs for the text and chat subsets only.

The headline result: the preregistered 1e23 MoE run at #4697 finished. Final paloma macro_loss was 2.234, less than 1% off the #4447 preregistered 2.252 fit extrapolated from runs 333× smaller — 130B total / 16B active / 1T tokens, with 6× better FLOP efficiency than the dense Delphi formula. David Hall announced it in #general, noting the after-the-fact dropless eval is what the headline number reflects (training-time evals were pessimistic because tight drop tolerance overwhelms load-balancing on clumpy eval batches) and flagging a separate infra post-mortem to come. Percy Liang framed the wider context in his #general roll-up: the first major GPU run kicks off in mid-June, so data mixtures, architecture, and optimizer all need to be locked down fairly soon.

Recipe-lock work converged this week around the canonical 1pct-noclip baseline from #5763, with the 2×2 warmup×clip ablation now complete. #5783 (2pct-noclip @ d1024) and #5784 (1pct-clip at all three scales) attributed the d1024 regression vs may_arch_default to dropping the grad clip, not halving warmup; the follow-on #5789 0% warmup variant landed within noise across scales. Larry kept the no-clip choice with the rationale that grad clipping behaves differently across scales as gradient magnitude shifts and the gain isn't worth that. The per-group LR sweep #5773 revealed an embed-AdamH stability cliff — pushing embed LR up by even 1.2× induces large grad spikes — so #5804 swept the alternative (embed on plain Adam, LR× ∈ {0.7, 1.0, 1.3}) at all three scales; the LR knob barely matters and Larry moved embed back to AdamW, accepting late-run crystallization as the lesser risk versus 10–10³× grad spikes if the embed-AdamH LR scaling is off.

Three feature flags graduated onto the production recipe. Renormalizing sigmoid combine weights to a fixed sum X=2 in #5797 won d512 at −0.0051 macro / eq-TPS 1.027 (and superseded the earlier routed-expert output-scale attempt #5782); the storage-only split of w_gate/w_up in #5794 kept a clean d768/d1024 win without any optimizer change and is in; and the segment-safe PKO fix #5795 landed on pko_first_bos_zero (PKO+zero, THEN norm) as the canonical leak-fix after the consolidated table found removing PKO entirely costs +0.034 / +0.018 macro at d512/d768. AuroraH on the expert I/O #5785 and NorMuon at #5793 were both shelved for TPS hit / marginal gain; the per-layer shared-expert sigmoid gate #5800 won d768 by ~0.003 but was dropped over instability risk; the GN w_up 0.8× / w_down 1.2× LR shape #5792 and the finer QK/VO LR splits #5788 were declined for lack of strong theory or material gain. These four flags compose the "no-arch" baseline now feeding the 3×6×5 = 90-run #5818 tokens/active-param × LR sweep that Larry Dial kicked off, the input for the next round of LR scaling fits at the May recipe.

Branching out from recipe-lock, three new agent-launched experiments queued at week's end: the no-XSA + per-head context RMSNorm variant in #5854 passed gate-1 at both scales (wall-clock 1.108× / 1.144× from removing XSA's per-head dot/scale/subtract) and went to gate-2; three parallel attn/MLP block layouts at gate-1 #5932, a muon_epsilon sweep at d512 #5933, and a GLU-vs-SwiGLU gate-1 #5934 all kicked off via the agent.md autonomy contract. David Hall also did long-overdue triage on the legacy Good-10T / Great-10T ablation buckets (#4015–#4046) in #moe, closing out the ones with clean verdicts — keep attention gating, shared expert, GatedNorm, XSA, and SWA; E=128 passes; no first-k dense; don't switch to inv-sqrt schedule — and flagging the open ones (E={128,256,512} sweep, K ∈ {1,2,4,8} sweep, MuonH/NorMuonH work, QB vs sign+aux, raised z-loss #5600) that are still moving. Will Held's #5414 long-context plan firmed up around a Command A+-style 3:1 short:long with context extension to 32k→64k→128k post-pretrain, with checkpoints saved at 70/80/90% of each MoE Grug ladder run for downstream extension iteration. Backing this up, #5036 landed Grug backward-flow logging (sampled every 50 steps by default) with W&B HTML rendering and residual-stream DAG artifacts, and #5718 wired the Sonic raw-Triton MoE backend through jax-triton for the GPU-side hand-off. #5625 opened a Dion optimizer port (full sharded multi_transform, CPU tests only so far) as a parallel option for that hand-off.

The draft TE-THD path from last week, #5749, did not land. David Hall closed it as superseded once the FA4/CuTe approach proved faster than Transformer Engine on the same shape, and reopened the implementation as the cleaner attention-only diff #5880, which merged on May 22. The new backend lowers Grug dynamic packed segment IDs to per-token lower bounds and dispatches to FA4/CuTe forward and backward kernels behind an opt-in gpu_fa4_cute entry point, without materializing a dense [B, S, S] mask. SM100 dispatch and a B200-specific tile config went in with it, and a noob-friendly explanation in PR comments noted that the compact segment_ids == segment_ids.T representation is all Levanter needs because the only masks in use are causal and segment-id.

The B200 numbers, tracked in the new experiment issue #5896, are the first time the FA4/CuTe path has measured ahead of a working external baseline on this shape. On B=1, S=8192, Hq=32, Hkv=8, D=128, segment_len=256 BF16, the patched FA4 with forward_tile=(128,64), backward_tile=(64,64) runs forward+backward in 1.021 ms versus TE flash at 1.229 ms; forward-only is still slightly behind at 0.347 ms versus 0.317 ms. The backward tile choice was the lever — focused sweeps showed (64,64) backward is about 1.63x faster than (128,64) with identical small-reference numerics, while a wider (128,128) forward tile regressed the same segmented shape badly. TE auto and fused remain unusable for comparison because both clean and overlay environments hit a Multiple libcudart libraries found conflict inside TE fused attention, so the published gap is against TE flash only and confidence is one B200 shape replicated, not a stable shape grid.

On the MoE side, the SonicMoE thread reached a stopping point. #5718 landed the production-facing sonic local backend that calls SonicMoE's raw Triton gather/combine through jax-triton with a custom VJP while keeping the grouped GEMMs in JAX ragged_dot and the concat W13 layout; the earlier sonic_xla and down_gather reference backends were removed, and jax-triton/Triton moved into the GPU extras. In review David Hall framed the value as more about plumbing — landing the jax-triton/jax-cutlass connection and a working relationship with Dao-AILab — than about Sonic itself, noting that the GMM work in #5894 is the bigger lever and that Sonic does not yet handle EP. For the local GMM, #5899 tuned the Blackwell ragged-dot default to a wider block_n=256 Triton tile while preserving the existing block_n=128 path on other GPUs; on the EP8 local shape from a recent moe_1e23 run, the new tile beat the prior baseline by about 1.19x on W13 (1.8789 ms -> 1.5735 ms) and 1.20x on W2 (1.0781 ms -> 0.8985 ms), with a public-path confirmation matching at W13 1.5811 ms, W2 0.8996 ms.

The reproducible BF16 B200 Grug-vs-Megatron profile that all of this is supposed to feed into, #5815, still has no accepted result — the stack comparison still rests on the single-GH200 synthetic profile where Megatron-Core was about 28% faster than strict Grug. The companion FP8 issue #5816 was opened with the explicit decision to validate FP8 only after BF16 is reproducible, and David Hall's earlier draft generic dense FP8 wiring #5589 still sits unmerged with a self-assessment that it is probably wrong as written. Two pieces of perf housekeeping landed alongside: #5882 switched the Levanter Pallas GPU fused cross-entropy tests to the streaming forward path because the tiled Mosaic lowering does not support the required dot_general on current NVIDIA GPUs and raised GPU batch block sizes to 128, and #5893 cut the paged decode prefill chunk test from a Cartesian grid to representative cases to reduce compile-heavy test cluster load while keeping coverage of prefix sizes, chunk sizes, and sequence slot patterns.

No direct PRs landed against this epic and no multi-host H100 run was attempted this week; the manual two-host canary remains the high-water mark, and the cold-start rendezvous tracker #5480 is still open and untouched. The one categorized issue, #4311, was used by David Hall to fix an explicit acceptance target for the H100 throughput line: within roughly 10% of the best Megatron/TE BF16 result on the same relevant shape, measured on stable post-warmup step time and tokens/s. BF16 stays the baseline; FP8 is not yet a reliable speedup claim. The strongest comparison on record is still the single-GH200 synthetic profile where Megatron-Core plus NGC PyTorch plus TE plus Apex was about 28% faster than strict Grug, which is attribution evidence rather than a full-node parity number.

The active work this week sits one layer down, shared with #5357 H100 kernel perf, and it largely pivoted to B200. David Hall merged #5880, an explicit GPU FA4/CuTe Grug attention backend for dynamic packed segment IDs that lowers segment IDs to per-token lower bounds and wires forward and backward CuTe kernels behind an opt-in path. The companion B200 issue #5896 records the measured 8192-token baseline: on B=1, S=8192, Hq=32, Hkv=8, D=128, segment_len=256, the patched FA4 with forward_tile=(128,64), backward_tile=(64,64) ran 1.021ms forward+backward versus Transformer Engine flash at 1.229ms, with forward-only still slightly behind TE flash at 0.347ms vs 0.317ms; TE auto and fused remain blocked by a mixed libcudart.so.12/libcudart.so.13 conflict and are not yet a valid comparison. #5899 tuned the Blackwell ragged-dot forward tile to block_n=256 for the MoE BF16 GMM microbench while preserving block_n=128 elsewhere; the companion tracker #5894 reports the new tile beating the previous m128_n128_k32_w4_s4 by about 1.19x on W13 (1.879ms to 1.574ms) and 1.20x on W2 (1.078ms to 0.899ms) for the EP8 local MoE shape, with public-path confirmation on Schmidt B200. The SonicMoE local backend from last week, #5718, also merged.

The other thing that moved this week and bears directly on multi-host viability is Will Held's #5871, which fixed a multi-host hang introduced by the background tracker. BackgroundTracker was materializing jax.Array metrics on its daemon thread, but for sharded arrays that materialization is a cross-host collective and JAX collectives must dispatch from the main thread in identical program order on every host; the desync surfaced as RuntimeUnexpectedCoreHalt: An unexpected peer shows up in the launch group with a different launch id, and Grug MoE hit it every step through _summarize_router_metrics's sharded routing histograms. The fix calls jax.device_get on the trainer thread before the payload crosses the queue and was validated end-to-end on a v5p-32 4-host slice where the unfixed config halted at step 3 and the fixed run completed 50/50. The repro was on TPU, but the failure mode is the same any-host JAX-distributed program order issue that a multi-host CoreWeave H100 launch would hit, and it removes a class of silent-stall surface area that the eventual H100 run would otherwise sit on top of.

Net for the week: the H100-on-CoreWeave goal saw no direct progress, but it now has a written acceptance bar from David Hall on #4311, an upstream multi-host correctness fix in #5871, and continued kernel gap-closing on the GPU side. The attention path now has a measured B200 baseline that beats TE flash on forward+backward at the 8192 target shape, the Blackwell GMM tile is a step closer to the EP8 MoE optimum, and the Sonic local MoE backend is in. What is still missing is a launched multi-host run and any movement on the cold-start rendezvous itself.

The bulk of leftover activity was repository hygiene driven by the @claude-nightshift bot, which landed daily multi-cleanup sweeps and a recurring slow/flaky CI investigation (#5805, #5839, #5876, #5903, #5915, #5796, #5904, #5916, #5936). Alongside that, Russell Power and Will Moss tightened the developer toolchain: pre-push lint rules #5843, a refactored pre-commit file selector #5840, advisory lint-catalog review #5873, skill metadata checks #5895, and trusted PyPI publishing for marin-* libs #5870 with a follow-up gate fix #5868.

Code-quality work rounded things out: Rafal Wojdyla added type annotations across marin.utils #5878, Will Moss pruned stray local imports #5908 and silenced leaky-thread test warnings #5835, Russell Power renamed skills to verb-first and dropped vestigial ones #5858, and David Hall fixed executor-step resource preservation #5834. Rohith Kuditipudi repaired the egress report's Discord auth and switched its output to a gist #5758, and David Hall documented PR review automation #5881.

External GitHub contributions clustered around Zephyr and the new infra/datakit seams. Will Moss pushed seven PRs through the Zephyr stack, including #5814 for multi-threaded workers, #5859 for parquet splitting and batched returns, and #5860 migrating fineweb / exact dedup to the new features, alongside refactor proposals in #5850 and #5855. Tai Vu opened #5841 adding a NuminaMath-1.5 datakit source and #5935 a Datakit-backed MathNet SFT pilot, plus a full-vocabulary entropy metric for RL in #5684. Alex Merose filed #5861 experimenting with DataFusion-backed SQL queries as a Zephyr Dataset source. Rohith Kuditipudi fixed the egress-report Discord auth in #5758. Yiyuan Li filed two tokenizer-efficiency issues, #5837 and #5842, comparing Marin coverage against Qwen3 and probing tokenmonster integration, and Moo Jin Kim opened #5853 requesting on-policy distillation support.

On Discord, @dlwh announced the compute-optimal 1e23 MoE finished at 2.234 Paloma macro-average loss against a 2.252 prediction preregistered in #4447 from runs 333x smaller, with 6x better FLOP efficiency than the Delphi dense formula. @Percy Liang posted a full project update covering the 18T-token corpus, the new Iris infrastructure, upcoming GPU access, and the mid-June kickoff for the first major GPU run. The tokenizer thread also produced a substantive exchange where dlwh pushed back on the tokenmonster paper's confounds while engaging Bilibird's vocab-compression numbers, feeding directly into Nativeatom's filed issues and Pranshu Chaturvedi's 32K parallel-tokenizer runs on Iris.

Seven members introduced themselves: Sunghyun Cho, an ML systems engineer with PyTorch and DeepSpeed contributions; Hoyoun Jung, working on LLM pretraining and training efficiency; Pushkar Patel, learning inference engineering and kernels; Jaewoo Kim, focused on LLM inference optimization and compression; Kota, a Nueva School student researching looped LLMs and efficient training; Jiaxuan Zou, an undergraduate at Xi'an Jiaotong and intern at Renmin University's Gaoling School of AI; and Jitesh (optima92), an MTS at resolve.ai who previously led Gemini Canvas and Gemini Extensions / Tool use. Among silent joiners in welcome-room, Tianyi Zhou is a tenure-track assistant professor at UMD working on human-AI alignment and controllable LLM training, and Morris Yau is an MIT PhD candidate (advised by Andreas, Jegelka, Moitra) whose theoretical work on neural sequence models intersects the scaling-laws and optimizer threads; Alexander Rosenberg Johansen (alrojo), a Stanford CS PhD student in convex optimization, also joined quietly.

News traffic orbited scaling and training dynamics: introspective / feedback-conditioned training, MoE sequence-level load balancing, attention and residual sinks, agent learning from early experience, and overtraining vs. downstream fine-tunability that fed back into the Nemotron 3 Super thread in #data-curation.