Marin: Week of April 20th summary

The 1e23 MoE run #4697 moved from launch into ongoing operation this week. @dlwh posted a tracking update on Apr 20 cataloging three concurrent W&B runs for the d5120 / 48-layer / 129B-total / 16B-active model: an earlier divergent EP8 ragged relaunch, the current EP8 ragged relaunch (moe_1e23_d5120_bs2048_ep8_ragged_48l_rayuvtpu_20260417_124933), and the known-good EP4 ring baseline. The run-specific launch artifacts were kept on the long-lived codex/grug-moe-debug-artifacts branch with the freeform debug log split off to codex/grug-moe-debug-log; the reusable config plumbing landed cleanly via #4964, while #4959 was closed as experiment-only. Mid-week, @Helw150 regenerated the 1e23 forecast for Percy's ICLR presentation using @eric-czech's VPNLS approach, getting a slightly more conservative paloma/macro_loss projection of 2.295 against the original 2.25 prediction (asymptote pinned at 1.6, following Larry's setup). @eric-czech warned that VPNLS came in too conservative on the Delphi forecasts unless fitted without E; @ClassicLarry noted the 1e22 MoE point in the comparison plot was from the prior WSD less-tuned LR recipe and not in the fit, leaving the curve only mildly off-trend. By the end of the week @ClassicLarry was examining grad norm on the active 1e23 against the 1e21 reference, observing that nearly all the gradient comes from output_proj which sits in AdamH at constant step size, so the existing >1.0 clip mostly does an awkward downscale of the embed and other non-AdamH params; @Kaiyue-Wen argued clipping is essentially never triggered now and even if it eventually triggers it should be benign for Adam/Muon, raising module-wise gradient normalization as a way to ignore grad norm entirely the way parameter norm is now ignored.

The dominant story of the week was the agent-driven sweep of architecture and recipe ablations against the post-isoflop compute-optimal baseline. The shape that emerged: a handful of small individually-passing changes turned out to be cleanly additive. Partial RoPE on every layer #4946 dropped macro_loss by 0.020 at d512 and d768 with essentially no throughput cost, giving 1.09–1.12× effective speedup; partial key offset (PKO) on the every-fourth long-window layers #4802 matched or exceeded that, with the every-4th variant capturing the full benefit at 1/4 the layers, mirroring the nanogpt approach. Combining PKO every-4th + partial rope every-layer #4951 hit 1.226× at d768. #4976 then forced the last layer to be long+PKO so the model always ends on an induction-capable layer, lifting effective speedup to 1.19–1.23× across all four scales with a projected 1e23 macro_loss delta of −0.012. Doubling routed experts to E=128 / K=4 #4767 contributed another 1.08–1.20× (improving with scale), and reusing the third-to-last layer's attention input for the last two layers #4987 added 1.02–1.10× essentially for free. Stacked together as the "combined best" run #4999, the configuration delivered 1.21–1.52× effective speedup across d512–d1280 with macro_loss drops of −0.054 to −0.067, and the fitted scaling law projects −0.035 at 1e21 and −0.023 at 1e23 (pinned exponent), or roughly 1.4–2.1× speedup against baseline at hero scales depending on whether the exponent is held or fit. @ClassicLarry flagged in #moe that the 52% gain at 3e19 was largely driven by partial key offset, an idea originally invented in nanogpt — @Kaiyue-Wen called it "crazily good." Late in the week #5154 opened a barebones-transformer probe (no XSA, no GatedNorm, no MoE, MHA, dense MLP) specifically to chase down why PKO shows ~20% benefit on the Marin MoE recipe but only 0–2% on the speedrun nanogpt/nanochat recipe, hunting for any tokenization artifact.

The MHA thread that opened on Apr 24 stands as the most consequential late-breaking result. Removing GQA entirely #5151 dropped macro_loss by 0.024 at d512 and 0.034 at d768 for ~2–4% throughput cost, giving 1.10–1.18× effective speedup — confirming the prior #4371 GQA estimate at the new compute budgets. Stacking MHA with PKO every-4th + last #5152 compounded almost additively: 1.28× at d512 and 1.37× at d768 from a single experiment, suggesting MHA is providing the KV capacity that PKO's induction heads then put to work. This sets up an obvious next combined-best refresh that swaps GQA out of the recipe entirely.

The negative-result column is equally informative for narrowing the recipe. Min LR ratio sweeps #4972 and the constant-final-LR variant #4981 both confirmed that fully decaying LR to zero is optimal at these run lengths — any non-zero floor strictly hurt. Doubling expert granularity to E=128 / K=8 with half expert dim #4899 improved quality slightly but cost ~30% throughput, failing decisively. Sandwich GatedNorm (post-MLP) #4993 hurt both quality and throughput. Shifting the AdamH:Adam LR ratio off 13/3 #5000 failed at gate 2 — the existing ratio is well-calibrated. Router/shared norm splits #4973, layer-grain prediction #4807, full attention residuals #5113 (good quality, throughput too expensive — block-residual variants #5110 won as bs=4 with 1.06–1.12× across all scales instead), GatedNorm init scale sweeps #4904, value embeddings #4986, and RoPE-before-QK-norm #5114 all failed or were neutral. The GatedNorm position ablation #4952 isolated the MLP GatedNorm as load-bearing while embed/final/attn positions are ~neutral, and the slim-layer probe #4900 identified that the long-window layer is the one MoE position that cannot be removed — pointing to a future experiment of beefing up the MLP at every-4th layers via a wider skip path. Depth-width shifts #5002 showed −1 layer passing gate 1 at 1.07–1.12× by trading negligible quality for ~10% throughput, suggesting the L = round(d / (64 + 4·log₂(d) − 9)) heuristic is slightly too deep; @ClassicLarry deferred a base=80 retune behind the other recipe changes since it would invalidate the isoflop sweeps.

The optimizer side picked up new contributors. @pc0618 launched a Muon AOL-coefficient sweep #5115 at gate 1 and a MuonH swap with 2× batch ablation #5134 after @Kaiyue-Wen argued in #moe that MuonH should show a clearer step-wise advantage if the AdamH LR is held fixed, and that Muon should benefit disproportionately from a 2× batch increase since orthogonalization overhead amortizes better at larger batches. The first MuonH launches hit a JAX ShardingTypeError on the ('data', None, None) × ('expert', None, None) broadcast and were patched with a reshard before the scale-invariant multiply path. By Apr 25 the early read was that AdamH baseline, Muon AOL, MuonH base-batch, and MuonH 2× batch all looked indistinguishable, with @Kaiyue-Wen noting the y-range made the comparison hard to read at a final-difference scale of ~0.02. @WhenWen opened parallel work on AdamH gradient normalization: per-module gradient RMS-1 normalization #5180 via #5181 regressed d512 enough to fail gate 1, prompting a global-tree variant #5182 via #5183 that preserves relative module scales — gate 1 was still in flight at week's end, with one relaunch needed after a parent preemption moved the executor prefix between regions and broke checkpoint resumption. @dlwh also landed sampled backward-flow probes for Grug #5036 with a residual-stream DAG renderer logged to W&B every 50 steps, giving the modeling group a new lens on activation and cotangent flow during these recipe sweeps. Looking past the hero run, @Larry noted in #moe that the 1e22 needed capacity factor pushed to 4 post-hoc to get an honest eval — c4_en BPB 0.4882 at cf=1 vs 0.4259 at cf=4 — because OOD eval distributions flood single experts and cause drops; @Kaiyue-Wen suggested a small DSv3-style sequence-level auxiliary loss to discourage the imbalance, while open questions remain about how to handle this at inference without giving up the EP throughput. The DeepSeek-V4 paper that dropped on Apr 24 prompted Larry to register intent to test their gating change in Marin — sigmoid is "a little odd" given gate values are always sampled from the right tail with no real 0–1 bound.

The Ray sunset arrived. @yonromai drove the long-tail migration from #4453 through to its terminal state in a sequence of stacked deletes: trivial v1→v2 import swaps in #4970, ports of export/levanter_checkpoint.py and inference/vllm_smoke_test.py to fray v2 in #4980 and #4983, removal of dead RL code (evaluate_environment.py in #4975, JAX_TRANSFER_SERVER weight transfer in #4979 — confirmed unused by @rjpower in #reinforcement-learning), the eval-tree v1 scaffolding in #4953, the Ray+Iris hybrid integration test in #5068, and then the heavy excisions: cluster/ray.py and the executor's lazy import ray branches in #5028, Levanter's ray_tpu.py/launch_on_ray.py in #5031, the Marin Ray glue in marin.cluster in #5131, the 16 Ray cluster YAML templates and operator docs in #5132, the Ray operator tooling and its CI in #5089, ray_run.py itself in #5087, the docs sweep retiring ray up/ray down/launch_on_ray from tutorials and runbooks in #5076, the fray.v1 package in #5137, fray.v2.ray_backend in #5138, and finally the fray.v2.* → fray.* rename in #5140 once nothing else needed the version suffix. @Helw150 tore the Ray references out of the internal guidelines in #4985. @yonromai's opening salvo on Apr 23 in #infra — "Is it okay if I delete all the Ray resources on GCP, except for big run?" — was the operational counterpart: 9 of 10 head VMs and 34 firewall rules went down on the 23rd, leaving only marin-big-run alive for the in-flight v4-2048 production job. Net deletion across the epic is on the order of 22k LOC; fray is now Iris-only with a LocalClient fallback for tests. Docs lag is tracked in #5029 (still open).

Iris absorbed the consequences of being the sole scheduler. @Helw150's #5081 turned on the budget system: migration 0037 set budget_limit=75000 with max_band=PRODUCTION for seven admins, the same limit at INTERACTIVE for thirteen named researchers, and forced new users to budget_limit=0/max_band=BATCH so unlisted submitters get opportunistic scheduling only. @rjpower followed with #5083, exposing --preemptible/--no-preemptible on iris job run so callers can override the small-CPU heuristic explicitly (closing #4540), and #5108 added a client_revision_date stamped at wheel build that lets the controller reject root submissions from clients more than a configurable interval old, with a 2-week grace for in-flight installs (closing #4840). The legacy monolithic Heartbeat RPC and its 2k-line surface — HeartbeatRequest/Response, use_split_heartbeat flag, HeartbeatAction enum, DispatchBatch, all begin/complete/fail_heartbeat paths — were removed in #5092 once #4984 flipped use_split_heartbeat=True on prod marin and the split Ping+StartTasks+StopTasks+PollTasks path had burned in elsewhere. The audit story moved from a SQLite txn_log/txn_actions pair to structured log lines (event=… entity=… trigger=…) in #5082, picking up API-key create/revoke instrumentation along the way. The deprecated GetTaskLogs RPC was removed in #4231 in favor of FetchLogs with regex source patterns. Underneath all of this, @rjpower began a major refactor of the controller's storage layer: #5147 introduced a typed ControllerStore bundling per-entity stores with a temporary self._db escape hatch, and #5164 then migrated ~30 inline SQL sites in ControllerTransitions behind TaskStore/WorkerStore/JobStore methods, lifted transaction scope to the entrypoint, added a TaskScope ADT, on-commit hooks for the attribute cache, and closed the submit-with-replace TOCTOU. Behavioral equivalence is proved by the 13-scenario replay-golden harness in #5165, which produces byte-identical DB state on the refactor branch versus main.

The scheduler had a bad Wednesday and a productive recovery. On Apr 22 around 01:15 UTC, Michael Ryan flagged "I think something might be up with the iris scheduler" in #infra; @rjpower traced it to a crashed scheduler loop, narrowed the cause to "reservation hold job preemptions had special casing which broke the scheduler", and noted that "likely 50% of the complexity of the iris scheduler is due to how I hacked those in… which I need to delete with great passion (hopefully by next week)". The fix landed as #5032 within the hour. A more pernicious class of bug surfaced over the rest of the week — a StartTasks→PollTasks race in which a poll's DB snapshot, taken before a concurrent StartTasks assignment commit, would omit the new task from expected_tasks, the worker would kill it as unexpected, the controller would promote that to JOB_STATE_KILLED, and the kill would cascade across the entire pool, surfacing in zephyr as the misleading "Worker job terminated permanently… Workers likely crashed" abort. @ravwojdyla-agent attacked the worker side first in #5043 with a 30s submission grace window in _reconcile_expected_tasks; @rjpower followed with #5046 (a wire-level TASK_STATE_MISSING signal that maps to WORKER_FAILED in the controller so the task retries via its preemption budget), #5054 (the controller-side companion to #5043), and ultimately #5090, which moved per-worker PollTasks out of its dedicated thread and runs it inline at the end of each scheduling iteration so the same thread that commits assignments owns the running-tasks snapshot. The autoscaler got its own structural rewrite in #4816, which lifted the slice lifecycle into an explicit (from_state, type(event)) → to_state transition table with a sum-type outcome consumed via a single match; failed slices are now atomically removed from _slices and surfaced through BecameFailed.handle rather than left in place. @rjpower's #5078 added a manual-slice CLI (iris cluster create-slice/delete-slice) that the autoscaler ignores, useful for pinning capacity outside the demand model.

Worker reliability moved on several fronts. #4883 replaced the time-decay health score with two independent ping-based termination conditions — 10 consecutive ping failures, or 10 monotonic BUILDING→FAILED transitions — and removed the heartbeat_failure_threshold config knob; FAILED from RUNNING deliberately doesn't bump either counter to keep poison-pill jobs from reaping every worker that ran them. #4940 wired Connect deadline checks into the RPC interceptor on entry and after semaphore release, so FetchLogs and heartbeats whose clients have already given up shed their work as DEADLINE_EXCEEDED instead of compounding the pile-up. #5021 cut the FrayActorJob polling load by switching wait_ready/is_done from the heavy GetJobStatus RPC to lightweight GetJobState with backoff out to 5s; #5024 restored the ConnectError(NOT_FOUND) public-API contract that #5021 inadvertently regressed; #4917 capped wait_for_job's state-poll backoff at 30s and moved wait_for_job_with_streaming off its fixed time.sleep. SIGSEGV exits in container processes — almost always libtpu/glibc/C-extension faults rather than user logic — now route to preemption_count (default budget 100) rather than the terminal failure_count=0 path in #5013, and #5038 reaps TPU hosts that keep failing launches (e.g. iommu/vfio group already held), excludes virtual reservation-holder tasks from expected_tasks, and fires on_stop on natural return. #5017 disabled the periodic memray loop after it began segfaulting the processes it was profiling; on-demand memory profiles still work, and #5033 stopped passing --subprocesses to py-spy when the controller and worker profile themselves, since they spawn non-Python children that py-spy can't fingerprint. @dhidary's in-flight #5150 adds enable_external_ip=False to the GCP configs and opens an IAP tunnel for projects (TRC grants, security-locked enterprise orgs) where constraints/compute.vmExternalIpAccess bans public IPs.

The week's operational narrative widened to compute economics and agent ergonomics. Eric Czech opened a long thread in #infra noting that practical cross-region sweeps are getting compressed into shorter and shorter windows as v4/v5/v6 availability bounces, and asking whether OA could offload checkpoint storage to R2 or a separate GCP project to escape hai-gcp-models egress. @rjpower pushed back on the storage-split path — egress prevents external projects from helping much for checkpoints — and argued that "in a sense, we may need a rewrite of Executor for the modern era", since the executor still resolves output paths in the root launcher's region and reuses them for child jobs in other regions (filed as #4969, still open). The recommended path remains mirror:// for checkpoints; the durable fix is fair-share priority bumping inside Iris, which works only if everyone submits to the same queue rather than letting agents place jobs themselves. The agent-babysitting pattern crystallized: by Apr 25 Eric reported a sweep of 8 training jobs running for "a little over a week with Claude babysitting it to keep things going", with anecdotal job-completion rates close to what manual checks would have produced, and @rjpower noted that optional retries on executor launches would close the remaining transient-bug gap. @Helw150 announced the deprecation of stanford-crfm/marin-tokenizer and the migration to marin-community/marin-tokenizer (#4977, accidentally deleted upstream), and an unrelated kitokenizer regression — add_special_tokens=False dropping per-document BOS — landed in production between Apr 8 and Apr 22 15:50 UTC and corrupted ~63 TB of tokenized/* data per the GCS audit cross-referenced in #5149 (@rjpower's gist of recent writes is the source of truth; @Helw150 and @ahmeda14960 are doing the affected-cache triage). The wandb>0.24.0 floor went in via #5011 to dodge the silent-upload bug from #5010. A handful of papercuts closed alongside: #4988 — --reserve ignoring --region when claiming reservation workers — was fixed by #4989; iris --cluster=<typo> now fails fast with the cluster list rather than a generic missing-controller error in #5016; iris --cluster=NAME with auto-tunnel became the documented primary CoreWeave connection pattern in #5001; step_runner's unmet-deps message no longer crashes with unhashable type: list per #4991. Larry's "iris giving us a lot of compute right now wow" on Apr 21 captured the underlying reality: with Ray gone and the budget system live, the cluster is finally doing the job it was redesigned to do, and by Apr 25 @ahmeda14960 could report 4,888 v5p chips up — enough headroom to consider a Qwen3 235B/A22B 3T-token training run in a week.

This week introduces a new infrastructure epic, #4474, that bundles three streams of plumbing work that had been running in parallel: rethinking the Levanter token store/cache, hardening Iris's K8s logging path so it can survive cluster-scale ingest, and closing a set of correctness and ergonomics gaps in the broader Iris/Levanter stack. The epic carries forward the design conversation that @ravwojdyla opened with new contributor Neville two weeks ago in #4445 — whether the producer-side consolidate_shard_caches dance the tokenize pipeline does to keep TensorStore happy can be replaced by something simpler (and ideally Parquet-shaped, "due to gravity"), without giving up the random-access-from-thousands-of-readers properties training depends on.

The store-side work this week was Neville's. #5070 landed a focused optimization to JaggedArrayStore's write path — a PreparedBatch._from_sequences() fast path that pre-allocates a single flat array for Python-list inputs and skips per-item np.asarray calls — taking a 1B-token write benchmark from 44.4M to 57.5M tokens/sec on local disk, and removing the unused TreeBatchPreparer while there. The larger swing, #4814, remains in draft: it removes consolidate from tokenize entirely and replaces it with an in-memory virtual ShardedTreeCache for downstream readers. Local tests pass and a tutorial CPU train works; @ravwojdyla granted Neville production Iris-cluster access so the next validation step is a 10BT tokenize run from the datakit-smoke workflow, with Iris dashboard and ops docs as the entry point. Together with #5023, which moves token-boundary rendering for perplexity-gap literal examples out of the document-scan hot path and behind a lazy contract, these three PRs are the first concrete steps from the rav/Neville design discussion into landing code.

The week's most visible incident was a tokenizer-correctness bug rather than a store-design issue, and it pulled the same surface area into focus. #5034, opened by @ahmeda14960 after he and Michael Ryan noticed mismatched bpb/perplexity between caches built three weeks apart, traced to Levanter's BatchTokenizer: after the early-April migration to the kitoken backend, the init probe used add_special_tokens=True but the hot path did not, so kitoken's explicit BOS gate stayed off and every text cache built between roughly 2026-04-08 and the fix landing on 2026-04-22 was missing the leading <|begin_of_text|> token (id 128000) on every document. The shared marin-community/marin-tokenizer was unchanged; the bug was purely on Levanter's side. #5040 from Ahmed dropped the probe, prepends BOS manually after encode_batch, and threads append_bos/enforce_bos through cache metadata so post-migration caches will now rebuild on next use. "yikes, no not intentional," Russell replied; "i hate HF tokenizers so so much." The blast-radius accounting moved to #5149, where @rjpower's three-week GCS audit gist identified ~63 TB of tokenized/* data inside the bug window; an empirical scan of the actual TensorStore arrays then confirmed the major training datasets were safe — primarily the validation/eval caches Ahmed and Michael had built were affected, and Russell posted the all-clear later in the week.

On the K8s/Iris-logging side, @rjpower is in the middle of decoupling the log server from the controller process so it can ship as its own image. #4947 relocates iris.cluster.log_store to iris.log_server.store, lifts JwtTokenManager out of the controller, deletes the subprocess-based log-server auto-spawn path, and adds an opt-in enable_log_server_sidecar mode that runs iris-log-server as a second container on the controller VM. The supporting plumbing for this — a JwtVerifier stateless enough to live in non-controller processes, plus a ServiceURL resolver that lets gcp://finelog-server just work — landed as the +2696/-126 #5161. Independently, two operational fixes: #5020 raises the controller's local log-store retention from 50 segments / 5 GB (less than a day at marin's ~6–7 GB/day ingest) to 1000 / 100 GB (~2 weeks), since the read path has no GCS fallback once a parquet ages out, and #5019 parallelizes cross-region log downloads with a 16-thread ThreadPoolExecutor that had previously been single-threaded. #5174, from @ravwojdyla, makes the log UI itself more useful: each line of a job-aggregated log view now starts with a T<N> link to the originating task's detail page, parsed from the existing LogEntry.key. The longstanding #4509 — moving K8s workdir files from ConfigMaps (which had accumulated to 17,800+ orphans on the CW cluster, degrading the API server) to bundle-store blobs — closed this week.

The remaining surface of the epic was a set of Levanter checkpointing and harness fixes. #4386 closed via #4387, adding a temporary_base_path to CheckpointerConfig so time-policy checkpoints route to region-local Marin temp buckets with TTL while step-policy/permanent saves stay durable; the still-open #5066 wires the new temp-checkpoint roots into the Grug and Marin Levanter launch paths, deriving them from the configured output path via marin_temp_bucket(..., source_prefix=...) and keeping forced checkpointer saves permanent. #5022 from Calvin-Xu fixes a more insidious bug — train_lm.py never waited for the final async checkpoint write before exiting and train_dpo.py waited on a freshly-constructed Checkpointer instance with nothing pending — by storing the checkpointer on the trainer and calling wait_until_finished() after training. @eric-czech cherry-picked two upstream Levanter eval-harness fixes (#4912 for the unsupported implementation kwarg closing #4852, #4911 for missing resource_mapping in broadcast_shard); #4977 finished the rename to marin-community/marin-tokenizer across configs, docs, and tests; and #5171 closed #5139 by dropping Levanter's protobuf<7 upper bound now that Ray (the original reason for it) is gone and TB/XProf compatibility was reverified. Two open items round out the picture: #5026, a step_lock self-race where successful executor steps report LeaseLostError on the terminal write_status because the lock is released before the lease's finalizer runs, and #5030, dlwh's request for backward-flow probe support on real ArrayStacked/scanned Grug models so logging doesn't fall back to stack.unstacked() and defeat the point — a baseline non-stacked run is up as the reference behavior.

A quiet week for the MFU epic itself. With the Triton ragged_dot kernel from #4297 and the JAX 0.9.2 compatibility result from #4455 both landed the prior week, no new throughput PRs or comments arrived against the open sub-issues — #4300 (TPU v4 25%-30% MFU sustained), #4302 (H100×8 MoE MFU), #4311 (Megatron throughput on 8×H100), #4312 (close end-to-end MFU gap), #4313 (2×8×H100) — and none gained activity in the date range. The Grug-vs-Megatron head-to-head that @chloechiaw volunteered to run on April 12 has not yet produced a follow-up in the visible record. With the 1e23 v4-512 MoE run now mid-flight and absorbing most of the modeling team's attention, the epic appears to be in a holding pattern between the kernel work that closed in mid-April and a likely renewal of the H100 measurement push once the big run finishes or stabilizes.

The headline this week was teaching agents to babysit jobs without leaning on a developer’s laptop. #5018 framed the problem precisely: Codex heartbeat sessions babysitting long Iris jobs were failing because they shelled out to uv run iris, depended on local GCP/W&B token caches, and broke whenever oauth2.googleapis.com auth refresh stalled — the recent perplexity-gap-post4962-rerun watch died exactly that way. #5042 from @dlwh landed the answer: a marin-mcp-babysitter entry point with resident Iris controller and log clients, structured tools for jobs, tasks, logs, workers, processes, and profiles, plus Zephyr progress parsing and a diagnosis layer that classifies stuck-assigned, OOM/exit 137, TPU/XLA bad-node, quota-backoff, dead-worker, and zombie-coordinator states. A follow-up the same day #5071 switched worker metadata to protobuf json_format and added Zephyr coordinator thread-liveness so zombie coordinators surface in the structured output. dlwh flagged in #code-review that he’d “sworn never to learn MCP” but it was the only path to giving Codex automations the network access babysitting needs — an admission Russell Power immediately tried to parlay into handing him k8s next. By Saturday Eric Czech reported in #infra that he’d had Claude babysitting an 8-job sweep for “a little over a week,” and used the same MCP surface to ask the agent for parent-job lifetime stats — about a day on average.

The Nightshift loop ran four nights in a row this week, each producing a small multi-scout cleanup PR with a haiku for a commit poem and a real verification trail attached. #4949 opened the week with three mechanical removals (an unused BuildResult dataclass in iris, twelve lines of commented-out slicing code dead since Sep 2024 in levanter’s JaggedArrayStore, two orphan helpers in marin), and the cadence held through #4998, #5044, #5122, and #5153. The wins are starting to extend beyond dead code: #5122 caught a real correctness bug in levanter.schedule.value_at_step — the forward iteration returned the first segment whose start <= step, so any multi-segment IntSchedule (including train_batch_size in the trainer) silently used the earliest value forever after the first boundary. The scout reversed the iteration, added regression tests covering scalar values, multi-segment boundaries, mid-segment lookups, and the pre-first-segment error case, and shipped it. The pattern of scouts filing no_change findings when the sandbox blocked their writes is also now load-bearing: #5044 — the same PR that landed two scouts’ cleanups — recorded the value_at_step bug as a finding-only note one day before #5122 picked it up and fixed it.

Two issues this week sharpened the meta-loop around how agents do research and maintain themselves. #5117 split out of this epic to codify a forage skill: a dedicated phase before the first hypothesis lands in an agent-research logbook, with explicit Marin-specific search surfaces (long-lived research/<topic> branches, the experiment issue template, .agents/logbooks/<topic>.md, snapshot tags, prior runs in experiments/ and W&B). dlwh’s thread quote in the issue body sets the tone — agents already know how to do lit search, so the skill’s value is “Marin’s spin,” not generic search heuristics — and the issue is explicit that no new MCP servers, paper-ingestion tools, or curated paper library should fall out of it. #5155 opened a parallel question for documentation: build a skill (or skills) that either generate docs/ on demand or flag staleness, so the canonical reference doesn’t silently rot while the agent-authored code around it churns.

Outside the formal epic items, the “I asked Claude” pattern kept widening its surface area. Ahmed used Claude to diagnose a tokenization regression where caches generated yesterday and a month ago produced different bpb on Paloma — “Claude is claiming our caches no longer have BOS by default”, filed as #5034, with an offer to “have claude make a quick PR” as the obvious next step — and later did “a very scientific analysis with claude” to produce #5149, the cleanup list of who needs to delete or re-tokenize which BOS-affected caches, even pinging dlwh that “Claude thinks u own” proteindocs. willheld posted an “Agent MoE Experiment” combining the best gates the agent had found into the leaderboard, and Larry’s “the agent.md file has formulas to convert a difference in macro_loss to a percent speedup” made clear the MoE recipe leaderboard now treats agent-readable instructions as a primary interface. rjpower landed #5039 to fix a silent ops-log loss (the global logs/ gitignore had been swallowing every postmortem written under .agents/ops/logs/) and to write the first ops log entry under the new path; yonromai’s #5073 mopped up the singular-vs-plural .agents/project/ path mistake from #4545 and updated the three AGENTS.md references that pointed at the old path. Both are small, but they are the kind of plumbing that decides whether the agent substrate compounds or quietly leaks.

The dashboard kept shipping polish and gained a real RPC observability story this week. @rjpower's #4950 tightened the RPC stats histogram from the coarse 1/2/5/10 schedule to three buckets per octave from 1ms to ~60s, redacted request previews server-side, and redesigned RpcStatsPanel.vue with inline per-method sparkline histograms, p50/p95/p99 ticks, expandable per-method sample scopes, and a Recent / Slow-and-errors split — closing #4706's ask for a real RPC status surface and giving operators something concrete to point at when "the controller feels slow." The same week resolved #4707 (silent log-pusher RPC failures), #4895 (controller audit logging), and #4564 (CPU profile hanging the controller) — longstanding "we can't reconstruct what happened" pain points. @dlwh's #3297 — Iris reporting a job RUNNING while W&B marked the underlying training run crashed — was also closed. #5072 remains open as the next-step proposal: structured logging on the log service so worker/task resource history can be offloaded from the controller's sqlite and queried analytically.

The smaller dashboard papercuts piled up too. @wmoss gave Iris a tab icon in #5077 ("felt like Iris needed more personality"); @ravwojdyla-agent made the jobs view mobile-friendly in #5170 with a clean sm-breakpoint switch between mobile and desktop layouts, added a star-and-filter affordance for top-level jobs in #5007 (URL-synced via ?starred=1, persisted in localStorage), and scoped the controller's state filter to top-level jobs only in #4994 so expanding a parent stops hiding its non-matching children. @yonromai fixed the infra status page history strips to render oldest-to-newest in #5172, and a Claude-authored fix in #5186 made the dashboard sparkline charts fill their container, closing @wmoss's #5185. #4942 swapped the opaque (peer, user_agent) tuple in stats extraction for a small dataclass, and #4896 repaired last week's cross-region ops detector so checkpoint writes from resubmitted jobs no longer get false-flagged. On the CLI side, #5025 caught that IrisClient.list_jobs was ignoring its states/prefix arguments on the wire and page-walking the entire jobs table on every iris job list — filters now push into JobQuery server-side. #5037 rewired bug_report.py off the deprecated GetTaskLogs RPC, and #5045 stopped a RemoteLogHandler leak across the controller test session that was producing flaky CI.

The other big strand was a Zephyr-to-Iris status pipeline. @wmoss's #5063 added stage-level throughput stats (items and bytes flowing through scatter ops) and a stack of follow-ups — #5141, #5143, #5144, and finally #5187 — ferries those stats to the controller, persists them in sqlite, and surfaces a markdown task-status field on the Iris dashboard that Zephyr populates with current-stage and progress info. @ravwojdyla's #5175 caught that map-only stages were emitting items=0 (0.0/s), bytes_processed=0.0MiB for the entire shard runtime — "noisy and misleading" — and #5176 suppresses those when nothing has been counted. @ravwojdyla-agent's #5136 made Zephyr re-raise parquet/vortex schema-mismatch failures with both the expected and inferred schemas, so the diverging field is visible without spelunking. On the testing side, @rjpower landed #5165 — a pytest-driven event-replay framework with 13 curated scenarios and committed golden DB dumps — explicitly as a behavioral fingerprint of main ahead of the in-flight controller SQL-store refactor.

Discord made clear the dashboard work is landing on real users. Larry on Apr 21: "iris giving us a lot of compute right now wow." Two hours after Russell's scheduler-restart for new optimizations, Michael Ryan flagged "I think something might be up with the iris scheduler?" — the scheduler thread had crashed on a reservation-preemption corner case; Russell noted it "need[s] some better monitoring around that or make the sql guards more robust" and shipped a fix the same hour. After Friday's restart he posted an unprompted change-log of the included Iris fixes — the kind of operator-facing comms the dashboard work is meant to support. Eric Czech, who has had Claude babysitting an 8-job sweep for over a week, used iris job summary to pull stats showing top-level parent jobs typically last about a day before something kills them — "anecdotally I think this is at least fairly close to what I would have guessed… a big improvement over the week or two prior." On the friction side, Eric and Tim O'Donnell are re-running gcloud auth login constantly; rav uses a service account, Russell suspects the openathena.ai org's session policy. Eric also caught another stuck Iris worker; Russell pulled the worker log and pointed at #5038 as the fix. The week's ugliest discovery was Ahmed's BOS-token regression in tokenized caches — same model on a month-old cache and a fresh one produced different bpb on Paloma — which became #5149, a multi-day blast-radius scan, and Russell's wishlist note that "I hate HF tokenizers so so much" plus a Versioned("tokenizer=20260423") proposal for eval tokenization steps. None of that is observability per se, but it underscores why #5072's structured logs and #4895's controller audit log matter: when something silently changes, the only durable defense is being able to query history.

The week's headline for the canonical pipeline was @ravwojdyla's landing of the datakit testbed baseline, an end-to-end ferry → tokenize → train arm wired off a single CLI entrypoint: per-source sample → tokenize → Grug-MoE training on a v5p-8, with a STAGING_PREFIX and TARGET_TOTAL_TOKENS_B at the top of the file and proportional byte-fair sampling that whole-shard-copies when the budget allows and row-samples the tail when it doesn't. Underneath sits the canonical source registry, a frozen DatakitSource dataclass per entry mirroring the 102 datasets in the marin-community/token-counts HuggingFace repo (99 active, 97 pinned with both revision and repo set — the subset the ferry can materialize); every staged path was cross-checked against gs://marin-us-central1/raw/. A companion datakit-smoke staging validator now runs alongside the daily ferry and exits non-zero with a per-path <status>: <url> report when any of the 70 unique staged_path prefixes is not SUCCESS — turning the registry into an executable contract rather than a lookup table.

The pipeline expanded source coverage in the same motion. #4892 added normalize_*_step factories for nsf_awards, nemotron_v1 (one step per quality/kind split, seven in NEMOTRON_V1_SPLITS), and nemotron_v2 (one per family/subset), validating nemotron_v1 normalize end-to-end on the quality=medium-low/kind=actual slice (1.24B records, 6,299 shards, 14.47 GB peak on 16 GB workers). The weekly Nemotron ferry wired the full normalize → minhash → fuzzy_dups → consolidate → tokenize chain on quality=medium against a verify-only download step that asserts the pre-staged gs://marin-eu-west4/raw/nemotro-cc-eeb783 dump exists and has .jsonl.* files, reusing the canonical override_output_path so StepRunner's cache check short-circuits on the existing STATUS_SUCCESS marker. #5158 tightened the tokenize boundary further by sizing the zephyr window and Levanter cache batch_size from parquet row-group metadata so each unit of work aligns with roughly half a row group end-to-end.

Running the full registry surfaced two correctness bugs that are easy to read as the price of finally exercising the long tail. #5162 reported that 5 of ~100 sources were producing mostly-empty parquet shards — coderforge was the worst case, with 643 of 672 shards carrying only the 176-byte footer. Root cause: scatter routes records into exactly num_output_shards buckets via hash(key) % num_output_shards, but the reduce stage was spawning max(input_shards, num_output_shards) tasks, so when the input had more shards than the output the surplus reduce tasks ran on indices no record hashes to and emitted empty files. #5166 fixed it with a regression test. Separately, #5142 hardened the parquet writer's schema inference: _accumulate_tables was inferring the schema from the first 8 records, so an optional field that happened to be None in the prologue got pinned to pa.null() and later real values crashed with ArrowInvalid — common-pile/stackv2's nested metadata.gha_language (959 null / 1041 str across ~2,000 records) was deterministically failing. Worse, pa.Table.from_pylist was silently dropping top-level keys missing from the pinned schema, a latent data-loss bug. The fix routes both cases through pa.unify_schemas with permissive promotion and reconciles prior chunks on yield.

The most consequential pipeline failure of the week, though, was tokenization-side and pre-existing: #5034 caught that Levanter's BatchTokenizer had silently stopped prepending the BOS token to each document when the tokenizer backend migrated from HF's PreTrainedTokenizerFast to the new KitokenMarinTokenizer in early April. Ahmed Ahmed flagged it after seeing different bpb/pplx on the same model evaluated against caches built a day apart versus a month apart — "Claude is claiming our caches no longer have BOS by default? was this intentional" — and Russell's reply, "i hate HF tokenizers so so much", captured the mood. The fix landed mid-week; #5149 tracks the cleanup blast radius across every cache built between roughly 2026-04-08 and 2026-04-22 15:50 UTC. Russell scanned for affected dirs with changes in the last three weeks and noted that the big training caches were spared — most of the damage landed on eval sets and on Rafal's nemotron experiments — but proposed a Versioned("tokenizer=20260423") notation on the eval tokenization steps as the safest forward path. In a quieter parallel thread, romain noticed the stanford-crfm/marin-tokenizer HF repo had been deleted out from under CI #4974; #4977 renamed everything to marin-community/marin-tokenizer, and #4971 opened the longer-horizon roadmap for derived 32K/64K tokenizers now and a trained v2 family later.

Beneath the data work, Zephyr itself absorbed a sustained polish pass. @hsuhanooi opened a sequence of focused PRs on the scatter/shuffle hot path: a byte-budgeted scatter write buffer that bounds write-side RSS regardless of item size or output-shard count (replacing the fixed 100K-row-per-shard threshold that could accumulate unbounded memory before close()), Arrow IPC and msgspec msgpack codecs as opt-in alternatives to cloudpickle for JSON-shaped chunks (each prefixed with a one-byte format tag so both formats coexist in the same scatter file), a /dev/shm-backed sidecar bytes cache that turns ~5ms GCS reads into microseconds on the second access, and a worker idle-poll backoff cap raised from 1.0s to 5.0s because subprocess-per-shard isolation now makes single-second polling pure busy-waiting. @ravwojdyla in parallel landed preemption-requeue exemption from MAX_SHARD_FAILURES — three clean preemptions had been aborting pipelines because the 3-attempt task budget was being consumed by infra requeues — split the attempt counter into TASK and INFRA kinds, and made INFRA requeues unbounded. #4968 (still open) folds M mapper sidecars into R per-reducer manifests in a new CombineMeta stage so each reducer does one manifest read instead of M; #5135 teaches connected-components to resume from the last complete iteration on opt-in. #5146, closed this week, made StepRunner.run() walk transitive deps internally so the per-script visit() boilerplate in scripts/datakit/ can go away. The cumulative direction is unmistakable: the pipeline is being shaken down by being run, end to end, every day.

With the marin-* wheels now publishing nightly, this week's work shifted from packaging mechanics to the API surface those wheels expose. The headline change was @rjpower's #5156, which sketches out service-design guidelines and, in the process, deletes roughly 2.9k lines while adding 416 — a substantial net pruning of the surface area an external importer has to reckon with. The PR sparked a small design discussion with @wmoss and @ravwojdyla about whether Tailscale or Cloudflare could replace the SSH/proxy stitching that currently fronts the cluster, with Will demoing the Tailscale UX (Google auth, taskbar toggle) and Rav noting that ngrok would equally suffice for the tunneling pieces.

The StepRunner got two fixes that move it closer to being something a Bolinas-style consumer can call with confidence. #5148 rewrites scheduling to do a deduped post-order walk of the dep graph so callers can pass only terminal steps and the runner resolves the rest, deleting an unreachable iterable-exhausted branch in the process. #4992 fixes a more embarrassing bug: the unmet-deps error path was passing a whole StepSpec (a frozen dataclass with a list[StepSpec] deps field) into a dict.get, which crashed with TypeError: unhashable type: 'list' instead of producing the intended RuntimeError naming the waiting step. A regression test now pins the right error message. On top of those, @wmoss's #5157 proposes an as_step_fn decorator that lets users skip the lambda op: ... boilerplate when building StepSpecs; Rav pushed back that the convenience introduces magic ("not good for humans, nor agents") and the PR is still open pending that discussion.

Three older tracking issues from the original Executor-as-tracing-model thread were closed during the week — #2287 (typed outputs from a tracing executor), #2349 (HF model download/convert step), and #2408 (download steps for models and tokenizers) — effectively retiring the Executor design discussion now that StepRunner + the marin-* packages are how the library ships. @wmoss's Claude-generated architecture docs PR #4960 was also closed this week in favor of tracking the broader docs-via-skill outcome in #5155; the discussion (Will, dlwh, Rav, @yonromai) converged on the view that hand-written architecture docs go stale fast on a project this young and that an "architecture-overview" skill run on a nightshift schedule is a better fit. Russell also opened #5067 proposing a consistent type|domain naming scheme for the GitHub workflows and a "minimal YAML" policy that pushes logic into independently runnable Python scripts — another step toward making the repo legible from outside.

Two smaller items rounded out the week. #4801 bumped pyrefly from 0.42.0 to 0.61.0 and regenerated the baseline (live errors stayed at 0; suppressions trimmed 173→169), keeping the type-checker honest as the package boundaries continue to shift. And on Discord, romain in #infra flagged that huggingface.co/stanford-crfm/marin-tokenizer appears to have moved to marin-community/marin-tokenizer without an announcement — a small but telling reminder that the HF-org migration that accompanied the marin-* rename last week is still surfacing rough edges for downstream consumers.

Pass rates landed where the epic name has been pointing all month. The TPU canary ferry was 7-for-7 this week, the CoreWeave GPU canary 7-of-8 (~87%), and the datakit smoke ferry 6-of-7 (~86%) — all three above or within rounding of the 90% target, and a clean inversion of the picture two weeks ago when central2 Ray restarts were shredding interpretability of the numbers. The single CoreWeave miss on Apr 21 was run 24718701520, which is the exact failure @rjpower diagnosed with #5004: bash does not expand ~ inside the double-quoted KCI assignment in the GPU canary diagnostics step, so every kubectl call had been stat’ing a literal ~ path and emitting zero-byte logs that hid the real failure. The PR moved everything to $HOME, also uploads the claude-code-action transcript as an artifact (the GHA step UI was hiding it), and broadens the triage tool allowlist so the agent can run common shell helpers and write slack_message.md. @yonromai’s #5003 bumped actions/checkout, actions/setup-python, and the docker actions to Node-24-compatible majors ahead of GitHub’s June 2 forced migration, removing a wall of deprecation warnings that had been noise on every marin-canary-ferry-cw manual trigger.

The CoreWeave integration pipeline got a two-step timeout headroom fix after a green-but-tight regime started biting. @yonromai’s #5111 bumped hf_save_steps from 1 to 2 in the CW integration test, removing a duplicate end-of-train HF checkpoint save: with hf_save_steps=1 and num_train_steps=2, the per-step run_hooks(info) and the end-of-train run_hooks(info, force=True) both fired on step 1 and uploaded the model twice to the same destination, costing 3–4s of duplicated S3 I/O against a 600s budget that was already at 96% utilization on green runs. #5112 then raised the outer shell timeout from 600s to 900s — the prior 24806172401 run had failed with exit 124 about 42s after the final checkpoint save, before pytest could clean up. @rjpower’s #5125 bumped further, to a 1800s shell wrapper and 900s pytest-timeout, after observing that the full marin pipeline runs 8 sequential Iris sub-jobs each paying ~60–90s of pod startup plus dep sync, and that on the last CoreWeave CI run train_lm finished only 34s after pytest-timeout fired at 600s. The combined effect is enough headroom for cold-pod variance without losing the inner pytest deadlines that surface real hangs.

The cross-layer test-skipping problem from prior weeks closed out: @rjpower’s #5086 expanded paths-filter in the fray, zephyr, levanter, and marin workflows to include all upstream layers each project imports from, fixing #4728. The concrete motivating instance was iris commit cf05d94ee on Apr 12, which changed Constraint to store values: tuple[AttributeValue, ...] with no .value attribute and silently broke five fray assertions for two days because nothing fray-touching landed in the interim. The HF tokenizer 401 issue #4974 — stanford-crfm/marin-tokenizer was deleted out from under levanter-ray-tests on Apr 20, surfaced by @yonromai’s agent-generated repro showing anonymous 401 / authenticated 404 (HF’s “hide existence” pattern) — was traced in #infra to @dlwh’s recollection that “Yifan deleted it,” and resolved by migrating in-tree references to marin-community/marin-tokenizer via #4977.

Two open items frame the next phase. The GPU canary cutover from SlimPajama to Nemotron in #3704 — bundling the Nemotron CC upload fix, larger download-worker memory, the romain-nt CoreWeave Iris config, and the Zephyr/Iris startup-wait mitigation discovered while debugging — is sitting in draft awaiting a final ready-to-merge from @ravwojdyla. And @rjpower opened #5065, an Infra Stability Epic asking what testing gates should be required before considering an Iris/Zephyr change “done,” sketching a small/medium/large playbook with seconds/minutes/hours/overnight gate tiers and asking that the tests be easily accessible to agents with builtin reporting — positioning canary pass rate as one signal in a larger release-gating story rather than the only one. The cluster restarts that historically dragged pass rate down were less of a factor this week, though Russell restarted the Iris scheduler twice on Apr 22 after a reservation-hold preemption special-case crashed the scheduler loop (fixed in #5032) and again on Apr 24 to pick up new budget code — both quick enough not to dent the ferries.

SWE-ZERO crossed from pipeline validation into measurable downstream signal this week. The 140B-token scale-out #4719 is now ~20% complete (2.44M of 12.3M rollouts, 27.9B of 140B tokens) running ~9 v6e-4 jobs after @AlienKevin consolidated the fleet to US-only to eliminate trans-Atlantic egress. More consequentially, the quality-validation experiment #4898 shipped: SFT on Marin-8B base with 10K, 50K, and 100K SWE-ZERO trajectories, evaluated three times each on a fixed 100-task SWE-bench Verified subset under mini-swe-agent v1. Resolve rate goes from 0.0% baseline to 3.3% +/- 0.6% (10K), 4.0% +/- 2.0% (50K), and 5.3% +/- 1.5% (100K) — a positive scaling trend, but the headline is that purely synthetic, execution-free rollouts produce real SWE-bench capability on an 8B model whose training data contains essentially none of the eval repos. The three published checkpoints (10K, 50K, 100K) and the eval-trace dataset are the first artifacts that turn last week's "the pipeline works" into "the data teaches the model something." A 500K SFT is in flight to extend the curve.

The flip side surfaced a sharp generalization-failure mode worth flagging. At 100K trajectories the model degenerates into immediate !!! repetition on 60% of trials versus ~20% for 10K/50K, and Kevin's root-cause analysis traces it to markdown code fences in user prompts: 100K degenerates on 77% of code-block prompts vs 15% on prompts without — a U-shaped sensitivity curve where 10K is too undertrained to disambiguate input vs output code blocks, 50K resolves the distinction, and 100K erodes it again via catastrophic narrowing on the SWE-ZERO format distribution. The mean resolve rate is still highest at 100K because gains on non-degenerate trials outweigh the degeneration, but it's a clean reminder that single-run SWE-bench numbers at small scale are unreliable (a 4% single-run result was initially read as a regression before the multi-run rerun corrected it to 5.3% +/- 1.5%) and a concrete signal for the eventual 140B SFT design — likely lower LR for longer training, or augmentation with code-fence-heavy prompts.

The agentic-SFT NemotronTerminal arc hit a less encouraging milestone. The Marin-32B 15% Terminal-Corpus run #4760 finished all 859 steps at loss ~0.49, but TB2 evaluation produced 1/89 = 1.1% across three trials — versus 17.2% for the Qwen3-32B SFT at step 500 of #4307 on the same harness, and 27.4% for the published model. Debugging the gap consumed most of the week: an early hypothesis was vLLM TPU architecture incompatibility (Marin-32B differs from Qwen3-32B in attention heads, vocab size, intermediate size, and uses llama3 RoPE scaling), then XLA compilation timeouts, then chat-template thinking blocks. The actual root cause, found by logging the raw vLLM response, is that the model emits ~2K characters of coherent reasoning and then falls into zorazorazora... repetition for the remaining 124K characters until hitting the 24K-token output cap. Sweeping inference settings on the git-leak-recovery task showed the degeneration is mitigable: temperature ≥0.8 (with or without repetition penalty) produces coherent output, while repetition/frequency penalties at temp=0.6 do not. The Qwen3-32B SFT itself improved to 17.2% with n_concurrent=10 sharded eval (up from 14.5%) and remains the team's strongest TB2 baseline at only 8.7% of full training. Whether the Marin-32B underperformance is fundamentally architectural or a fixable inference-config artifact is now the open question.

The instruction-data surface area expanded along several axes. PR #4329 from @Helw150 merged, adding download-and-transform pipelines for six HuggingFace rollout/reasoning datasets — CoderForge-Preview, GPT-OSS-20B rollouts, Nemotron-Terminal-Corpus, Principia-Collection, Superior-Reasoning-SFT, and SYNTHETIC-1 — each with pinned revisions and an experiment script under experiments/rollout_data/. @taivu1998 opened three SFT/RL PRs: PR #4996 registers FineProofs-SFT as both a raw-messages and proof-only view for long-context reasoning, PR #4997 adds NuminaMath CoT and TIR with a staged CoT→TIR experiment that warm-starts Stage 2 from the Stage 1 HF export, and PR #4684 wires native Reasoning Gym environment support into Marin RL with a minimal curriculum example and tests for environment loading and rollout statistics. PR #4620 (the explicit KL config refactor) gained additional review activity and remains open. PR #5035 unifies RL rollout decoding across vLLM and Levanter around a shared configuration, records applied decoding on rollouts, and adds real Levanter top_p support through the native inference engine — a substrate change directly relevant to how SWE-ZERO-style rollouts get produced consistently across backends. In #code-review, @taivu1998 requested review on the FineProofs and NuminaMath PRs together, and in #data-curation @cs2716 surfaced core.ac.uk's 2.7TB open-access research-papers corpus as a candidate source.

The online-distillation thread from last week continued to develop in the background — not as a new PR, but as a steady drumbeat of relevant references. @willheld shared AI2's BAR post in #news on Apr 20 — a recipe for training domain experts independently and merging them into a single MoE — to which @dlwh replied "oh man i wanted to do this at some point," and willheld noted that BAR plus FlexOlmo are good companion reads to the prior week's distillation conversation. On Apr 24 willheld also posted the TIP paper on token-importance in on-policy distillation (which trains a student on its own rollouts under token-level teacher supervision), explicitly cc-ing for "PPL gap sets" — a direct connection to the proxy/prediction work in the downstream-scaling thread, where @RohithKuditipudi is soliciting mid/post-training mixes from @ahmeda14960 and willheld to run across the scaling ladder for issue #4547. The synthetic-data epic and the prediction/proxy epic are visibly converging — SWE-ZERO produces the trajectories, the scaling ladder will tell us whether smaller-model SFT outcomes predict larger-model behavior, and the BAR/TIP/OPD literature is shaping how the team thinks about the eventual recipe.

The data conversation this week was reorganized end-to-end around a single diagnostic question: where, exactly, do Marin base models lose bits to Llama and Qwen, and what data should fix it? @dlwh landed the gap-report machinery in #4962 — pairwise Levanter-loadable LM comparisons that attribute bits-per-byte gaps to datasets, segments, and surface-form literals — and used it to file a confidence-portfolio epic in #5005. The framing in that issue is explicit: don't define one scalar for "good base model"; build a portfolio of evidence that catches broad failures before expensive post-training and RL evals. The first 8B reruns from #5023, summarized on #4961, located the largest losses in code-shaped surfaces (github_cpp +0.0345, github_python +0.0264, c4_en +0.0279, redpajama +0.0494 bpb against Llama 3.1 8B) and in byte buckets that survive cleaning — text/url, text/non_ascii_word, whitespace/newline, text/number. The conclusion is that the gap is a raw-technical-surface gap, not a GitHub-only crawl gap, and the proposed fix is a new raw_technical_text tranche that unions surface-preserving sources (StarCoder, StarCoder2 extras, Common Pile stackv2/github_archive/ubuntu_irc/python_enhancement_proposals/stackexchange, and raw Dolma algebraic-stack/arxiv/open-web-math/stackexchange) with a follow-on raw-HTML tranche tracked separately on #5012.

From there @dlwh opened a coverage map of "unknown-unknown" PPL slices and matching data-sourcing tasks that ran the length of the week — #5056 raw web/markup/image-text, #5057 binary/network/security artifacts, #5058 bio/chem notation, #5059 time-series/tabular/geospatial, #5060 formal-methods/RTL, #5061 package metadata, #5062 game/music notation, #5093/#5094 diagnostic logs, #5095/#5100 diff/patch corpora, #5096/#5101 paraphrase/translation robustness, #5097/#5102 ASR/OCR-noisy text, and #5098/#5099 GH Archive structured output. Sister coordination issues #4963 (chat/agent/numeracy PPL), #5006 (strong-model sampling as gap-discovery), and #5053 (LM-Eval dev splits as PPL datasets) round out the portfolio. The matching wave of slice PRs landed against the long-tail registry: SVG-backed raw markup #5130, first-wave tabular #5129, formal-methods/RTL #5128, bio/chem #5127, UWF Zeek security #5126, package metadata #5124, capped diagnostic-log builders #5104, diff/patch builders #5103, GH Archive #5119, ASR/OCR #5118, paired robustness #5120, game/music #5080, synthetic reasoning #5133, and a tokenizer-axis diagnostic #5085 for whitespace-sensitive formats split off from #5079. Two cross-cutting infrastructure PRs underpin the wave: #5169 caches per-model PPL scores once per bundle so the cost scales with models rather than model pairs, and #5084 adds a typed registration helper that keeps default validation sets untouched while the long-tail bundle is opt-in.

The first 32B-scale validation came at week's end. #5123 wired a capped Marin 32B vs Qwen3 32B run across the first-wave log/diff/robustness/ASR-OCR/GH-Archive slices, and the result on #5005 was sharp: across 3,520 docs Marin is +0.0142 bpb worse overall, but the loss is concentrated in GitHub-shaped structured programmer text — GH Archive +0.0211 bpb, GHALogs +0.0233, LogHub Apache +0.2895, CommitPack +0.0691 — while Marin actually wins on PAWS paraphrase (-0.1878) and FLORES eng-deu translation (-0.0700) and on both ASR and OCR slices. Pattern attribution put text/url at the top (+103,954 lost bits across 4.13 MB at +0.0252 bpb), with text/number and whitespace/mixed next. The signal is consistent with the 8B reruns and gives the long-tail program a real result to point at, though @dlwh explicitly raised the worry of benchmaxxing on PPL proxies, asking whether agentic-RL suitability can be approximated this way at all without RL'ing.

Multilingual coverage moved in the same week. #5008 added a reusable FineWeb2 multilingual eval bundle (top-50 languages by row count plus native-script Indic configs, pinned to HF parquet test splits and tagged for aggregate/script/language/Indic Levanter metrics), and #5074 wired the matching gap-rerun runner that compares Marin 8B against Llama and Qwen with Paloma and uncheatable held in the same report so multilingual regressions are visible next to broad raw PPL. On the data-source side, @Helw150's #4326 — a download/filter pipeline for HPLT v3.0 English keeping only non-CommonCrawl sources (WIDE, survey crawls) and applying register-based quality filters validated against a Haiku oracle at ~99% agreement, projected ~450B unique English tokens — picked back up after @ravwojdyla apologized for missing it and offered to rebase onto the new datakit registry from #5159. @Helw150 also opened #5009, a DFS stitching pipeline for Common Pile Stack v2 that groups file-level records by owner/repo@commit and concatenates each repo's files in depth-first path order to produce one document per repo, targeting long-context code pretraining and closing #4978. Long-context audit work continued separately on #4735, where the FinePDFs English slice review from #4738 ran against finepdfs_by_language["eng_Latn"] in us-east5 to produce per-source bucket statistics for the exp2062 long-context pilot.

The Luxical-as-data-integration thread on #3049 reached a close. The N=1000 capacity-ladder run (Luxical-192d / Arctic-L-1024d / BGE-large-1024d plus Dolma-3 fasttext baselines and oracle test-retest ceilings) put Luxical at Spearman 0.707 on Claude-rubric quality, Arctic at 0.771 — decisively above the fasttext baseline at 0.258 on the Claude rubric, but below fasttext on Nemotron-bucket quality (0.415 vs 0.588) and on 24-class topic classification. Capacity is not the bottleneck (BGE at 1024d ties Luxical at 192d on quality), and oracle noise is not either (Claude self-agrees at 0.964). @ravwojdyla's tldr: "we can certainly make the embeddings work, but the question is how do we try to make those decisions going forward" — the proposal is to formalize the run's artifacts into a reusable "datakit testbed" for benchmarking dedup strategies, candidate embedders, quality filters, and topic classifiers against a canonical set of probes. In Discord, @cs2716 flagged CORE (2.7 TB of open-access research papers with extracted text or download links) as a possible source for the technical-text tranche, and a new MIT MathNet release (30k problems across 17 languages) showed up in #news as a candidate for the multilingual-reasoning side of the #4963 coverage map.

External contributors carried significant chunks of infra and modeling work. @wmoss landed Zephyr/Iris PRs #5145, #5077, #5064, and #5063 on stage-level throughput, Arrow Flight, and dashboards. @hsuhanooi's scatter-codec investigations included an end-to-end msgpack-vs-cloudpickle benchmark on #5091 that closed after pickle won on Marin's string-heavy GCS workloads, plus the OOM-proof scatter buffer in #5055. @WhenWen drove an agent-run depth-MuP LR sweep across d512–d1280 on #5178 (no promotable speedup) and an AdamH global-grad-norm variant #5183 that cleared gate 1. @nevillelyh validated the consolidate-less tokenize path in #4814 with end-to-end Iris canary jobs and merged JaggedArrayStore optimization #5070. @eric-czech shipped levanter eval-harness fixes #4911 and #4912, and @MaxiBoether chimed in on #4445 from DatologyAI to share their in-progress open-source data loader.

Ten new members posted introductions: uwu1, RE/founder on multimodality and kernels; David Hidary, Columbia AI undergrad on a TRC grant; Iheb from TII Abu Dhabi, main contributor on Falcon-H1R / H1 / Falcon 3; Howard (zhipeng) from LFAI&Data, announcing a multi-modal speedrun under the Open Model Initiative built on modded-nanogpt and Marin speedrun; Yishun Lu, Oxford post-doc in optimization/HPC; Jeremi Nuer, UCSB student in robot learning and mech interp with a NeurIPS workshop paper on MoE superposition; Taksch Dube, Kent State PhD on multi-agent systems and applied category theory; Timur Kharisov, DL-theory PhD on optimal LR schedules and edge-of-stability; Nick DePalma, self-driving researcher with JAX-based robotics infra; and Wasiu "Truth", maths grad and OnlyDust/Protocol Labs OSS contributor. David's TRC context intersects with his first PR #5150 adding internal-IP-only iris deployments via IAP tunneling for TRC-bound TPUs; Timur brings context on the LR-schedule and stability questions running through grug MoE.

Eric Czech's #infra thread on resumable cross-region training turned into a long discussion with @rjpower on TPU availability, the cost model of mirror://, and whether the Executor needs a "modern era" rewrite that decouples reservations from data-locality — followed by Eric's report on a week-long Claude-babysat 8-job sweep that mostly held without manual intervention. Ahmed M Ahmed's BOS-token regression report opened a multi-day cleanup in #5149: caches built without per-doc BOS were silently degrading runs and impacted tokenization outputs got rebuilt. In #moe, Larry's 1e22 capfactor finding — OOD evals flood a single expert and drop tokens; raising eval-time capacity factor closes the gap from 0.4882 to 0.4259 bpb on github_cpp — fed into EP=8 1e23 planning and Kaiyue-Wen's pitch for a sequence-level auxiliary loss in the dpsk-v3 spirit.

Reading in #news clustered around modular post-training and MoE routing — AI2's BAR recipe for merging independently-trained experts, the DeepSeek-V4 PDF with skeptical commentary from Larry on its sigmoid-gated unbounded right-tail, "TIP: Token Importance in On-Policy Distillation", and a layer-residual rethinking thread proposing alternatives to the ResNet-style depth highway untouched since 2015.