Research & Benchmarks · Apr 10 → Nov 4, 2026

Measured outcomes across 14 Apple Silicon releases.

Every benchmark, failure mode, and speed trade-off measured on Apple Silicon—from our first falsified v0.0 lossy-buffer test (April 10, 2026) and the retracted v0.2 erratum (May 11, 2026) through v4.9 18-repo post-Sept-2026 held-out PR localization, 5-language Universal Ctags symbol audits, 99.5 ms warm search, 50-file uncommitted edit sync, and active CLAUDE.md / AGENTS.md rule attachment.

Operating-Modes Summary Scorecard — Active-Control Day-0 Cold Benchmarks (360 Upstream PRs) & Internal Warm Workflows Measured on Apple M1 Max (32 GB Unified Memory, 400 GB/s). Row 1 leads with our pre-registered Day-0 cold benchmark across 180 real merged upstream GitHub bug-fix PRs against ripgrep active control and sense 1.14.2 embedding control (0 s setup wait); Row 2 reports the warm dev-set check; Rows 3–4 add post-Sept-2026 held-out PRs (Set 3 + Set 5 across 18 new repos) and 5-language Universal Ctags symbol audits; Rows 5–8 report warm-indexed and zero-GPU workflows.

Operating Mode	Developer Workflow & Measured Metric	localctx PrimaryCompact E4B (4.24 GB) or 0 MB GPU	localctx SecondaryZero-GPU search, 26B-A4B, or p50	Active / External Controlripgrep, Universal Ctags, sense, or cloud	Report
Active-Control Day-0	Cold Day-0 bug search vs. `ripgrep` active control (`180` upstream GitHub PRs: `120` frozen + `60` unseen, `0 s` setup wait, `v4.7`)↑ higher is better · right file on first try	46.7% (ask E4B)0 s setup · 881 tok/q (7.8× vs. rg)	44.4% (search, 0 GPU)0.42 s p50 · 67.0% top-5 multi-file	ripgrep: 33.9% (6,876 tok/q)0.07 s p50 · sense 1.14.2: 38.3% (477.2 s index; 42.2% vague bugs)	v4.6 held-out →
Warm Dev-Set	Internal warm-cache Q&A check (`basecamp/fizzy`, 1,017 files, 43 self-authored dev-set tasks)↑ higher is better · file & line citation accuracy	42 / 43 (97.7%)E4B warm index (self-authored)	43 / 43 (100.0%)26B-A4B warm index (self-authored)	Cloud grep baseline5–8 full-file grep rounds	v4.6 report →
Held-Out 18 Repos	Post-Sept-2026 unseen PR localization (Set 3 `n = 90` + Set 5 `n = 90` across `18` new repos, `0 s` setup, `v4.9`)↑ higher is better · first-try & top-3 file match	53.3% Set 3 (ask E4B)37.8% Set 3 search · 0 s setup	38.9% Set 5 (search, 0 GPU)65.6% Set 5 top-3 · 99.5 ms warm p50	ripgrep: 23.3% (Set 3) · 28.9% (Set 5)53.3% Set 5 top-3 · wins mastodon 40% & rubocop 30% vs. search	v4.9 report →
Zero-GPU Operation	5-language Universal Ctags symbol audit & warm search (Rust, Go, Python, TS, Ruby · `v4.9`)↑ higher is better · independent symbol recall & precision	300 / 300 defs (100.0%)99.6% precision (249 / 250 verified)	200 / 200 refs (100.0%)65.5 ms warm CLI flag query p50	Universal Ctags ground truth500 independent targets across 10 repos	v4.9 report →
Warm Indexed	Claude tokens saved on real coding tasks (`20/20` live agent tasks)↓ lower is better · cloud input tokens	−76.7% fewer tokens113,173 billed cloud input tok	2.83× faster (p50)6.02 s median task completion	484,681 tok · 17.01 s p50Cloud grep-and-read baseline	v2.0 report →
Warm Indexed	Warm local answer speed (`16/16` cross-file probes)↓ lower is better · p50 response time	1.39 s< 2.0 s across 16 / 16 probes	1,612.9 ms meanRuby, Rust, Go & TypeScript	4,855.1 msCold CLI process startup	v3.0 report →
Zero-GPU Operation	Keeping `CLAUDE.md` & `AGENTS.md` rules active at Turn 30 (`Gate M-CONVENTION`)↑ higher is better · rule adherence	22 / 22 (100.0%)0.84 ms warm p50 · 0 MB GPU	0 / 8 false matches77.9 mean tokens · 5/5 broken paths caught	5 / 22 (22.7%)Unpinned Turn-30 cloud session	v4.6 rules →
Warm Indexed	Speed with 50 uncommitted local file edits (`0` Python)↓ lower is better · warm answer time	E4B: not measured on Gate 7128 / 128 exact output match vs. reference	4.30 s p50 (26B-A4B)16.28× faster · 0.00000 output diff	69.95 sv4.5 full-partition rebuild	v4.6 report →

Release Progression Matrix — 14 measured releases across 16 standalone benchmark reports (April 10 – November 4, 2026) All measurements captured on Apple M1 Max (32 GB Unified Memory, 400 GB/s). Terracotta cells mark passing release milestones; warm-stone cells mark falsified or retracted negative-control milestones (v0.0–v0.2 and v4.4 cold vague-bug test).

Release	Date	Primary Architectural Hypothesis Tested	Headline Measured OutcomePrimary gate or falsification	Report
v4.9	Nov 4, 2026	18-repo post-Sept-2026 held-out PR search (Set 3 + Set 5), 5-language Universal Ctags symbol audit, Agent-Audit 50 & warm search cache↑ higher is better · unseen-repo file match & symbol recall · 300/300 defs & 200/200 refs vs. Ctags (99.6% precision); 99.5 ms warm Set 5 search p50 (0 divergences); all per-repo losses disclosed	53.3% Set 3 (ask) · 65.6% Set 5 top-3 (0 GPU) · 100% Ctags recall	v4.9 report →
v4.6	Oct 12–31, 2026	Zero-Python native Apple Silicon runtime, speed with 50 uncommitted file edits, 120-PR cold bug search & active project rules↑ higher is better · full-repo Q&A & rule adherence · 128/128 exact output match (0.00000 diff); 20 sub-65 ms zero-GPU code/Rails tools; 22/22 CLAUDE.md/AGENTS.md rule match	43/43 (26B) · 4.30 s 26B 50-edit sync (16.28×)	v4.6 report →
v4.5	Sep 25, 2026	12-language compiled syntax parser, cross-file symbol graph & full-repo (7.85 GiB) warm 26B-A4B index↑ higher is better · multi-file & 10-language accuracy · Replaces regex cross-file matching; exposes 69.95 s 50-uncommitted-edit rebuild delay fixed in v4.6	20/20 multi-file · 21/21 10-repo · 42/43 (97.7%) 26B	v4.5 report →
v4.4	Sep 10, 2026	Zero-setup Day-0 onboarding across 10 languages & warm E4B vs. unindexed cold 26B-A4B test↑ higher is better · vague bug report accuracy · 16/16 Rails view→JS→Ruby on cold 26B, but vague bug reports (no symbol names) drop 80% → 30% without a warm index	20/21 Day-0 (95.2%) · Cold 26B vague bugs: 30% (warm E4B 80%)	v4.4 report →
v4.3	Aug 26, 2026	Indexing all 1,017 basecamp/fizzy files (567,912 tok), 4-agent parallel load & 50-query battery soak↓ lower is better · thermal speed drift · 9/10 exact ±3 line hits in 60.6K slices; regex 3-file hops stall at 7/20 (35.0%) before 12-language parser	+0.1% thermal drift · 90% deep line hit · 35% regex hop limit	v4.3 report →
v4.2	Aug 11, 2026	Scanning 32 repo partitions in 17.75 ms, top-1..5 slice routing & compact E4B vs. large 26B-A4B↑ higher is better · multi-file Q&A accuracy · 6 deterministic line-citation checks across 1.81M tokens; 87.5% overall / 85.7% multi-file Fizzy on 26B	44.4% → 88.9% multi-file accuracy (2.0×)	v4.2 report →
v4.0	Jul 24, 2026	Splitting large repos into 64K disk partitions vs. a single giant 256K buffer across 1.81M tokens↓ lower is better · warm answer time · 3.29× faster index build (572.2 tok/s); 25.52 ms zero-copy native disk attach; 4.2 GB active GPU RAM	2.04 s vs. 11.02 s warm Q&A (5.41× faster)	v4.0 report →
v3.0	Jul 9, 2026	Always-warm Unix-socket background daemon, 18 µs RAM snapshot reset & 15-step Metal speed tuning↓ lower is better · p50 & mean response time · 27.7× fewer tokens sent to Claude across Ruby, Rust, Go, and TypeScript repos (16/16 under 2.0 s)	1.39 s p50 · 1,612.9 ms mean (2.68× faster than cold CLI)	v3.0 report →
v2.0	Jun 24, 2026	Paired 20-task live Claude coding loop on basecamp/fizzy, 25.5 ms disk reload & low-RAM 4-bit indexing↓ lower is better · cloud input tokens & task time · −38.5% indexing GPU RAM (0.000000 output diff); 113,173 vs. 484,681 billed cloud tokens at 20/20 accuracy	−76.7% Claude tokens · 2.83× faster task p50 (6.02 s)	v2.0 report →
v1.1	Jun 9, 2026	150-question multi-step Q&A vs. top-6 vector RAG, LongBench-v2 & 2.5 ms uncommitted git edit tracking↑ higher is better · multi-step & live-edit accuracy · 30/30 uncommitted git edits (0% stale); 49/50 basecamp/fizzy localization (43.4× fewer tokens)	+14.0 pts vs. vector RAG · 30/30 git edits · −92.0% tokens	v1.1 report →
v1.0	May 26, 2026	High-precision 4-bit Apple Metal context engine (8K–256K) & persistent disk caches across multi-size Gemma 4 checkpoints↑ higher is better · 256K generation tok/s · 4×–64× lower numerical drift vs. stock MLX; 16/16 parity on E4B & 26B-A4B configs; 654.1 ms 256K disk reload	3.16× faster 256K speed (17.42 tok/s) · 16/16 model parity	v1.0 report →
v0.2	May 11, 2026	Dropping older 4-bit/2-bit context blocks across 2K–128K files (S1) and 15 multi-file prompts (S2)↑ higher is better · long-file fact recall · May 2026 audit caught chunk-eviction bug starving 768/1,024 positions (Defects F1–F3); v0.2 retracted	RETRACTED (Erratum: Defects F1–F3 · 2-bit 1/7 recall)	v0.2 erratum →
v0.1	Apr 24, 2026	Compressed GPU span scoring vs. zero-GPU keyword search on live Gemma 4 26B-A4B code questions↑ higher is better · right span on first try · 5/5 synthetic vector tests fell to 3/5 on real prompts from common-word noise; CPU keyword control won	FALSIFIED: zero-GPU keyword control won (20/20 vs. 19/20)	v0.1 report →
v0.0	Apr 10, 2026	Constant-memory lossy rolling GPU buffer on planted-fact lookup across 64 to 256 steps↑ higher is better · planted-fact recall · GPU safety checks passed (9/9), but older tokens faded within 3 updates; exact window control won	FALSIFIED: exact cache won (1.000 vs. 0.0148 lossy buffer)	v0.0 report →

18-Repo Held-Out Search & Symbol Audit

November 4, 2026

localctx v4.9 · 8 min read

Apple M1 Max · 32 GB

18-Repo Held-Out PR Search, 5-Language Universal Ctags Symbol Audit & 99.5 ms Warm Search

Post-Sept-2026 held-out evaluation across 18 unseen open-source repositories (Set 3 n = 90 + Set 5 n = 90, 0 s Day-0 setup wait): 53.3% first-try file match on localctx ask and 37.8% on localctx search vs. 23.3% ripgrep on Set 3; 38.9% search vs. 28.9% ripgrep (65.6% vs. 53.3% top-3) on Set 5; 300/300 symbol definitions and 200/200 references recalled against Universal Ctags across Rust, Go, Python, TypeScript, and Ruby (99.6% precision); 99.5 ms warm search median (65.5 ms CLI flags); and full per-repository loss disclosures.

53.3% Set 3 first-try file match (vs. 23.3% rg) 65.6% Set 5 top-3 file match (0 GPU, vs. 53.3% rg) 300 / 300 5-language symbol def recall (99.6% precision) 99.5 ms Warm Set 5 search median (65.5 ms CLI flags)