Benchmarking pi-l1-cache: a 138µs in-memory cache layer for the pi coding agent
The pi coding agent lets extensions intercept the provider round-trip with before_provider_request and after_provider_response events. That is plenty of room for a fast in-memory cache — but only if the interception itself is cheaper than the round-trip it saves. pi-l1-cache (v1.2.0, installable as npm:pi-l1-cache) is a deliberately minimal take on this: one file, zero dependencies, an FNV-1a hash, a hard memory cap, TTLs and a 95%-CPU kill switch. We installed it from npm, loaded it against a real pi session, and benchmarked the published artifact rather than the README.
The headline number is honest and useful: every intercepted request costs about 138µs — but only if the request is byte-identical to a cached one does the provider round-trip (1–3s) get skipped entirely. That is the whole value proposition, and it comes with sharp edges worth knowing before you adopt it.
How it slots into pi
The cache sits between the agent loop and the provider:
pi → [L1: RAM Map] → [L2: Redis via LiteLLM] → Provider
~138µs ~150ms 1-3s
On before_provider_request the extension hashes model.id + JSON.stringify(messages) + JSON.stringify(parameters), looks up a Map, and returns a cached response object if the entry is fresh. On after_provider_response it stores the response (with a JSON-length-based size estimate) and runs eviction. A 10-minute setInterval sweeps expired entries, cleaned up on session_shutdown.
Key derivation is content-addressed: model, messages and parameters all feed the hash, so the cache only ever answers requests that are literally identical to one it has seen. It never guesses or generalises.
Configuration is environment-based rather than config-file based:
L1_CACHE_ENABLED=false # hard off
L1_CACHE_MAX_ENTRIES=500 # entry cap (default 200)
L1_CACHE_MAX_MB=50 # memory cap (default 20MB)
L1_CACHE_TTL=7200 # TTL seconds (default 3600)
Runtime control comes from a single slash command:
/l1-cache stats # entries, MB, hit rate, init CPU
/l1-cache clear # reset counters + map
/l1-cache enable # / disable
What we measured
We benchmarked the actual npm artifact (pi-l1-cache@1.2.0) under Node 22.23 using the package's own exported test hooks, then exercised the real interceptor path with a mocked ExtensionAPI exactly as pi boots extensions.
| Measurement | Result |
|---|---|
fastHash standalone (FNV-1a) | 670k ops/s — 1.49µs/op |
Node native sha256 (OpenSSL) | 850k ops/s — 1.18µs/op |
| Full put: hash + map set + size check | 249k ops/s — 4.0µs/op |
| Interceptor MISS path (hash of real 15-message history) | 138µs/request |
| Interceptor HIT path (identical lookup) | 138µs/request |
| Cache hit vs uncached provider round-trip | ~7,000–20,000× wall-clock |
| Uniqueness over 50k synthetic keys | 50,000/50,000 (no collisions) |
| Eviction under 200-slot cap, 20k inserts | cap respected; 19,800 evicted |
Two findings deserve scrutiny.
The "100× faster than SHA-256" claim does not survive Node
The README claims FNV-1a is "~100× faster than SHA256". In Node it is the opposite: crypto.createHash('sha256') is a native OpenSSL call and wins outright (0.8×). Our loop recreated a hash per iteration, which is the realistic per-request pattern, and native SHA-256 still beat a per-byte JavaScript loop. The claim has since been corrected, but the episode is a good reminder to benchmark published extensions instead of trusting their billboards.
Does it matter? No — because the hash is not the bottleneck. Deriving the key is dominated by JSON.stringify(messages), which the extension runs on every request, hit or miss. Our 15-message history sample put both paths at ~138µs, of which the actual FNV loop is microseconds. Whether the hash is FNV-1a or SHA-256 is irrelevant at these message volumes; JSON.stringify is the fixed tax, and 138µs is still three orders of magnitude below the slowest cacheable round-trip.
Hit rate is the real design constraint
Because the key is model + messages + parameters verbatim, the extension is a deduplication cache, not a semantic one:
- Re-running the same prompt, same history, same temperature → hit.
- Any token change, any
max_tokensbump, any non-zerotemperaturesample → miss. - Long sessions where every turn appends new history produce near-zero hits, because the whole conversation is part of the key.
This makes its practical hit rate highly workload-dependent. It shines for deterministic, idempotent calls: repeated tool invocations, retries after provider 429s (a genuinely common case — our test bench hit a monthly quota error repeatedly, the precise scenario this cache short-circuits), same-day debugging of the same failing test. It will do almost nothing for free-form creative generation.
Eviction and memory accounting
Eviction is correct and cheap to reason about: oldest-first under both an entry cap (200) and a byte cap (20MB), evicting in 10–20% batches. We hammered 20,000 inserts through the 200-slot cache and the size never exceeded the cap; 19,800 entries were evicted. Hash distribution over 50k sample keys produced zero collisions in the 32-bit space, and fastHash is collision-safe enough for cache identity in practice.
Memory tracking is an estimate, not a measurement: JSON.stringify(response).length × 2. That is fine for a hard cap, but note two things. First, the stats command formats megabytes to one decimal, so a cache holding <50KB displays as 0.0MB — cosmetic, not functional. Second, the estimate never accounts for the Map's own key overhead or the FNV hash of the key itself, so real RSS will always exceed the reported totalMemory. The 20MB default is a soft upper bound on stored payloads, not process memory.
The CPU guard deserves credit for restraint: it checks /proc/loadavg once at startup and disables the cache if load exceeds 95%, rather than sampling on the hot path. A per-request execSync CPU probe would have destroyed the 138µs budget the cache exists to protect; the author made the right call, and a middle ground (a slow periodic probe, say every 30s) would preserve the safety valve with negligible cost.
Security and operational caveats
- Staleness is by design, not accident. Cached responses are frozen for the TTL. If your provider prompt or tool outputs are time-sensitive, disable the cache for that context or drop the TTL.
- Process-local by construction. The L1 map dies with the session — that is what L2 (Redis, LiteLLM) is for. Restarting pi is the simplest cache flush, which is also the documented behaviour after install.
- Multiplicative latency on miss. Since key derivation (138µs) runs before the provider call, uncached requests carry a small fixed overhead that never goes to zero. At ~7k requests/s throughput this is immaterial; in a proxy path serving thousands of distinct brute-force requests it would not be.
- The package ships zero runtime dependencies and lists pi-core packages in
peerDependencies— worth checking before you install anything claiming to extend pi, since extensions execute arbitrary code by design.
Verdict
pi-l1-cache is a well-scoped, correctly-behaved extension with honest (if overstated) performance goals. The 138µs interception cost is invisible next to a 1–3s provider call, eviction respects its caps under stress, and the single-file, zero-dependency design is exactly the right shape for a cache. Its ceiling is hit rate: it only pays off for deterministic, repeated workloads — retry storms included — and it is transparent about that via /l1-cache stats.
If you run pi and bump into provider quotas or re-run the same prompts, this is a legitimate two-line win:
pi install npm:pi-l1-cache
/l1-cache stats # after restart
For anything else, the lesson generalises beyond this one package: benchmark the extension, not its README, and let your hit-rate determine whether the layer earns its place in the hot path.