Market Brain Goes Native

Move the market agent's brain off Turso and into its own Durable Object's SQLite — the way Flue and Cloudflare intend. Decision + build spec · 2026-06-25

In one breath: The market agent is a Durable Object, and a DO reads its own SQLite locally — microseconds, no network hop. Turso only ever earned its place by offering in-database vector search; the moment we moved embeddings on-the-fly (tiny live vector set), that advantage evaporated and all that's left of Turso is a per-query network hop to an external vendor, against the framework grain. So the brain comes home: the 3.1M-fact relational spine moves into the DO's native ctx.storage.sql. The lone thing that ever made this unsafe — the team renaming the DO (-v8→-v9) to escape poisoned sessions, which orphans the data — is fixed by wiring Flue's surgical session-clear (harness.sessions.delete()) to an admin route, so we keep the same DO id forever. Vectors live in Cloudflare Vectorize — a native-CF binding, not the DO (SQLite can hold a blob but can't search it); only the relational facts go in the DO. The 10 GB cap is deferred (13× headroom, tier to R2 when a hot market nears it).

1 · Why move — what changed

Turso was adopted for the market brain for exactly one reason that no longer holds, plus one that was never really a benefit:

Strip those and what remains is pure cost: a network round-trip on every query (and the agent reads its brain constantly), a second vendor to provision/secret/rotate across 20 markets, and code that fights Flue's grain (the framework's native storage is the DO's own SQLite; the market agent bypasses it to reach Turso over HTTP). Going native deletes all three.

2 · The durability model — the foundation

The whole decision rests on one fact that's worth stating with zero ambiguity, because it was muddied earlier by sloppy "the DO gets thrown away" language.

The proof is Flue itself: Flue persists its sessions in DO SQLite (flue_sessions, flue_session_entries, …). If DOs were throwaway, you'd lose conversation history every time the instance slept — Flue's entire session design would be impossible. It works because DO storage is durable. Flue betting its sessions on it is the evidence.

So what was losing the brain? Not the platform — us. DO data only disappears three ways: you call deleteAll(), you delete the class, or you address a different id. The -v8→-v9 move was the third — not deleting anything, just pointing the agent at a new, empty object (the old one's data is still sitting there). We did it because a poisoned session had no clean reset, and renaming was the only hammer we had. The native design removes the cost of that hammer: the brain lives on a stable id (so it never moves) and is backed up to R2 (so even a deliberate recreate costs a few-minute re-ingest, not data loss). Sessions are disposable; the brain is durable and re-loadable. That's the whole game.

3 · The recovery model — disposable sessions, a re-loadable brain Pete, 2026-06-25

An earlier draft chased a scalpel: a surgical "clear just the poisoned session, keep the object" route. We built it and hit a real Flue beta.5 wall — an action runs in an isolated child harness with its own scoped sessions, so harness.sessions.delete('default') from inside the agent clears the action's session, not the live poisoned one, and beta.5 exposes no way to run a turn in a named session to reach it. The clean in-place clear isn't reachable today.

It doesn't matter — and chasing it was the wrong frame. A poisoned session is just chat history; there's nothing in it worth preserving. The thing worth protecting is the brain, and the native design protects it two ways:

So the recovery path for a poisoned session is simply: recreate the agent (green-field sessions), re-load the brain from R2. Disposable sessions, durable brain. No scalpel required.

4 · What we keep, lose, and defer

5 · Target architecture — the full storage topology

One durable storage, two interfaces, by data shape. A Durable Object has no POSIX disk — its SQLite is the disk. Over that one storage Flue gives you both a SQL interface (tables/rows — for the millions of structured facts you query and join) and a file interface (@cloudflare/shell's Workspace / WorkspaceFileSystem, already wired here as getDefaultWorkspace() — real paths, read / write / edit / grep, for the markdown the agent reads and cross-links every beat). The wiki and the soul / agent / journal files want the file interface; the parcels and facts want SQL. Same disk, each fit to its data — not "files shoved into a table," a genuine file API on the only durable local medium a DO has.

Outside the DO, three things in two roles. R2 and Vectorize are the brain's outboard storage — they exist only because a DO can't hold blobs-at-scale or search vectors itself. Convex is a different animal: the app surface, where the brain publishes what the website shows; it is never a path to the brain. And when one agent needs another's brain, it touches none of the three — it calls the other DO directly over RPC, reading that DO's storage without waking its model.

The SQL port is mechanical: every data fn in store.ts / parcels.ts already takes an injected client, so the swap is one-for-one — libSQL's client.execute({sql,args}) becomes the native ctx.storage.sql.exec(sql, ...args).toArray() directly. No impersonating adapter: Flue's native handle is the API, and using it as-is is the point. The only logic that changes is the vector path — libSQL vector_top_k → a Vectorize binding query (returns parcel_ids → join to local facts) — and the parcel_embeddings BLOB column leaves the schema entirely.

6 · The build — phased

1. Sessions are disposable
   No surgical session-clear (beta.5 doesn't expose it — see §3). Recovery from a poisoned session = recreate the agent + re-ingest the brain from R2 (Phase C). The Phase-A admin_clear_session action + route were removed (commit 7c83e51) — superseded, non-functional for the live session, and breaking flue build.
2. Keep poison rare
   Lean on the shipped guards: typed tool outputs (no parse-loop) + the dream beat (JOURNAL.md rewrite that breaks stuck loops). Prevention over surgery.

1. Native handle + fakeSqlStorage() test helper
   A tiny Sql type — { exec(q, ...binds): { toArray(): Row[] } }, matching Flue's SqlStorage — and a synchronous-sqlite fake for tests. No libSQL, no impersonating adapter.
2. Port store.ts + parcels.ts to the native handle
   Mechanical: each data fn's signature (c: Client, …) → (sql: Sql, …); body await c.execute({sql,args}) → sql.exec(query, ...args).toArray(). SQL text and source column names preserved verbatim. Existing tests re-point to fakeSqlStorage(); green = behavior-preserving.
3. Vector path → Vectorize binding; drop the BLOB column
   Replace the libSQL vector_top_k call with a Vectorize query() (returns parcel_ids + scores → join to local facts). Remove parcel_embeddings from the DO schema. Stand up the empty index + [[vectorize]] binding now; writes land with the on-the-fly embed path (Phase-2 comps). Same ParcelMatch return shape.
4. Market agent + its tools read the native handle
   Inside the DO, tools resolve the handle via getCloudflareContext().storage.sql (Flue's ambient accessor) instead of getTursoClient(). Convex mirror writes untouched.

1. Archive the cleaned dump to R2
   Push the 764 MB cleaned hz-import.db to R2 — the source of truth for native ingest (and any future re-ingest). Do this before touching Turso.
2. Resumable ingest route
   Auth'd POST /admin/ingest streams rows from the R2 dump and batch-inserts into ctx.storage.sql (INSERT OR IGNORE, key-paginated, resumable across alarm beats — there is no DO bulk file-import).
3. Verify counts
   Assert exact match: parcels 18,970 · parcel_facts 3,114,247 · parcel_listings 482 · pending_resolution 387. Spot-check parcel_brief (zpid 103871512 → parcel 292329201006).

1. Expose read RPCs on the market DO
   Market DO methods for the cross-agent reads: parcelBrief, listListings, scout reads. These run the store fns with the DO's own adapter.
2. Point advisor-scout & staff tools at the RPCs
   Advisor scout stops reading Turso via shared env; staff parcel_brief calls the market DO. Validate: parcel_brief in Slack, advisor scout's nightly read.

1. Remove the libSQL client + Turso refs
   Market path reads only the native handle. Grep-clean of @libsql/client / TURSO_* in the market path.
2. Drop the secrets last
   After native is proven in practice, remove TURSO_DATABASE_URL/TURSO_AUTH_TOKEN from the Worker. Full suite green; deploy.

1. Stand up the Workspace file layer
   Wire getDefaultWorkspace() (@cloudflare/shell Workspace over ctx.storage.sql) into the market agent so it has real file I/O — read / write / edit / grep over paths — for wiki/*.md, SOUL.md, AGENT.md, JOURNAL.md. Seed the identity files; prove a read/write round-trips and survives a respawn.
2. Foundation for the deferred wiki/dream beats
   This phase only stands up the home. The rich wiki authoring (interlinked briefs, the comps engine) and the dream beat's JOURNAL.md rewrite are existing deferred work that now has a native place to live. Not required for the Turso cutover — buildable any time.

7 · Evidence — the two investigations 2026-06-25, code-verified

Cloudflare limits (confirmed against docs)

• DO SQLite: 10 GB/object (Paid); only FTS5/JSON/Math extensions (no sqlite-vec — so vectors go to Vectorize, not the DO); no bulk file-import (batch-insert via SQL API); soft ~1,000 writes/s/object.
• D1 (the considered-and-passed middle path): 10 GB, has wrangler d1 import, but a binding hop (not local) and still no native vector — only wins if we needed shared access without re-plumbing.
• Vectorize: native-CF Worker binding (not an external vendor), 10M vectors/index, ≤1,536 dims (we use 768), eventually consistent (~seconds — fine for async on-the-fly embedding). The committed home for all market vectors.

Flue internals (confirmed in @flue/runtime)

• Session state lives in DO SQLite: flue_sessions, flue_session_entries, flue_agent_submissions, flue_agent_turn_journals, flue_image_chunks, flue_meta.
• harness.sessions.delete() exists but in beta.5 is only reachable from inside an action, whose child harness scopes it to the action's own session — it can't reach the live poisoned default (no API to run a turn in a named session). So in-place session-clear isn't available; recovery is recreate + R2 re-ingest instead.
• Code deploy (same class + id, no new migration) preserves storage. Only a class rename / delete_sqlite_classes / new id drops it.
• Version brick = schema-version mismatch thrown at boot (recoverable by clearing sessions), not storage corruption.

8 · Risks & decisions

Companion to Market Data Migration (Phase 0, shipped). This spec is the native-storage revision: the relational brain moves from Turso into the market agent's own Durable Object. Build via subagent-driven development, Opus on every delegated agent, branch → PR → squash-merge. · 2026-06-25