The engine — a shared on-demand instance per bundle

How Outcrop's engine runs, who starts it, how clients find it, and when it goes away. The pattern is a shared on-demand engine instance per bundle — prior art: SQL Server LocalDB's "automatic instance", the Gradle daemon, and language servers. Not a daemon (nothing to install or administer), and deliberately not the sidecar pattern (a sidecar serves one client and shares its lifecycle; this engine is shared and owns its own).

User vocabulary stays what it always was: the engine. It starts when you need it, is shared when that helps, goes away when you're done, and outcrop engine status shows it to you.

Why

Every client either cold-boots the whole engine per invocation (CLI, MCP, LSP) or depends on a manually managed TCP server (canvas, port 7411) — the latter produced a real field failure: a zombie server squatting the port. A prosumer should never administer a database process. The engine must boot seamlessly with any client, be shared where that buys coherence, and remove the zombie class by construction.

Three-layer coordination

Trust kernel-attached properties, not file contents.

1. Liveness/mutex: flock on .outcrop/engine.lock. The engine holds an exclusive advisory lock for its whole life. The kernel releases it atomically with process death — crash, SIGKILL, anything. Lock held ⇒ engine alive; lock acquirable ⇒ engine dead. No pid checks, no heartbeats, no stale states. This is the only authority.
2. Discovery: .outcrop/engine.json, hints only. Carries the socket path (authoritative — sockets may relocate, see below), plus pid and version for display/diagnostics. Never trusted for liveness; garbage or torn contents are harmless because of layer 3.
3. Truth test: connect and say Hello. A client adopts an engine only after a successful Hello RPC over the socket returning engine version and canonical bundle root. This is the only check that verifies what clients actually need: a responsive, compatible engine on this bundle.

Transport

• Unix domain socket, default .outcrop/engine.sock, owner-only permissions. When the UDS path-length limit (~104 bytes) bites, the socket relocates to $TMPDIR/outcrop-<root-hash>.sock and the engine.json socket field is how clients find it.
• One socket serves both planes: outcrop.v1 (data, the frozen contract) and outcrop.engine.v1 (control: Hello, Attach, Shutdown, Status) — so the Hello truth test exercises the same connection data will use.
• The control plane lives in proto/engine.proto — a separate file so proto/outcrop.proto stays frozen.
• TCP (outcrop serve --grpc) remains an explicit opt-in and is the local Windows path; the engine module is unix-only for now (revisit trigger: Windows named-pipe transport). It is not a remote surface: no engine ever serves another machine's client (decisions/clients-host-their-own-engine) — cross-device consistency is git sync against a remote, whose TLS and auth do the transport-security job a served engine would have needed.

Lifetime

• Lease = an open Attach stream. Every persistent client (canvas, MCP session) opens a long-lived server-stream after adopting. The kernel closes the stream when the client dies — however it dies — so the engine's session table is always true. One-shot CLI calls never lease.
• Linger: the engine exits after max(last lease dropped, last RPC served) + linger (default 45s, OUTCROP_ENGINE_LINGER_MS to tune; 0 = exit immediately when idle). An absolute idle cap bounds the degenerate cases. Rapid relaunches and CLI bursts adopt a warm engine instead of rebuilding indexes.
• Other exits: watcher root gone (bundle moved/unmounted) → exit now; Shutdown RPC (outcrop engine stop) → graceful exit. stop falls back to SIGTERM on the hinted pid if the socket is unresponsive, and verifies death by acquiring the flock — unforgeable proof.
• An idle engine holds nothing: no git index lock, no open transactions (staged state is durable on disk by design), no unswept durable state. Merge proposals are durable under .outcrop/proposals/ (see below), so linger-exit loses only warm caches.

The adoption dance (cross-language contract)

Implemented by Rust (outcrop-server::engine::rendezvous) and re-implemented thinly by Swift (EngineLauncher). Keep both exactly to these steps:

1. Read .outcrop/engine.json. Absent or unparseable → no hint; never an error.
2. Connect (hinted socket path, else the default) with a ~500 ms deadline; call Hello(client_kind, client_name, protocol_version).
3. Adopt iff Hello succeeds and its bundle_root canonical-equals ours and versions are compatible. Anything else = adoption failed (no distinction needed).
4. Spawn-permitted callers only: spawn outcrop engine run --root <root> detached, then re-poll steps 2–3 with backoff up to ~5 s. There is no client-side lock — race safety is the engine's flock. Losing spawners exit 0 after confirming another engine answers Hello; every poller converges on the winner.
5. Version mismatch → do not spawn (the socket is held by a live, wrong-version engine). Fall back to solo mode with a warning that names outcrop engine stop.
6. Persistent clients open Attach immediately (the lease).

Who adopts what

Client	Policy	Why
Canvas	spawn-or-adopt	It has no other way to exist; holds a lease
MCP	adopt-or-spawn	Shared state unlocks cross-surface propose/resolve; holds a lease
CLI	adopt-if-present only	Speed when warm, hermetic purity when not; never spawns; --no-engine / OUTCROP_NO_ENGINE forces solo
LSP	standalone (non-goal)	Tight latency budget; sharing a runtime with heavy queries risks editor stutter; its working set is small. Revisit when graph features land in editors

Degradation: solo mode

If flock errors (network filesystems — Dropbox, iCloud Drive, NFS — where advisory locks are unreliable), Outcrop degrades to solo mode: every client runs its own in-process engine, exactly today's behavior. This is safe, not merely tolerable, because the files are the database — commits serialize through git, writes are atomic, external edits are first-class. Sharing is a performance state, never a correctness state. The same fallback applies to any adoption failure in the CLI (silently) and to version mismatch everywhere (with a warning).

The engine as a library (in-client engines)

Decided: every client hosts its own engine (bundle/decisions/clients-host-their-own-engine). The shared on-demand instance above is the desktop realization of that decision. iOS and Android link the engine in-process — iOS cannot spawn processes, so the rendezvous/spawn machinery cannot exist there at all. That obliges the engine module to ship two facades over one core:

• Binary — outcrop engine run, everything above: flock, socket, linger, adoption.
• Library — a lib target exposing both planes (outcrop.v1 + outcrop.engine.v1) over an in-process channel, so a mobile client's generated gRPC code is identical to desktop's; only the channel construction differs. Nothing in the engine's API may assume flock/spawn/linger — those are properties of the shared deployment, not of the engine.

In-process hosting is solo mode by design rather than by degradation: one client, one engine, and the same correctness story (the files are the database; commits serialize through git). The feasibility facts are verified: the transaction layer drives git through libgit2 (git2 crate, no subprocess anywhere), and DataFusion + tantivy are pure Rust — the stack cross-compiles. Unmeasured until the first device build: stripped in-process binary size.

Durable merge proposals

Proposals move from server memory to .outcrop/proposals/<id>.json (atomic temp→fsync→rename writes), so they survive engine restarts and can be resolved by a different client than the one that proposed — the human-reviews-agent-work loop. They persist as JSON machinery under .outcrop/, not as records: a proposal carries machine merge plans and git object ids, which are engine state, not knowledge. (This is the deliberate answer to the "why isn't this a record?" challenge in the ethos: it fails the test — it is machinery.) Stale proposals (referencing commits that no longer resolve) report cleanly: "re-propose".

Observability

outcrop engine status shows: engine version, bundle root, uptime, total RPCs served, and each attached client (kind, name, pid, attached-at). When something feels stuck the diagnostic is one command, not lsof archaeology. outcrop engine start|stop|run complete the verb set.

Known limitations & revisit triggers

• Windows: no UDS engine; TCP serve remains the path. Trigger: Windows client gets a UI.
• macOS App Sandbox: sandboxed builds can't reach arbitrary UDS paths; dev builds are unsandboxed. Trigger: App Store distribution.
• LSP adoption: deliberately out. Trigger: editor features that need the shared link graph; adopt read-only if so.
• Fairness: per-connection limits are deferred until agent load is real. Trigger: first observed interactive-latency degradation under agent traffic.
• Library facade: doesn't exist yet — the engine module is socket-only today, and the in-process channel plus the no-spawn/flock/linger API boundary are unbuilt. Trigger: the first iOS/Android client build (cross-platform client MVP).