System Architecture (Canonical Topology)

This skill is the single source of truth for joelclaw system wiring.

Freshness notes:

• 2026-08-23 recall cutover: memory_observations and the system-log JSONL/slog path are retired. Production joelclaw recall composes lane-separated flowing reflections, flowing observations, and curated Brain pages. Every request has explicit scope and access. Chorus/Rhizomatic is parked with no briefing injection or live claims; service stop still needs steering sudo. Claude auto-memory is a pointer index, not a content store.

• 2026-07-10 topology: the old Panda-centric mental model is migration debt. Current responsibility lanes:

• Central: Flagg/Mac Studio. It is authoritative for agent-mail and Run capture ingress; verify each remaining service family during migration.

• Relay: Panda account-bound leftovers and explicit decommission blockers.

• Satellite: Blaine/other capture clients, connectors, outboxes, and transcript backup freshness.

Treat older “Panda is the whole system” wording as stale unless re-verified against live receipts.

Use it for:

• "why did this run / not run"
• "which worker handles this function"
• "what is listening on port X"
• "how does event Y flow"
• "where does this Run/capture/memory record go"
• "is this Central, Relay, satellite, or shadow runtime work"
• full-stack routing/debug across CLI → Inngest → workers → gateway → telemetry

Ground-Truth Scope + Evidence Snapshot

This document is grounded in direct reads of:

• apps/docs-api/src/index.ts
• packages/restate/Dockerfile
• packages/restate/src/index.ts
• packages/restate/src/workflows/dag-orchestrator.ts
• packages/agent-execution/src/microvm.ts
• packages/system-bus/src/serve.ts
• packages/system-bus/src/inngest/functions/index.host.ts
• packages/system-bus/src/inngest/functions/index.cluster.ts
• packages/system-bus/src/inngest/client.ts
• infra/worker-supervisor/src/main.rs
• ~/Library/LaunchAgents/com.joel*.plist
• k8s/* (all files)
• infra/pds/values.yaml
• packages/gateway/src/daemon.ts
• packages/gateway/src/channels/*.ts
• ~/.joelclaw/gateway/AGENTS.md
• ~/.joelclaw/gateway/.pi/settings.json
• ~/.local/caddy/Caddyfile
• ~/.colima/default/colima.yaml + colima status --json
• packages/cli/src/cli.ts, packages/cli/src/config.ts, packages/cli/src/inngest.ts
• packages/system-bus/src/observability/* (key files: emit.ts, otel-event.ts, store.ts)
• packages/telemetry/src/emitter.ts
• packages/system-bus/src/lib/pi-output.ts
• packages/inference-router/src/tracing.ts
• CONTEXT.md
• docs/gateway.md
• docs/inngest-functions.md
• docs/runbooks/satellite-rig-setup.md
• docs/runbooks/flagg-gate5-staged-migration.md
• infra/central/README.md
• docs/prd-rhizomatic-network-canary.md
• infra/central/launchd/*.plist.template
• scripts/joelclaw-capture-session.ts
• scripts/joelclaw-capture-codex-session.js
• ADRs in ~/Vault/docs/decisions/ (required + topology-adjacent)
• canonical OTel events plus durable Brain .svx receipts (the former system-log JSONL is archived and retired)

Related docs verified

• docs/architecture.md — Restate/Firecracker runtime + workload execution flow
• docs/deploy.md — Restate worker deploy + auth/identity/PVC procedures
• docs/cli.md — workload command tree + runtime bridge
• docs/observability.md — not inspected in this update

Refresh receipt: 2026-06-15

This refresh folds in work from:

• Central vocabulary + Project Thread docs (6b3a1b05, CONTEXT.md, docs/gateway.md)
• Flagg Central scaffold and Gate 5 migration runbooks (6e02a6cd, d36b52f2, infra/central/*)
• worker-hosted Run capture (f06501a8, docs/inngest-functions.md, packages/system-bus/src/serve.ts)
• Inngest SDK hardening + connect-mode recovery (d7dd7788, 6c5d2a8e)
• Talon paging/debounce hardening (c03edc5c, 1f086cfb, d26351cf)
• satellite rig setup for Blaine/Flagg (bc5738f3, 9cc02f6e)
• historical Rhizomatic/Chorus network canary (6bebf5b1, docs/prd-rhizomatic-network-canary.md), parked by the 2026-07-17 decision

---

0) Current Operator Map

The old mental model was "Panda is joelclaw." That is no longer precise enough.

Use these terms:

Term	Meaning	Current truth
Network	Users + Machines coordinated by one Central	Logical boundary, not the tailnet/k8s cluster
Central	single authoritative joelclaw service for the Network	Flagg is authoritative for agent-mail and Run capture; verify remaining service families individually during migration
Central host target	Machine consolidating Central responsibilities	Flagg / Mac Studio, machine_id=mac-studio-central
Relay Machine	machine that hosts account-bound/local-hardware-bound relays while delegating state to Central	Panda becomes this after cutover; satellites stay thin
Satellite Machine	thin local Pi/Codex/Claude runner with capture/search/repair hooks	Blaine and Flagg bootstrap through scripts/setup-satellite-rig.sh
Run	one captured agent invocation	raw JSONL + metadata first, SQLite FTS is the live search index
Conversation	sibling Run label for an interactive context	not the source of truth
Project Thread	private #brain-joel operator workroom for a bounded objective	coordination only; does not authorize public replies

Current authority split (verified 2026-07-10):

• Flagg is authoritative for agent-mail and Run capture ingress. The agent-mail daemon binds Flagg loopback; Blaine and Panda use SSH connector LaunchAgents so every joelclaw mail client reaches the same mailbox without exposing the service on the tailnet.
• Panda is migration debt plus Relay responsibilities. Its independent agent-mail daemon and Talon are removed. A reboot-survivable SSH connector now binds Panda IPv4 loopback 127.0.0.1:3111 and forwards legacy /api/runs and /webhooks ingress to Flagg. The legacy system worker still owns the IPv6 listener until its system LaunchDaemon is booted out with sudo.
• Satellites stay thin. They run Pi/Codex/Claude, local capture hooks, and connectors to Central. Do not install independent stateful Central services on a satellite without a specific reason.
• SQLite indexes Runs. NAS/local Run blobs are the source of truth; sessions.db is the compact live FTS index. The retired Typesense runs_dev and run_chunks_dev collections must not be recreated.
• Flagg splits book search from the operational Typesense node. The system LaunchDaemon on 127.0.0.1:8108 holds operational projections. The user LaunchAgent com.joelclaw.typesense-books on 127.0.0.1:8110 holds docs and docs_chunks_v2. DOCS_TYPESENSE_URL routes book readers and writers. A tailnet-only TCP forward exposes 8110 to Blaine.
• The book node is not a replica. A Typesense replica would copy every collection and repeat the same memory and startup cost. The separate process gives book indexing its own failure and restart boundary.

Cutover rule: avoid split-brain. Panda and Flagg must not both accept authoritative writes for the same Central service family. Gate 5 permits shadow smoke tests and migration rehearsal, but authority flips only inside an approved freeze/cutover window.

---

1) Physical Topology

Legacy Central snapshot: Panda

Mac Mini "Panda" (host macOS)
├─ launchd services (gateway, worker supervisor, caddy, talon, agent-mail, etc.)
├─ Colima VM (driver: VZ, arch: aarch64, runtime: docker, VM IP: 192.168.64.2)
│  └─ Talos node: joelclaw-controlplane-1 (k8s v1.35.0, internal IP 10.5.0.2)
│     ├─ namespace: joelclaw
│     │  ├─ inngest (StatefulSet + NodePort 8288/8289)
│     │  ├─ redis (StatefulSet + NodePort 6379)
│     │  ├─ typesense (StatefulSet + ClusterIP 8108)
│     │  ├─ restate (StatefulSet + NodePort 8080/9070/9071)
│     │  ├─ system-bus-worker (Deployment + ClusterIP 3111)
│     │  ├─ restate-worker (Deployment + ClusterIP 9080; full agent image + Firecracker)
│     │  ├─ dkron (StatefulSet + ClusterIP 8080)
│     │  ├─ docs-api (Deployment + NodePort 3838)
│     │  ├─ livekit-server (Deployment + NodePort 7880/7881)
│     │  ├─ bluesky-pds (Deployment + NodePort 3000)
│     │  └─ minio (StatefulSet + NodePort 30900/30901)
│     └─ namespace: aistor
│        ├─ aistor operator (Deployments: adminjob-operator, object-store-operator)
│        └─ aistor-s3 object store (StatefulSet + NodePort 31000/31001)
├─ Caddy reverse proxy (tailnet HTTPS fan-in)
├─ Gateway daemon (embedded pi session)
├─ Firecracker substrate (requires Colima nestedVirtualization=true for /dev/kvm; OFF by default — unstable under load)
└─ NAS "three-body" (NFS tiers per ADR-0088)

Flagg shadow / next Central target

Mac Studio "Flagg" (host macOS; target Central host)
├─ system tailscaled path required for cutover
├─ Central Service Account: joelclaw:staff
├─ service root: /Users/Shared/joelclaw/
│  ├─ services/{redis,typesense,inngest,minio}/
│  ├─ backups/central/
│  ├─ logs/central/
│  └─ src/joelclaw/ (service-owned checkout)
├─ shadow Compose stack (not authoritative)
│  ├─ Redis 7-alpine
│  ├─ Typesense 30.1
│  ├─ Inngest self-hosted
│  ├─ Restate 1.6.2 (Docker named volume for data)
│  └─ MinIO smoke surface
├─ system LaunchDaemon templates
│  ├─ com.joelclaw.central.colima
│  ├─ com.joelclaw.central.compose
│  ├─ com.joelclaw.central.health
│  └─ com.joelclaw.central.nas-mounts
├─ Chorus/Rhizomatic (parked historical canary)
│  ├─ no session briefing injection and no live claims
│  ├─ com.joelclaw.chorus-rhizomatic may remain loaded only until steering completes the sudo stop
│  └─ old 4821/7331 endpoints are historical troubleshooting context, not active dependencies
└─ NAS "three-body" proof path
   ├─ /Volumes/nas-nvme -> three-body:/volume2/data
   └─ /Volumes/three-body -> three-body:/volume1/joelclaw

Flagg Gate 4 is complete: shadow Central recovered after hard reboot with no GUI login. Gate 5 is not complete until Flagg owns Central state, workers, endpoints, and verification while Panda is frozen as rollback-only.

Known runtime endpoints

• Colima VM IP: 192.168.64.2 (colima status --json)
• Kubernetes API (stable operator tunnel): https://127.0.0.1:16443
• Talos API (stable operator tunnel): 127.0.0.1:15000
• Tailnet hostnames seen in config:
  • panda.tail7af24.ts.net (Caddy routes)
  • pds.panda.tail7af24.ts.net (PDS values)
  • flagg.tail7af24.ts.net (Mac Studio shadow / target Central host)
  • blaine.tail7af24.ts.net (satellite)
• Current live Run capture URL for satellites:
  • https://panda.tail7af24.ts.net/api/runs
  • served by Panda host system-bus worker on localhost:3111
  • do not use http://panda:3000 or http://panda.tail7af24.ts.net:3000; Panda has no durable Central web listener there.

Tailscale mesh state

• tailscale status --json failed in this environment: UNKNOWN — needs manual verification

---

2) Process Inventory (Long-Running)

Herdr launch-domain split

• com.joelclaw.herdr-server is the default interactive server. It runs as a per-user LaunchAgent in gui/<uid> so pane descendants inherit the Aqua bootstrap namespace and can reach user Keychain services.
• com.joelclaw.herdr-system-server is the boot-safe automation server. It remains a system LaunchDaemon and owns only the named system Herdr session.
• Never install the default server as a system LaunchDaemon. A UserName on a LaunchDaemon changes the Unix identity, not the launchd bootstrap namespace. Native macOS clients in those panes cannot resolve the user's com.apple.securityd.xpc service.
• The installer and cutover contract lives in infra/install-herdr-default-launchagent.sh. The durable rationale is .brain/resources/herdr-launch-domain-contract.svx.

Host launchd inventory (Panda live Central snapshot)

Snapshot source: launchctl print gui/$(id -u)/<label> and plist inspection.

Launchd label	State	PID (snapshot)	Role	Ports / endpoints
com.joel.system-bus-worker	running	75292	Host worker supervisor (worker-supervisor)	supervises child bun on 3111
com.joel.restate-worker	retired / rollback-only	—	Historical host Restate wrapper (scripts/restate/start.sh)	superseded by deployment/restate-worker on 9080
com.joel.gateway	running	81275	Gateway daemon (packages/gateway/src/daemon.ts)	WS :3018, Redis bridge
com.joel.caddy	running	9347	Reverse proxy	3443, 5443, 6443, 7443, 8290, 8443, 9443
com.joel.talon	running	96359	Infra watchdog	health 127.0.0.1:9999
com.joel.agent-secrets	running	98048	Secret lease daemon	no public port
com.joel.imsg-rpc	running	61110	iMessage JSON-RPC socket daemon	Unix socket /tmp/imsg.sock
com.joel.kube-operator-access	running	varies	stable kubectl/talos operator tunnel	local 16443 (kube), 15000 (talos)
com.joel.voice-agent	running	71887	voice agent runtime	local 8081
com.joel.local-sandbox-janitor	scheduled	(launchd timer)	ADR-0221 local sandbox janitor (scripts/local-sandbox-janitor.sh → joelclaw workload sandboxes janitor)	logs in /tmp/joelclaw/local-sandbox-janitor.{log,err}
com.joelclaw.agent-mail	spawn scheduled	(none in launchctl snapshot)	agent-mail MCP HTTP service	observed listener 127.0.0.1:8765 (python process)
com.joel.colima	not running	—	startup helper for Colima	n/a
com.joel.k8s-reboot-heal	not running	—	periodic k8s heal script	n/a
com.joel.system-bus-sync	not running	—	sync guard watcher	n/a
com.joel.gateway-tripwire	not running	—	gateway tripwire script	n/a
com.joel.content-sync-watcher	not running	—	fs watch -> content/updated event	n/a
com.joel.vault-log-sync	not running	—	Vault log sync watcher	n/a

Flagg Central launchd scaffold

Source: infra/central/README.md, infra/central/launchd/*.plist.template, and docs/prd-rhizomatic-network-canary.md.

These labels are part of the Flagg Central shadow/cutover scaffold. They are not proof that Flagg is authoritative.

Launchd label	Domain	Role	Ports / endpoints
com.joelclaw.central.colima	system LaunchDaemon	starts the dedicated joelclaw-central Colima/Docker substrate as service infrastructure	Docker socket under /Users/joelclaw/.colima/joelclaw-central/docker.sock
com.joelclaw.central.compose	system LaunchDaemon	starts the shadow Central Compose stack	Redis, Typesense, Inngest, Restate, MinIO bound to 127.0.0.1 by default
com.joelclaw.central.health	system LaunchDaemon	bounded health + recovery state machine	health.sh can invoke recover.sh --all after repeated degraded passes
com.joelclaw.central.nas-mounts	system LaunchDaemon	mounts/verifies Flagg NAS tiers	/Volumes/nas-nvme, /Volumes/three-body
com.joelclaw.chorus-rhizomatic	system LaunchDaemon	Parked historical canary; no briefing injection or live claims; stop pending steering sudo	Old endpoint 127.0.0.1:4821/mcp; old satellite tunnel 127.0.0.1:7331

Flagg reboot acceptance rule: Central is not eligible for cutover until infra/central/scripts/reboot-proof.sh passes from another machine after hard reboot with no GUI login.

Process supervision behavior: worker-supervisor

Source: infra/worker-supervisor/src/main.rs

• Default config:
  • worker dir: ~/Code/joelhooks/joelclaw/packages/system-bus
  • command: bun run src/serve.ts
  • port: 3111
  • health endpoint: /api/inngest
  • sync endpoint: /api/inngest (PUT)
  • health interval: 30s
  • restart after 3 consecutive health failures
  • restart backoff: 1s → 30s max
• Pre-start kills stale process on port 3111.
• Runs host import preflight before spawn:
  • bun --eval "await import('./src/inngest/functions/index.host.ts');"
  • on failure, skips spawn and retries with exponential backoff
• Loads env from ~/.config/system-bus.env plus leased secrets.
• Forces WORKER_ROLE=host for the supervised host worker.
• Emits OTEL events via CLI on supervisor failures/restarts:
  • worker.supervisor.preflight.failed
  • worker.supervisor.worker_exit
  • worker.supervisor.health_check.restart

Worker supervision split note

• Talon is running (com.joel.talon), but host worker is still launched via com.joel.system-bus-worker -> worker-supervisor.
• ADR + system-log indicate Talon can defer worker supervision during coexistence.

---

Kubernetes process inventory

Node

• joelclaw-controlplane-1 (Talos v1.12.4, k8s v1.35.0, internal IP 10.5.0.2)

Core services