_Byron Arnao · written 2026-10-26 for a 45-minute architecture review · everything here is live and measured as of this date. Where a mitigation exists but is not yet carrying real traffic, it says shadow or not built._
How to read this in ten minutes: §1 is the decision I most want challenged (one host, on purpose) and what dies when it dies. §2 is the three questions for the meeting. §3 and §4 are the system and its control plane. §5 is the security broker. §6 is my own list of weaknesses. §7 is the adversarial critique this document already received and what changed because of it. Appendices hold the inventory. Diagram: architecture.svg.
Everything runs on one Apple Mac Mini (M4 Pro, 14 cores, 48 GB) in my house: five OpenClaw agent instances in Docker (via Colima), a shared vector-memory database, a git server, a metrics stack, a TLS proxy, 61 macOS launchd daemons that do the scheduled work, and a control plane that manages the fleet the way a small SRE team would. Public surfaces (about 200 small sites) are served from Vercel. Model spend is about $400 a month.
Why one host, on purpose: I can read every byte, the data never leaves a machine I own, and the whole thing costs electricity. The price is that the Mini is the control plane, the agents, the memory and the proxy at once.
What the fleet is for, in operational terms, and what happens when the box stops:
| Function | Who notices an outage, and how fast | Tolerable downtime (my proposal) | Tolerable data loss (my proposal) | Today's reality |
|---|---|---|---|---|
| Telegram agent answers (Gia) | Me, within minutes | 4 hours | 0 (chat history is on Telegram's side) | Restart from compose; container state on named volumes |
| Daily briefing podcast by 07:30 and the 07:30 Morning Card | Me, same morning | 1 missed day | 1 day of research notes | Pre-generated at 05:45 by a launchd job; no second producer |
| Outbound sends (email, text, LinkedIn) | The recipient, never; me, when I check | 24 hours | 0 (prepared sends live in a private queue on disk) | Queue is on the Mini's disk only |
| Shared memory (OB1, pgvector) | Every agent, next turn | 4 hours | 24 hours | Dumped nightly; in the DR bundle |
| Work log, commitments, state | Me, next session | 24 hours | 24 hours | JSONL on disk; nightly backup; weekly encrypted bundle |
| Public sites (Vercel) | Visitors | n/a (not on the Mini) | n/a | Unaffected by a Mini outage; owner consoles lose their gate |
| Collaborator git (Gitea) | One collaborator, same day | 48 hours | 24 hours | On the Mini; the Mini is on someone else's critical path |
Proposed recovery objectives for the whole box: RTO 8 hours, RPO 24 hours. I have never met them in anger. The restore script has been dry-run once and never used for a real restore. There is no UPS and no ISP failover; a power blip restarts everything (containers and keep-alive daemons come back on their own, scheduled jobs resume at their next slot).
What runs unattended versus only when I am at a keyboard: the agents, all 61 daemons, the 3-hour sentinel loop (security scans, atlas, state), the daily credential check, backups, the fixer, and the morning pipeline all run without me. The parts that need a human are a session of mine (planning, building, reviewing) and the taps that release outward sends.
Mitigations that are real today: nightly backup; weekly encrypted DR bundle (full 1.9 GB, core 3.4 MB) to cloud storage with a runbook; 34 heartbeats with a dead-man's check; containers recreated automatically if they exit. Mitigations that are not yet real: the security broker (shadow, see §5), the automatic fixer (shadow), immutable agent config (not built), a decommission programme (staged, has deleted nothing).
Known failures live at the time of writing, left in rather than hidden: 6 of 61 daemons exit non-zero (two tunnels, the photo sync to the PC, two health probes, one Cloudflare connector); 2 of 34 heartbeats are stale. The dead-man's check reports them daily. They have not been repaired, which says something about alert fatigue that I would rather discuss than conceal.
And the one I most want answered: which failure do you expect first, and would you run this?
Further questions I would take by email if there is time are in Appendix C.
| Item | Value |
|---|---|
| Machine | Mac Mini, Apple M4 Pro, 14 cores, 48 GB RAM, 1.8 TB SSD (41% used) |
| Container runtime | Docker via Colima (Linux VM: 4 vCPU, 8 GB, 200 GB) |
| Scheduling | macOS launchd: 61 system daemons (22 keep-alive services, 37 scheduled jobs) |
| Network in | Tailscale for owner consoles (tailnet-only HTTPS paths). Three deliberate public paths from the box: Tailscale Funnel on two ports (self-hosted analytics, a mentor API, the live fleet monitor) and one named Cloudflare tunnel. Nothing else listens to the internet |
| Network out | Direct for four agents; the hardened agent goes only through a Squid domain allowlist |
| Local models | Ollama on the host (qwen3-coder 30B, qwen2.5-coder 32B, gemma4) for two agents and for loops that should cost nothing |
| Second machine | A Windows PC runs a sixth agent (WhatsApp) and the photo library share; it is not managed by the control plane and its sync to the Mini fails whenever the PC sleeps |
OpenClaw is an open-source agent runtime: a gateway with channels (Telegram, WhatsApp, Signal), a skill system (markdown instructions plus scripts), scheduled jobs, a tool layer (exec, browser, web fetch), and a model-provider abstraction with fallback chains.
| Agent | Role | Version | Primary model | Channels | Skills | Jobs | Isolation |
|---|---|---|---|---|---|---|---|
| Gia | Production: Telegram, the daily podcast pipeline, research, publishing | 2026.9.7 | Claude Opus 5.5 | Telegram | 48 | 26 | Standard container; holds Gmail, LinkedIn, Twilio, Vercel, GoDaddy credentials in its environment |
| Gia2 | Hardened rebuild of Gia, shadow | 2026.9.7 | Claude Opus 5.5 | off until cutover | 48 | 3 | No credentials in the agent; broker + egress proxy + allowlist DNS + read-only root filesystem + cap_drop ALL |
| Mia | Secondary, WhatsApp allowlist | 2026.7.1 | Gemini 2.5 Flash | 9 | 1 | Standard container | |
| Cia | Coding on local models only | 2026.7.1 | Ollama qwen3-coder 30B | none | 2 | 1 | No cloud keys |
| Tia | Telegram on a local model with cloud fallbacks | 2026.7.1 | Ollama gemma4 | Telegram | 4 | 1 | Standard container |
Three agents are two release lines behind the other two. That is a known risk, not an oversight: the upgrade path (schema migration plus config rewrite) has bitten once and is done one agent at a time.
Inside Gia2 live two further personas: Ria, a responsible-AI research agent with seven host daemons of its own, and Muse, a slot for comparison runs on a different model family.
Every agent is described to itself by the same three layers: runtime config (models, tools, skills), persona and privacy rules, and orientation files loaded at boot. A change to any layer requires a re-index and restart; forgetting that was my most common early bug.
| Policy | Detail |
|---|---|
| Frontier work | Claude Opus 5.5 on the direct Anthropic API with prompt caching (on one repeated 30k-token call the second call cost $0.007 instead of $0.149; I do not yet track a fleet-wide cache hit rate) |
| Cheap lanes | Haiku 5.5 direct; Sonnet 5.5 via OpenRouter; Gemini 2.5 Flash; GPT-6.1 Sol via OpenRouter as a fallback |
| Local | Ollama for the two local agents and for background loops |
| Routing | Measured on my own task set: a fixed GPT-6.1 Sol beat every router on cost at equal accuracy (benchmarks); a small safety model ("Jev") is used as a gate, not a router |
| Fallback chains | Every agent has a 3 to 5 model chain; a daily job keeps the fleet on the newest Opus rather than a pinned version |
The agents do the domain work. The control plane runs the fleet. It is a set of Claude Code skills and small CLIs under one ops directory, plus standing rules the harness reads at the start of every session. All of the scheduled parts run from launchd without me.
| Function | Mechanism | Cadence |
|---|---|---|
| Work log | Append-only JSONL; re-renders one page and derives the to-do feed for the morning report | Every job |
| Session state + fleet atlas | Generated state file (what is in flight) and atlas (machines, agents, daemons, containers, surfaces, open problems); every session reads both before touching the box | Every 3 h |
| Commitments and goals | A commitments tracker and a goals file, rendered into the agents' memory and the morning report | Continuous |
| Outbox | Prepared outward sends (email, text) that one tap releases; the tap executes through the harness, never by bouncing me into an app; replies are detected and surfaced | On demand, checked every 45 min |
| Morning Card | One Telegram card at 07:30 with at most three items; everything else is pull-only | Daily |
| Security | Inventories 12 categories (daemons, containers, images, ports, Tailscale, Vercel projects, DNS, skills, crons, hooks, MCP servers), diffs against the last snapshot, scans only what is new; owner-gated Security Center | Every 3 h |
| Credential health | 30 checks, each secret tested inside the container that holds it, plus balances, token days-left, heartbeats | Daily 05:10 |
| Dead-man's switch | 34 heartbeats; any job older than twice its interval fails the daily check | Continuous |
| Fixer | A 5-minute daemon with playbooks, in shadow during a 90-day freeze: detects, snapshots evidence, escalates; does not act yet | 5 min |
| Backups and DR | Nightly fleet backup; weekly encrypted DR bundle to cloud storage with a runbook and a staged restore script | Nightly / weekly |
| Finance | Per-provider cost ledger from provider APIs and agent transcripts; revenue file; monthly target | Daily |
| Executive log | Every session and deploy auto-captured into a public-safe log | Continuous |
Standing rules that shape all of this: every link must be clickable on a phone; shareable means public; the private vault never enters any agent; incidents end as automation (a playbook, a "cannot automate because", or a backlog line); do it first (nothing reaches me as a bare to-do, the system arrives with the draft and one tap); a 90-day freeze on new daemons and consoles until 2027-01-16.
An agent reads attacker-controlled text (email, web pages, inbound messages) in the same process that holds credentials and a shell. The first design put allowlists inside one container; an adversarial review showed that was theatre, because an agent that can read its own secret files or post through a logged-in browser has no boundary. That review is what produced the design below.
Telegram / email-read / web
│
┌────────▼───────────┐ internal-only network ┌──────────────────────────┐
│ gia2 (LLM agent) │ ── fixed verbs only ───▶ │ gia2-broker (no LLM) │──▶ Gmail send, LinkedIn,
│ no credentials │ │ holds every credential │ Twilio, Vercel, host jobs
│ read-only rootfs │ │ policy: auto / tap / deny│
└────────┬───────────┘ │ pinned standing grants │
│ only route out │ Jev gate (shadow) │
┌────────▼───────────┐ └──────────────────────────┘
│ gia2-egress (Squid)│ domain allowlist + allowlist-only DNS
└────────────────────┘
email.send, linkedin.post, sms.send, deploy{project}, host.job{name}, slog). The broker builds the approval card from structured arguments; the model never writes the card text.email.send to a new address under a standing grant pinned to a different address (refused). Plus 17 egress and 15 broker tests. Nothing has yet tried to get out for real.A prompt-injected Gia2 can still read everything in its own context and send it to me, or propose a harmful action that I approve by mistake. Once Gia2 holds a Telegram bot token, the Telegram API is reachable and a raw call could message anyone who has started the bot. The four older agents have none of this protection: Gia, the production agent, holds every credential in its environment today. Per-channel cutover has not started. Full design and the critique that reshaped it: https://gia2-proposal.arnao.ai/
Before publishing, this document was reviewed by a frontier model (Claude Fable 5.1) instructed to act as a hostile reviewer, with the same brief a resiliency expert would bring. The critique in brief, and what changed (full text: critique.html):
| Service | What it is | Why it exists |
|---|---|---|
| OB1 (pgvector Postgres) | Shared "open brain": one vector-memory store every agent reads and writes | So five agents share one memory instead of five |
| Gitea | Self-hosted git | Shared repo for a collaborator who should not touch the Mini |
| Caddy | TLS reverse proxy on 443 | Fronts the gateways with internal certs |
| Prometheus + Grafana | Metrics | OpenClaw exports OTel diagnostics; broker actions and costs land here |
| Plausible (+ ClickHouse, Postgres) | Web analytics | Self-hosted analytics for the public sites |
| Open WebUI | Chat front end for Ollama | Local-model testing |
| A relay stack (Postgres, Redis, MinIO) | Relay for a separate product | Not part of the fleet proper |
| Count | Value |
|---|---|
| Agent instances | 5 on the Mini (+1 on the PC) |
| Containers | 25 (24 running) |
| launchd daemons | 61 (22 keep-alive, 37 scheduled; 6 exiting non-zero today) |
| Heartbeats | 34 registered, 2 stale today |
| Skills | 115 across agents; 50+ harness skills |
| Vercel projects / surfaces | ~200 / 344 (133 with no deploy in 90 days) |
| Model spend, last 30 days | $406 (Anthropic $263, Google $92, OpenRouter $14, domains $32, Twilio $3, OpenAI $2) |
| Backups | nightly, plus a weekly encrypted DR bundle |
_Written and published from the control plane itself. Source of truth for the numbers: the fleet atlas regenerated every three hours; last regenerated 2026-10-26 15:17 ET._