Arc OS — Architecture

Overview

Arc OS replaces 11,321 lines of custom Python/JS infrastructure with Claude Code native tools. The AI agent IS the backend.

System Architecture (Phase 60)

graph TB
    User[👤 User Browser :18888]
    TG[📱 Telegram]
    Local[💻 Local IDE]

    subgraph "Edge"
      Nginx[Nginx :18888<br/>basic-auth + deny-list]
      FE[React CRM + Phaser<br/>frontend container]
    end

    subgraph "Master Bot Bun :19210 — 127.0.0.1 only"
      ApiSrv[api-server.ts<br/>196 LOC]
      Routes[master-bot/routes/<br/>auth · internal · cli · websocket]
      Router[shared/routes/router.ts<br/>373 LOC dispatcher]

      subgraph "19 Domain Modules — Phase 48"
        D1[auth · projects · workers]
        D2[skills · sage · chat]
        D3[wiki · files · onboarding]
        D4[billing · invites · analytics]
        D5[+ 7 more]
      end
    end

    subgraph "Federated Bots"
      Master[Master Bot<br/>orchestrator]
      Child[Child Bots<br/>per-project]
      Claude[claude -p CLI]
    end

    subgraph "Intelligence Layer"
      Eval[Binary Evals]
      Ctx[Context Router top-5]
      Learn[Learnings inject]
      Karp[Karpathy nightly loop]
    end

    RAG[shared/rag.ts<br/>Cohere + sqlite-vec]

    DB[(SQLite WAL<br/>50+ tables<br/>migrations 001-050)]
    Vault[(AES-256-GCM Vault<br/>config/vault.json)]
    Bridge[Local Bridge CLI<br/>WebSocket relay]

    User -->|HTTPS| Nginx
    Nginx --> FE
    Nginx -->|/api/crm/*| ApiSrv
    Nginx -->|/api/sse/*| ApiSrv
    Nginx -->|/ws/terminal| ApiSrv
    ApiSrv --> Routes
    Routes --> Router
    Router --> D1 & D2 & D3 & D4 & D5

    TG --> Master
    TG --> Child
    Master -.spawns via tmux.-> Child
    Child --> Claude
    Claude --> Eval & Ctx & Learn

    Router -.queries.-> RAG
    Child -.auto-sync.-> RAG
    RAG --> DB

    Router --> DB
    Router --> Vault
    Child --> Vault

    Local --> Bridge
    Bridge -.ws relay.-> ApiSrv
    Karp -.nightly.-> DB

    subgraph "Hetzner arc-cloud-prod 157.180.44.232 — Phase 60"
      HNet[arc-cloud-net bridge]
      C1[arc-user-xxxx<br/>claude CLI + child-bot<br/>2GB RAM / 1.5 CPU]
      C2[arc-user-yyyy<br/>...]
    end

    Router -.SSH hetzner_ed25519.-> HNet
    HNet --> C1 & C2
    TG -.wake + /continue.-> C1

Key facts:

• 70+ REST endpoints across 19 domain modules in shared/routes/
• 127.0.0.1 bind — Bun never exposed externally; nginx proxies through loopback
• Multi-tenancy gate — canAccessProject(registry, chatId, project) on every protected route
• Zero-Knowledge E2EE (Phase 45) — WebCrypto PBKDF2 + AES-256-GCM master key in sessionStorage
• Two-server fleet (Phase 60) — API on Contabo (62.171.128.248), user containers on Hetzner (157.180.44.232), bridged via SSH

LEGACY (v1):
User → Telegram Bot (Python 111KB) → HybridEngine → command_queue.json
       → bridge_processor.sh → Claude → command_responses.json
       → FastAPI EventBus → WebSocket → Phaser UI

V2 (NATIVE-FIRST):
User → Official Telegram Channel → Claude Code Session
       ↓                                ↓
       reply/edit_message          Agent Teams (TaskCreate/SendMessage)
                                        ↓
                                   Write state → JSON files
                                        ↓
                                   Arc OS Bridge MCP → HTTP/SSE → Phaser UI

Reduction: 11,321 lines → ~3,700 lines (67% less code)

Components

1. Claude Code Session (The Brain)

The Claude Code session replaces:

• FastAPI backend (15 services)
• HybridEngine (39KB)
• bridge_processor.sh (19.6KB)
• response_watcher.py (14KB)
• event_bus.py
• supply_chain.py (13KB)

All orchestration happens natively through Agent Teams.

2. Arc OS Bridge MCP Server (State Adapter)

TypeScript MCP server (Bun) that bridges Claude Code state to the frontend.

MCP Tools (Claude Code side):

HTTP API (Frontend side):

3. Phaser Frontend (Visual Layer)

Phaser 3.80 + Vite. Connects to MCP server via HTTP/SSE.

Key difference from v1: no WebSocket — replaced by SSE from MCP server. No command input UI — Telegram is the command interface.

4. Telegram Command Interface (Tactical UI Layer — Phase 21.0)

Telegram is the primary control panel for the entire Arc OS. Two bot tiers:

Master Bot (@citadel_ceo_bot):

• Persistent reply keyboard: Мої Проєкти / Новий Проєкт / Watchdog / Web UI
• Inline project cards per child: Restart, Delete, View MANIFEST, Active Skills
• CRM URL buttons: Project, Tasks, Files, Settings (stub links to frontend)
• Callback query router for all inline actions
• Delete confirmation flow: inline keyboard → YES/CANCEL → edit message
• setMyCommands registration at startup

Child Bots (@cv2_pt_bot, etc.):

• Every Claude response gets inline buttons on the last chunk:
  • STOP (SIGKILL) / PAUSE (SIGSTOP) / RESUME (SIGCONT) — subprocess control
  • BTW — queues additional context for the next Claude call
  • Fix It — auto-generates fix prompt from last bot response
• Skills label button showing loaded project competencies
• Callback query router for all signal/context actions

CRM Integration (development):

CRM_BASE_URL = http://62.171.128.248:18888  (dev)
               https://crm.citadel.v2       (future prod)

All keyboard layouts defined in shared/ui_templates.ts — single source for UI changes.

Callback data protocol: action:target (max 64 bytes)

• Master: restart:pt, delete:pt, confirm_del:pt, cancel_del:pt, manifest:pt, skills:pt
• Child: stop, pause, resume, btw, fixit, skills_info

Data Flow

1. User sends "/status" via Telegram
2. Claude Code session receives via Telegram Channel
3. Rick reads state/office-state.json + TaskList()
4. Rick replies via Telegram (edit_message for live updates)
5. Rick calls update_agent_state() via MCP
6. MCP server writes to state/office-state.json
7. File watcher detects change → pushes SSE event
8. Frontend receives SSE → updates agent sprites

State Management

All state lives in JSON files under state/:

Agent Teams tasks managed by Claude Code natively at ~/.claude/tasks/citadel-v2/.

Agents (6)

Supply Chain

CEO → Rick (decompose) → Agent Team (execute)
                        → Cross-Review (Summer validates)
                        → Rick (synthesize)
                        → CEO (final result)

Enforced through:

• Blueprints: Department configs defining agent roles
• Skills: Dynamically-loaded expertise
• HANDOFF Protocol: Structured context passing between agents

Technology Stack

Lifecycle Hooks

Claude Code hooks (~/.claude/settings.json) auto-sync agent state:

SubagentStop event → subagent-stop.sh → office-state.json (agent=idle)
                                              ↓
                                   StateManager (fs.watch) → SSE → Phaser UI

Stop event → session-end.sh → latest_wrapup.txt + all agents=idle

Scripts: scripts/citadel-hooks/subagent-stop.sh, scripts/citadel-hooks/session-end.sh

Key fields: subagentName (SubagentStop), stopReason (Stop), cwd (both).

Semantic Search — Self-hosted RAG (Phase 71, 2026-06-05)

Self-hosted retrieval over the existing SQLite SSOT. Replaces the Phase 36 NotebookLM Bridge (decommissioned, port 19213 free, see services/.archived-notebooklm-bridge-2026-06-05/ + docs/public/architecture/rag-architecture.md).

Chat / arc kb / Cloud PM
        │
        ▼
shared/rag.ts search()
        │
        ▼
shared/embeddings.ts  ──►  Cohere /v2/embed
  embedQuery(text)         (multilingual, 1024-dim float32)
        │
        ▼
embeddings_vec (vec0 virtual)  +  embeddings (metadata)
  KNN (k * 4 overfetch)  →  project + doc_type filter at JOIN  →  top-K passages

Fallback: executeLocalKnowledgeSearch (keyword scan over wiki + issues + roadmap)
          when top-hit distance > 1.6 or zero results (covers brand-new projects).

• Write hooks (Phase 71.5): syncIssue / syncWiki / syncSkill fire-and-forget on the corresponding write paths. Cohere errors print + die; never roll back the SQL/disk write.
• Per-project namespace + sentinel _global_ namespace for fleet-wide skills. Search call sites query both in parallel + merge by distance.
• Public endpoint: GET /api/crm/projects/:name/rag/search?q=...&k=6&include_global=true&doc_types=wiki,issue,skill — powers arc kb search + the in-chat ask_notebooklm tool (tool name kept stable for downstream Anthropic tool-use compatibility).
• Backfill: scripts/phase-71-backfill-rag.ts (idempotent, --force overrides). Prod run 2026-06-05: 814 docs / ~3,150 chunks / 0 errors on Production key.
• Soak measured SLO: p50 ≤ 250ms ✅ · p95 ≤ 500ms ✅ · p99 ≤ 1500ms ✅. UK→EN cross-lingual recall 100% on grounded pairs.

Data & Task Routing Policy

Three-tier system. Each tier has a strict purpose. Mixing tiers = chaos.

┌─────────────────────────────────────────────────────────────────┐
│  TIER 1: Working Memory (Hot)                                   │
│  office-state.json + state/tasks/                               │
│  CLI: /citadel-task, /citadel-status                            │
│                                                                 │
│  Daily agent tasks. Lives during session. Dies after.           │
│  Completed task → /citadel-wrapup → Tier 3.                    │
├─────────────────────────────────────────────────────────────────┤
│  TIER 2: Strategic Backlog (Warm)                               │
│  GitHub Issues (Claude-CEO + citadel-v2 repos)                  │
│                                                                 │
│  Epics only: Phase 20, Phase 21, global bugs.                   │
│  No daily tasks. No "fix typo" issues.                          │
│  One Issue = one multi-week initiative or critical defect.      │
├─────────────────────────────────────────────────────────────────┤
│  TIER 3: Long-term Memory (Cold)                                │
│  docs/library-export/ + project wiki + closed issues            │
│  → embedded via shared/rag.ts (Cohere + sqlite-vec, Phase 71)   │
│  CLI: /citadel-wrapup, arc kb search "<query>"                  │
│                                                                 │
│  Encyclopedia. Session results, architectural decisions, RAG.   │
│  Read-only archive. No active tasks. No status tracking.        │
└─────────────────────────────────────────────────────────────────┘

Routing Rules

Lifecycle

CEO gives task → Tier 1 (agent works on it)
                     ↓ completed
                 /citadel-wrapup → Tier 3 (archived)
                     ↓ if strategic
                 gh issue create → Tier 2 (tracked long-term)

Anti-patterns (DO NOT)

• Do NOT create GitHub Issues for daily tasks ("fix typo", "update config")
• Do NOT store active task status in library-export/ files
• Do NOT use office-state.json for long-term decisions or architecture notes
• Do NOT duplicate: one task lives in exactly one tier at a time

Infrastructure (Phase 20.5)

Structured Logging (shared/logger.ts)

JSONL format, dual output (file + console), daily file splitting by category.

/var/log/citadel/
├── master/
│   ├── system-2026-04-02.log   ← lifecycle, config, health
│   ├── dialog-2026-04-02.log   ← (not used by master)
│   └── error-2026-04-02.log    ← errors (also in system)
├── citadel-v2/
│   ├── system-2026-04-02.log
│   ├── dialog-2026-04-02.log   ← user ↔ Claude messages
│   └── error-2026-04-02.log
└── <project-name>/             ← per onboarded project

Log rotation: config/logrotate-citadel.conf → /etc/logrotate.d/citadel (daily, 7-day retention).

Secrets Vault (shared/vault.ts)

AES-256-GCM encrypted storage for bot tokens.

Key source: SECRET_ENCRYPTION_KEY env → config/vault-key file → auto-generated
Storage:    config/vault.json (atomic writes: tmp + mv)
Cache:      In-memory after initVault() — no disk I/O per getSecret()
Fallback:   getSecret("FOO") → vault cache → process.env.FOO
Naming:     child:<project-name>:token

Master's own token stays in .env (vault loads inside master process).

Self-Healing Watchdog (master-bot/watchdog.ts)

Background monitor for child bots. Runs inside master bot process.

Every 30s: HTTP health check → /api/child/health (5s timeout)
  Healthy → reset failures
  Unhealthy → increment failures
  3+ failures + backoff elapsed → auto-restart (kill tmux → start new with vault token)
  Backoff: 30s → 1m → 5m → 15m → 60m (cap)
  10 consecutive failures → permanently disable + notify CEO

State persisted to config/watchdog-state.json (survives master restart). CEO notifications via Telegram: first restart, restart failure, permanent disable. /watchdog command: real-time status of all children.

Project Removal Protocol (Phase 20.4 + 21.0)

/remove_project <name> → triple-confirmation → full cleanup:

1. Kill tmux session: child-<name>
2. Mass Kill ghosts: ps aux | grep child-<name> → kill -9 all PIDs
3. Port check: ss -tlnp | grep :<port> → force-free if occupied
4. Remove from bot_registry.json + in-memory reload
5. Delete /opt/repos/<name>/ (safety: path must be under /opt/repos/, min 3 segments)
6. Delete /var/log/citadel/<name>/

Protected names: citadel-v2, citadel, claude-ceo, claude-CEO — removal blocked.

Ghost process problem (lesson learned): after tmux kill, orphan bun processes may hold ports. Mass kill + port check prevents "phantom bots" that respond to health checks but ignore Telegram.

Skill Management (Phase 21.0)

Child bots load skills from two sources at startup, deduplicated via Set:

Source 1: MANIFEST.md (JSON)
  → manifest.skills[]          (matched during onboarding)
  → manifest.library_skills[]  (library .md files matched)

Source 2: skills/ directory
  → *.md files → filename without extension

Merge: Set<string>(manifest.skills + manifest.library_skills + skills/*.md)

Example (PT project): 5 from manifest + 7 from skills/ = 12 unique skills.

Skills appear as:

• Inline button label on child responses: 🏷️ odoo-expert, docker-ops, ...
• skills_info callback: toast with full list
• /skills command in master: lists skills dir contents

Security (Phase 42 — Sentinel audit 2026-04-23, 4 passes, 13 patches)

Full overview: SECURITY.md. Audit report: security/audit-2026-04-23.md.

Secrets & storage:

• No credentials in repo; state files contain no secrets
• Bot tokens in encrypted vault (config/vault.json, AES-256-GCM). getSecret() fallback to process.env
• Encryption key auto-generated to config/vault-key (chmod 600). vault.json, vault-key, watchdog-state.json, data/citadel.db* gitignored

Auth:

• JWT HMAC-SHA256, 24h TTL, crypto.timingSafeEqual signature compare
• OAuth Google + GitHub with CSRF state (10min, one-time)
• Email/password with verification required before login (24h TTL), password reset (30min TTL), delivered via Resend
• Token extraction — extractChatId() reads Bearer header OR ?token= query (for browser EventSource), matches crmAuthMiddleware exactly

Multi-tenancy (Phase 42):

• canAccessProject(registry, chatId, project) gate on every protected route
• CEO (ceo_chat_id) and DB user.role === 'admin' see everything; others gated by owner_id match (DB SSOT)
• SSE routes, WebSocket terminal, /api/cli/*, /api/mcp/*, /api/crm/projects/:name/* all gated
• /ws/terminal/:name?mode=interactive requires CEO or admin

Network perimeter:

• Bun binds 127.0.0.1:19210 only — nginx proxies through loopback
• /api/internal/* rejects any request carrying proxy headers (X-Forwarded-For, X-Real-IP, Forwarded) as nginx-misconfig canary
• SSRF allowlist on handleScoutAnalyze — HTTPS + exact hostname match + redirect:"manual" (blocks redirect-chain bypass)
• UFW blocks :19210, :19200 externally (:19213 no longer in use since Phase 71.8 decommissioned the NotebookLM bridge)

Input / path safety:

• isValidProjectName() regex at every entry point (incl. internal endpoints)
• safePath() on all user-controlled file paths — handleSaveSkill found + patched Phase 42.6
• Nginx deny list: /.*, /(state|scripts|config|data|knowledge-base|issues|skills|blueprints|mcp-server)/
• CORS whitelist (CRM_ALLOWED_ORIGINS), headers on errors too
• Atomic writes for registry mutations (tmp.${pid} + mv)
• Vault token decryption for child startup: use bun -e with node:crypto (NOT Python)

Regression coverage: scripts/vps-sync.sh runs 7+ post-deploy smoke tests after every deploy (loopback bind, path traversal, chat/save validation, proxy canary, SSE ?token=, fail-closed validation, SSRF block if CEO_TOKEN set).

Master Bot Modules (Phase 25 — DEBT-1 Resolution)

The master bot is organized into focused modules:

master-bot/
├── bot.ts               ← Bootstrap: vault, registry, context, wire everything (106 lines)
├── context.ts           ← MasterContext type + domain interfaces
├── api-server.ts        ← Bun.serve() — HTTP, WebSocket, SSE, CRM routes
├── tg-api.ts            ← Telegram Bot API wrapper (sendMessage, getUpdates, etc.)
├── child-state.ts       ← Read child bot state, health, heartbeat, tmux, bridge
├── telegram-commands.ts ← All /command handlers, callback query handler, message router
├── telegram.ts          ← Slim polling loop + re-exports for backward compat
├── watchdog.ts          ← Self-healing child bot monitor
└── onboarding.ts        ← Interactive project creation wizard

bot.ts imports only from telegram.ts and api-server.ts. All other modules are internal implementation details.

Bridge CLI (Phase 25 — Local Gateway)

bridge/
├── bin/citadel-bridge.ts    ← CLI entry (commander.js): connect, pull, push, status, disconnect
├── src/
│   ├── auth.ts              ← JWT validation against CRM
│   ├── config.ts            ← ~/.citadel/bridge.json persistence
│   ├── inject.ts            ← CLAUDE.md <!-- CITADEL:START/END --> markers
│   ├── sync.ts              ← Pull skills-bundle + learnings, push local learnings
│   └── heartbeat.ts         ← Session activity reporter
├── package.json
└── tsconfig.json

CRM API Endpoints (50+ total)

Core CRM (Phase 22)

Dashboard & Wiki (Phase 32)

Multi-Tenant (Phase 33)

MCP & CLI (Phase 34)

Live Terminal (Phase 35)

Cloud PM & Knowledge (Phase 36, Phase 71 refresh)

Auth & OAuth (Phase 37)

Pinned Notes (Phase 41.8)

Backend: migration 009 (pinned_notes table), pinnedNoteQueries in shared/db.ts. Frontend: ContextRail.jsx fetches on mount, listens for crm-pin-created CustomEvent, optimistic DELETE.

Data Protection Layer (Phase 45)

Hybrid at-rest encryption — client-side crypto foundation + server-side vault encryption.

Client-Side (Browser)

• frontend/src/crm/crypto/e2ee.ts — WebCrypto wrapper (214 lines)
• PBKDF2 (100k iterations, SHA-256) → AES-256-GCM master key
• Key lifecycle: initMasterKey(password) on login → sessionStorage → clearMasterKey() on logout/401
• Recovery: generateRecoveryKey() → 1Password format (XXXX-XXXX-XXXX-XXXX-XXXX)

Server-Side (Bun + SQLite)

• shared/vault.ts — encryptField()/decryptField() using AES-256-GCM vault key (v1: prefix format)
• API keys: encrypt on save, decrypt on load (transparent to frontend) — routes/system.ts
• Chat messages: auto-encrypt INSERT, auto-decrypt SELECT — db.ts + migration 015
• Recovery keys: recovery_keys table (migration 016), 4 API endpoints

Security Headers

• CSP: default-src 'self'; script-src 'self' on all CRM responses
• X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Referrer-Policy: strict-origin-when-cross-origin

PII Sanitization

• shared/pii-sanitizer.ts — redacts emails, API keys (sk-ant-*, sk-*), JWTs, card numbers
• Applied to terminal JSONL log ingest (routes/chat.ts)

Recovery Key Endpoints

Full security details: docs/SECURITY.md, docs/architecture/PHASE_45_E2EE.md.

Standard Cloud — Multi-server Fleet (Phase 60)

Full details: docs/architecture/PHASE_60_STANDARD_CLOUD.md.

Servers

SSH Bridge

All Docker operations on Hetzner are initiated from Contabo via SSH:

Contabo root → ssh -i /root/.ssh/hetzner_ed25519 [email protected] docker <cmd>

CONTAINER_SERVER_IP in vault controls routing: 127.0.0.1 = local, any other IP = SSH.

Container Lifecycle

provision → ready → [idle 30min] → paused → [TG message] → ready
                                                           ↑ wake

Lifecycle cron: */5 * * * * on Contabo → scripts/cloud-lifecycle-cron.ts.

Cloud Waitlist

Users join via POST /api/crm/cloud/waitlist. Admin invites via POST /api/crm/cloud/waitlist/invite, which auto-upgrades plan to cloud. Table: cloud_waitlist (migration 029).