Troubleshooting — Problem Diagnosis

Authorization

401 — Invalid token

• Symptom: the API returns "Missing authorization" or the dashboard shows "Unauthorized"
• Causes: Token is missing, expired (24-hour TTL), or the header format is wrong
• Solution: Log in again. For SSE/WebSocket the token is passed via ?token=. The JWT is renewed every 24 hours

401 — Email not verified

• Symptom: Login succeeds but returns requires_verification: true
• Causes: The link in the verification email was not clicked (24-hour TTL)
• Solution: Check the email and click the link. Or ask an admin to verify manually. OAuth users are verified automatically

403 — No access to the project

• Symptom: File operations or the WebSocket terminal return Forbidden
• Causes: Multi-tenancy — the user is not the project owner. Or a path traversal attempt was blocked
• Solution: Check project ownership. The interactive terminal is only available to admin/CEO

Workers and Bots

Worker does not respond

• Symptom: A message was sent but there is no reply
• Causes: The Claude process is busy, the tmux session crashed, the port is taken
• Diagnosis: Check /health or /ping in Telegram. In the CRM — the status icon next to the worker
• Solution: Press STOP and retry. Or restart via CRM Settings → Restart

Status: "degraded"

• Symptom: Health check returns status: "degraded" instead of "ok"
• Causes: 3+ consecutive Claude subprocess errors
• Solution: Wait — the watchdog will restart it automatically. Or restart manually via /watchdog in Telegram

Timeout (5 minutes)

• Symptom: The bot returns "Claude timeout (5 min limit)"
• Causes: The task is too complex for a single message, large files
• Solution: Break the task into smaller steps. Switch to a faster model (Haiku)

Max turns reached

• Symptom: "Reached max turns" — Claude stopped after N steps
• Causes: The task requires more tool calls than the limit (default 20 for Developer, 10 for Consultant)
• Solution: Break the task up. Or increase max_turns via Worker Studio

Watchdog disabled bot

• Symptom: "Permanently disabled after 10 consecutive failures"
• Causes: 10 consecutive failures (missing token, deleted files, port taken)
• Solution: Fix the root cause, then restart via Master Bot /deploy or CRM Restart

Semantic search / RAG (Phase 71)

arc kb search returns an empty result for a fresh project

• Symptom: a new project without wiki/issues — the search returns No content found...
• Cause: RAG hooks (Phase 71.5) re-embed content on write, but until the first writes the index is empty. The CLI fallback to keyword search over wiki/tree also finds nothing.
• Solution: write something to a wiki/issue/skill — the embedding lands within 1-2 seconds. Or force it via arc memory refresh (re-embeds MANIFEST + ROADMAP + all wiki files).

Cohere 401 Unauthorized

• Symptom: the master logs show [rag-hook] ... failed: Cohere auth rejected (401)
• Cause: the COHERE_API_KEY in the vault expired or was rotated incorrectly.
• Solution: Platform Settings → RAG / Semantic search → Rotate. The live Test button probes the new key via /v2/embed.

Cohere 429 Rate Limited

• Symptom: Backfill or hooks fail with 429.
• Cause: A trial-tier Cohere key has a 1000 calls/month cap; production needs the Production tier.
• Solution: upgrade at https://dashboard.cohere.com/billing. The first prod backfill on 2026-06-05 burned out on exactly this — after switching to Production the 178-doc cycle completed with 0/178 errors.

Latency tail >500ms

• Symptom: individual search requests come back slowly.
• Cause: Cohere upstream tail variance — our p50 is ~195ms steady, but individual /v2/embed calls can take 800-1000ms.
• Solution: see docs/architecture/PHASE_71_SOAK_2026-06-05.md — this is a documented upstream limitation, not our code. Future phases: query-embed LRU cache, Cohere region pinning.

Frontend and Connectivity

WebSocket disconnects

• Symptom: The terminal or chat disconnects with code 1008
• Causes: The JWT token expired during the session (24-hour TTL)
• Solution: Refresh the page (F5) — the token renews automatically

SSE streaming does not work

• Symptom: The worker's reply does not appear in real time
• Causes: The project was not found in the registry, or nginx buffering is enabled
• Solution: Check the project name. SSE requires proxy_buffering off in nginx

CORS error

• Symptom: The browser console shows CORS blocked
• Causes: The origin is not in the CRM_ALLOWED_ORIGINS whitelist
• Solution: Add the origin to the CRM_ALLOWED_ORIGINS variable and restart the Master Bot

Message gets cut off

• Symptom: The reply in Telegram is truncated
• Causes: Telegram's 4096-character limit
• Solution: The bot automatically splits into parts [1/3] [2/3] [3/3]. If it doesn't — that's a bug in the split logic

Database

"Database not initialized"

• Symptom: The bot crashes with "Database not initialized. Call initDb() first."
• Causes: initDb() was not called before the first query, or the DB file was deleted
• Solution: Restart the Master Bot — it automatically initializes the DB and runs the migrations

"Database locked"

• Symptom: Random 500 errors under high load
• Causes: SQLite is being written to by multiple processes at once
• Solution: WAL mode is enabled by default. Restart stale processes

Quick Reference

Useful Diagnostic Commands

# Health checks
curl -s http://localhost:19210/api/master/health | jq .
curl -s http://localhost:19211/api/child/health | jq .

# Check tmux sessions
tmux list-sessions

# Master Bot logs
tmux capture-pane -t citadel-master -p | tail -20

# Child Bot logs
tmux capture-pane -t ws-arc-v2 -p | tail -20

# Check ports
ss -tlnp | grep '192[0-9][0-9]'

# Database state
sqlite3 data/citadel.db "PRAGMA integrity_check;"

Doc enforcement (Phase 49.1+)

git push is blocked with "doc-coverage check failed"

The pre-push hook requires doc updates when code changes. STDERR shows exactly which files need updating.

Quick fixes:

• Auto-draft: arc wrapup --generate → fill in the TODOs → commit
• Manual: see the mapping in CLAUDE.md (Documentation Law)
• Emergency bypass: git push --no-verify (leaves a trace in the git log)

Hook does not run on a fresh clone

bash scripts/setup-hooks.sh   # one-time per clone
git config core.hooksPath     # verify it equals ".githooks"

GitHub Integration (Phase 49.3)

Webhook returns 401

• Content-Type: application/json in the GitHub webhook (not form-urlencoded)
• The secret must match the output of arc github link
• Lost the secret → delete the webhook in the GitHub UI + arc github unlink, create a new one

Sidebar GitHub feed is empty

• The ContextRail is visible at a ≥1280px viewport
• Hard refresh (Ctrl+Shift+R)
• DB check: sqlite3 data/citadel.db "SELECT COUNT(*) FROM github_events WHERE project_name='<name>';"

Rate limit "429 Rate limited"

Cap = 100 req/min/project. Raise it in shared/routes/github.ts:RATE_MAX.

More details: GitHub Integration Setup.