Fleet Pi operational runbooks

​

Incident response

​

IR-1: Bedrock API outage (circuit breaker open)

1. Verify the circuit breaker state
   • Check application logs for bedrock-api circuit breaker events
   • Look for open state transitions in logs with requestId correlation
   • Run curl -sf http://localhost:3000/api/health to confirm the web server is still healthy
2. Check Bedrock service status
   • Verify AWS credentials are valid: aws sts get-caller-identity
   • Check Bedrock model access in the AWS Console for the configured region (default us-east-1)
   • Review AWS Service Health Dashboard for regional outages
3. Inspect recent error patterns
   • Search logs for the last 30 minutes: grep "bedrock-api" or grep "circuit breaker"
   • Identify if errors are throttling (429), auth (403), or model-level (400)
   • Note the errorThresholdPercentage (50%) and volumeThreshold (5) — the breaker opens after 3 failures within 5 calls
4. Wait for automatic recovery or force reset
   • The circuit breaker resetTimeout is 30 seconds; it will attempt a half-open call after that period
   • If Bedrock is confirmed restored but the breaker is still open, restart the dev server to reset the breaker state
5. Communicate
   • Post in the incident channel: “Bedrock circuit breaker open — root cause under investigation”
   • If AWS is at fault, set status page to “degraded” and estimate recovery based on AWS status updates

​

IR-2: Chat session corruption or data loss

1. Identify the affected session
   • Extract sessionId from browser localStorage or from the start event in recent /api/chat request logs
   • Locate the Pi session file path under .fleet/sessions/ inside the repo root
2. Check session file validity
   • Verify the session JSONL file exists and is readable
   • Ensure the file is inside the repo-scoped session directory (outside files are rejected by isUsableSessionFile)
   • Look for truncated or malformed JSONL lines at the end of the file
3. Validate localStorage metadata
   • If localStorage contains an invalid sessionFile (e.g. pointing to /etc/hosts or a non-existent path), the app silently starts a fresh repo-scoped session — this is expected behavior
   • Instruct the user to clear localStorage for the site if the stored metadata is corrupt
4. Attempt manual hydration
   • Call POST /api/chat/session with the sessionId to trigger hydrateChatSession
   • If the session file cannot be opened, the server returns an empty message list with sessionReset: true
5. Recover or recreate
   • If the file is corrupt beyond repair, archive it and let the user start a new session
   • If the issue is widespread, check disk space and file system permissions on .fleet/sessions/
6. Follow up
   • Document the root cause (disk full, permission issue, or Pi SDK bug)
   • Monitor SessionManager.open error rates for 24 hours

​

Troubleshooting

​

Bedrock errors

• ThrottlingException (429) — Bedrock is rate-limiting requests.
  • Check the requestId in logs to confirm it is the same across retries
  • The Pi SDK auto-retries with exponential backoff; do not manually retry
  • If sustained, enable request batching or switch to a lower-traffic model variant
• AccessDeniedException (403) — IAM role or profile lacks bedrock:* permissions.
  • Verify AWS_PROFILE and AWS_BEARER_TOKEN_BEDROCK environment variables
  • Ensure the IAM policy includes bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream
• ValidationException (400) — The requested model ID is invalid.
  • Check modelSelection in the request body against the registry
  • Model IDs use region prefixes (e.g. us.anthropic.claude-sonnet-4-6); the backend normalizes candidates but a completely unknown ID will fail
• ModelNotReadyException — The model is not enabled in the AWS account.
  • Visit the Bedrock Console > Model access and enable the model for the current region
• Network / timeout errors — The circuit breaker timeout is 30 seconds.
  • If Bedrock does not respond within 30 seconds, the breaker counts it as a failure
  • Check VPC endpoints or corporate proxy settings if running in a restricted network

​

Session hydration failures

• Invalid sessionFile in localStorage — The browser stores only Pi session metadata (sessionFile and sessionId). If sessionFile points outside the repo session directory, isUsableSessionFile returns false and a fresh repo-scoped session is created silently.
  • Remediation: Clear site localStorage and start a new chat
• Missing or moved session file — The session JSONL was deleted or moved after the metadata was stored.
  • Remediation: Check .fleet/sessions/ for the file; if missing, the session is unrecoverable
• Corrupt session JSONL — A malformed line causes SessionManager.open to throw.
  • Remediation: Inspect the file with head -n 20 and tail -n 5; remove trailing partial lines if safe, otherwise archive and start fresh
• Race condition during streaming — If a page refresh happens while the session is being compacted, the file may be in an inconsistent state.
  • Remediation: Wait 5 seconds and retry hydration; the compaction lock should release

​

Circuit breaker states

• Closed (normal) — Requests flow to Bedrock. Failures are counted.
• Open — All calls are rejected immediately with the fallback error: "Bedrock API is temporarily unavailable due to repeated failures. Please try again later."
• Half-open — The next call is allowed through as a probe. If the probe succeeds, the breaker closes. If it fails, the breaker opens again for another resetTimeout.

​

Chat session mirror (Neon Postgres)

• You need SQL search or analytics across Pi sessions.
• You run Fleet Pi across multiple surfaces and want a single source for chat history.
• You want durable provenance for tool executions and file mutations beyond what local SQLite captures.

# Runtime mirror (pooled app-role connection)
FLEET_PI_CHAT_DATABASE_URL=postgres://fleet_pi_app:...@ep-xxxx-pooler.neon.tech/neondb?sslmode=require

# Migration-only (direct owner connection)
FLEET_PI_CHAT_MIGRATION_DATABASE_URL=postgres://neondb_owner:...@ep-xxxx.neon.tech/neondb?sslmode=require

pnpm --filter web chat:migrate

• Mirror disabled unexpectedly: confirm FLEET_PI_CHAT_DATABASE_URL is loaded in the running process (check /api/health host env, not just the .env file).
• Rows missing for a recent session: grep logs for the session’s requestId and look for mirror sync warnings; the JSONL file is still authoritative and you can re-trigger sync by hydrating the session.
• Migration fails with permission errors: verify FLEET_PI_CHAT_MIGRATION_DATABASE_URL uses neondb_owner, not the app role.

​

Deployment release gate (Vercel + Neon)

• Required Vercel env vars for Better Auth and the chat mirror.
• Preview trust-zone markers (FLEET_PI_DEPLOYMENT_TRUST_ZONE=preview).
• Distinct preview and production Neon database markers on Preview.
• Optional owner-connection probes when FLEET_PI_AUTH_MIGRATION_DATABASE_URL and FLEET_PI_CHAT_MIGRATION_DATABASE_URL are supplied.

1. NITRO_PRESET=vercel pnpm --filter web build:vercel
2. verify-deployment-readiness against production-shaped env
3. verify-deployment-readiness against preview-shaped env (marker must appear in both auth and chat database URLs)
4. Optional owner-connection probes when migration secrets are available

1. Apply auth grants: pnpm --filter web auth:migrate
2. Apply chat mirror migrations: pnpm --filter web chat:migrate
3. Run readiness with owner URLs:
   FLEET_PI_AUTH_MIGRATION_DATABASE_URL=... \
   FLEET_PI_CHAT_MIGRATION_DATABASE_URL=... \
   pnpm --filter web verify-deployment-readiness
   
4. Deploy to Preview with an isolated Neon branch and preview-only secrets.
5. Sign in against Preview and confirm session create, resume, and delete succeed.
6. Promote to Production only after the gate and smoke checks pass.

1. Capture the verify-deployment-readiness output.
2. Confirm the Neon migration ledger and RLS state manually.
3. Set the missing env vars in Vercel — never disable owner-only mirror rules.
4. Redeploy and re-run the smoke checks.

​

Owner-only session mirror and user erasure

• DELETE /api/chat/session?sessionId=… — remove one owned session (cascades to runs, tool executions, and file mutations).
• DELETE /api/chat/account — erase all mirrored Pi sessions and BYOK provider credentials for the caller.

# Inspect only
pnpm --filter web quarantine-orphan-sessions -- --dry-run

# Quarantine ownerless rows (no delete)
pnpm --filter web quarantine-orphan-sessions

# Purge after explicit approval
pnpm --filter web quarantine-orphan-sessions -- --purge

​

Quick reference

​

Runtime cache pressure

• Pi runtimes are cached per session with a TTL controlled by FLEET_PI_RUNTIME_TTL_MS (defaults to 10 minutes).
• Lower the TTL if you want runtimes evicted sooner; raise it to keep them warmer between turns.
• Restart pnpm dev to forcibly drop all warm runtimes if you suspect leaked Pi state.

​

Workspace contract drift

• The contract version is pinned at WORKSPACE_CONTRACT_VERSION = 1 in workspace-contract.ts.
• Missing canonical directories should be re-created by workspace bootstrap on the next run; user-authored content is never overwritten.
• Use POST /api/workspace/reindex to rebuild the projection index without modifying canonical files.

​

Related files

• apps/web/src/lib/pi/circuit-breaker.ts — Breaker configuration and factory.
• apps/web/src/lib/pi/server.ts — Bedrock invocation and Pi runtime cache.
• apps/web/src/lib/pi/server-runtime.ts — Runtime TTL and PI_AGENT_DIR resolution.
• apps/web/src/lib/pi/plan-mode.ts — Mode allowlists and plan-mode extension.
• apps/web/src/lib/pi/run-provenance.ts — Provenance recording around session and tool events, including tool executions and file mutations mirrored to Neon.
• apps/web/src/lib/db/pi-session-mirror.ts — Safe sync helpers that mirror Pi sessions into Neon Postgres when FLEET_PI_CHAT_DATABASE_URL is set.
• apps/web/src/lib/pii/sanitizer.ts — Input redaction before logging.
• apps/web/src/lib/logger.ts — Pino logger with redaction and requestId correlation.
• apps/web/src/lib/workspace/workspace-contract.ts — Workspace contract version and section kinds.