Fleet Pi operational runbooks

​

Incident response

​

IR-1: Bedrock API outage (circuit breaker open)

1. Verify the circuit breaker state
   • Check application logs for bedrock-api circuit breaker events
   • Look for open state transitions in logs with requestId correlation
   • Run curl -sf http://localhost:3000/api/health to confirm the web server is still healthy
2. Check Bedrock service status
   • Verify AWS credentials are valid: aws sts get-caller-identity
   • Check Bedrock model access in the AWS Console for the configured region (default us-east-1)
   • Review AWS Service Health Dashboard for regional outages
3. Inspect recent error patterns
   • Search logs for the last 30 minutes: grep "bedrock-api" or grep "circuit breaker"
   • Identify if errors are throttling (429), auth (403), or model-level (400)
   • Note the errorThresholdPercentage (50%) and volumeThreshold (5) — the breaker opens after 3 failures within 5 calls
4. Wait for automatic recovery or force reset
   • The circuit breaker resetTimeout is 30 seconds; it will attempt a half-open call after that period
   • If Bedrock is confirmed restored but the breaker is still open, restart the dev server to reset the breaker state
5. Communicate
   • Post in the incident channel: “Bedrock circuit breaker open — root cause under investigation”
   • If AWS is at fault, set status page to “degraded” and estimate recovery based on AWS status updates

​

IR-2: Chat session corruption or data loss

1. Identify the affected session
   • Extract sessionId from browser localStorage or from the start event in recent /api/chat request logs
   • Locate the Pi session file path under .fleet/sessions/ inside the repo root
2. Check session file validity
   • Verify the session JSONL file exists and is readable
   • Ensure the file is inside the repo-scoped session directory (outside files are rejected by isUsableSessionFile)
   • Look for truncated or malformed JSONL lines at the end of the file
3. Validate localStorage metadata
   • If localStorage contains an invalid sessionFile (e.g. pointing to /etc/hosts or a non-existent path), the app silently starts a fresh repo-scoped session — this is expected behavior
   • Instruct the user to clear localStorage for the site if the stored metadata is corrupt
4. Attempt manual hydration
   • Call POST /api/chat/session with the sessionId to trigger hydrateChatSession
   • If the session file cannot be opened, the server returns an empty message list with sessionReset: true
5. Recover or recreate
   • If the file is corrupt beyond repair, archive it and let the user start a new session
   • If the issue is widespread, check disk space and file system permissions on .fleet/sessions/
6. Follow up
   • Document the root cause (disk full, permission issue, or Pi SDK bug)
   • Monitor SessionManager.open error rates for 24 hours

​

Troubleshooting

​

Bedrock errors

• ThrottlingException (429) — Bedrock is rate-limiting requests.
  • Check the requestId in logs to confirm it is the same across retries
  • The Pi SDK auto-retries with exponential backoff; do not manually retry
  • If sustained, enable request batching or switch to a lower-traffic model variant
• AccessDeniedException (403) — IAM role or profile lacks bedrock:* permissions.
  • Verify AWS_PROFILE and AWS_BEARER_TOKEN_BEDROCK environment variables
  • Ensure the IAM policy includes bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream
• ValidationException (400) — The requested model ID is invalid.
  • Check modelSelection in the request body against the registry
  • Model IDs use region prefixes (e.g. us.anthropic.claude-sonnet-4-6); the backend normalizes candidates but a completely unknown ID will fail
• ModelNotReadyException — The model is not enabled in the AWS account.
  • Visit the Bedrock Console > Model access and enable the model for the current region
• Network / timeout errors — The circuit breaker timeout is 30 seconds.
  • If Bedrock does not respond within 30 seconds, the breaker counts it as a failure
  • Check VPC endpoints or corporate proxy settings if running in a restricted network

​

Session hydration failures

• Invalid sessionFile in localStorage — The browser stores only Pi session metadata (sessionFile and sessionId). If sessionFile points outside the repo session directory, isUsableSessionFile returns false and a fresh repo-scoped session is created silently.
  • Remediation: Clear site localStorage and start a new chat
• Missing or moved session file — The session JSONL was deleted or moved after the metadata was stored.
  • Remediation: Check .fleet/sessions/ for the file; if missing, the session is unrecoverable
• Corrupt session JSONL — A malformed line causes SessionManager.open to throw.
  • Remediation: Inspect the file with head -n 20 and tail -n 5; remove trailing partial lines if safe, otherwise archive and start fresh
• Race condition during streaming — If a page refresh happens while the session is being compacted, the file may be in an inconsistent state.
  • Remediation: Wait 5 seconds and retry hydration; the compaction lock should release

​

Circuit breaker states

• Closed (normal) — Requests flow to Bedrock. Failures are counted.
• Open — All calls are rejected immediately with the fallback error: "Bedrock API is temporarily unavailable due to repeated failures. Please try again later."
• Half-open — The next call is allowed through as a probe. If the probe succeeds, the breaker closes. If it fails, the breaker opens again for another resetTimeout.

​

Chat session mirror (Neon Postgres)

• You need SQL search or analytics across Pi sessions.
• You run Fleet Pi across multiple surfaces and want a single source for chat history.
• You want durable provenance for tool executions and file mutations beyond what local SQLite captures.

# Runtime mirror (pooled app-role connection)
FLEET_PI_CHAT_DATABASE_URL=postgres://fleet_pi_app:...@ep-xxxx-pooler.neon.tech/neondb?sslmode=require

# Migration-only (direct owner connection)
FLEET_PI_CHAT_MIGRATION_DATABASE_URL=postgres://neondb_owner:...@ep-xxxx.neon.tech/neondb?sslmode=require

pnpm --filter web chat:migrate

• Mirror disabled unexpectedly: confirm FLEET_PI_CHAT_DATABASE_URL is loaded in the running process (check /api/health host env, not just the .env file).
• Rows missing for a recent session: grep logs for the session’s requestId and look for mirror sync warnings; the JSONL file is still authoritative and you can re-trigger sync by hydrating the session.
• Migration fails with permission errors: verify FLEET_PI_CHAT_MIGRATION_DATABASE_URL uses neondb_owner, not the app role.

1. Snapshot the production Neon branch.
2. Create the new role with SQL as neondb_owner. Roles created from the Neon console or API join neon_superuser and get BYPASSRLS, which defeats the forced RLS on pi_* tables.
3. Run pnpm --filter web chat:migrate and pnpm --filter web auth:migrate with FLEET_PI_APP_ROLE=<new-role>.
4. Update the Worker’s database URL secrets to the new role with wrangler secret, deploy, and confirm sign-in and chat work.
5. Drop the old role only after the new Worker version serves all traffic.

​

Deployment release gate (Cloudflare + Neon)

• Required Worker env vars for Better Auth or Neon Managed Auth and the chat mirror.
• Preview trust-zone markers (FLEET_PI_DEPLOYMENT_TRUST_ZONE=preview).
• Distinct preview and production Neon database markers on Preview, and preview database URLs that do not match production.
• Optional owner-connection probes when FLEET_PI_AUTH_MIGRATION_DATABASE_URL and FLEET_PI_CHAT_MIGRATION_DATABASE_URL are supplied.

1. Deploys production with wrangler deploy on every push to main. The deploy fails when CLOUDFLARE_API_TOKEN or CLOUDFLARE_ACCOUNT_ID is missing.
2. Deploys a Worker Preview named pr-<number> for pull requests labelled cloudflare-preview. Previews get Neon preview-branch secrets only, and the workflow refuses production database URLs. Pull requests without Cloudflare credentials skip the deploy with a warning.

• deploy-readiness runs on every push with the same production-shaped and preview-shaped readiness checks as the GitHub Actions release gate. It needs no credentials.
• On main, an approval hold follows deploy-readiness. Nothing deploys until someone approves.
• deploy-production then rebuilds, runs wrangler deploy, and records a CircleCI deployment for component fleet-pi-web in environment production. It uses the fleet-pi-deploy context (CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, VITE_NEON_AUTH_URL).

1. Apply auth grants: pnpm --filter web auth:migrate
2. Apply chat mirror migrations: pnpm --filter web chat:migrate
3. Run readiness with owner URLs:
   FLEET_PI_AUTH_MIGRATION_DATABASE_URL=... \
   FLEET_PI_CHAT_MIGRATION_DATABASE_URL=... \
   pnpm --filter web verify-deployment-readiness
   
4. Deploy a Worker Preview against an isolated Neon branch with preview-only secrets.
5. Sign in against the preview and confirm session create, resume, and delete succeed.
6. Merge to main to deploy production only after the gate and smoke checks pass.

1. Capture the verify-deployment-readiness output.
2. Confirm the Neon migration ledger and RLS state manually.
3. Set the missing secrets on the Worker with wrangler secret. Never disable owner-only mirror rules.
4. Redeploy and re-run the smoke checks.

1. List the Worker versions and pick the one to restore:
   pnpm --filter web exec wrangler deployments list --name fleet-pi-web
   
2. Start the rollback pipeline on main with the version ID and a reason:
   circleci pipeline run --project <project-slug-or-id> \
     --definition-id <rollback-definition-id> \
     --branch main \
     --parameters target_version=<worker-version-id> \
     --parameters reason="<why>"
   
3. The pipeline fails when target_version is empty. Otherwise it runs wrangler rollback <worker-version-id> --name fleet-pi-web --message "rollback: <why>" --yes.
4. Re-run the smoke checks against production.

​

Owner-only session mirror and user erasure

• DELETE /api/chat/session?sessionId=… — remove one owned session from the mirror (cascades to runs, tool executions, and file mutations). A hosted session in the Pi Durable Object is only hidden, because Pi Durable has no per-conversation delete.
• DELETE /api/chat/account — erase all mirrored Pi sessions and BYOK provider credentials for the caller. On Cloudflare it also clears the caller’s Pi Durable Object storage, which erases every hosted transcript.

# Inspect only
pnpm --filter web quarantine-orphan-sessions -- --dry-run

# Quarantine ownerless rows (no delete)
pnpm --filter web quarantine-orphan-sessions

# Purge after explicit approval
pnpm --filter web quarantine-orphan-sessions -- --purge

​

Quick reference

​

Runtime cache pressure

• Pi runtimes are cached per session with a TTL controlled by FLEET_PI_RUNTIME_TTL_MS (defaults to 10 minutes).
• Lower the TTL if you want runtimes evicted sooner; raise it to keep them warmer between turns.
• Restart pnpm dev to forcibly drop all warm runtimes if you suspect leaked Pi state.

​

Workspace contract drift

• The contract version is pinned at WORKSPACE_CONTRACT_VERSION = 1 in workspace-contract.ts.
• Missing canonical directories should be re-created by workspace bootstrap on the next run; user-authored content is never overwritten.
• Use POST /api/workspace/reindex to rebuild the projection index without modifying canonical files.

​

Related files

• apps/web/src/lib/pi/circuit-breaker.ts — Breaker configuration and factory.
• apps/web/src/lib/pi/server.ts — Bedrock invocation and Pi runtime cache.
• apps/web/src/lib/pi/server-runtime.ts — Runtime TTL and PI_AGENT_DIR resolution.
• apps/web/src/lib/pi/plan-mode.ts — Mode allowlists and plan-mode extension.
• apps/web/src/lib/pi/run-provenance.ts — Provenance recording around session and tool events, including tool executions and file mutations mirrored to Neon.
• apps/web/src/lib/db/pi-session-mirror.ts — Safe sync helpers that mirror Pi sessions into Neon Postgres when FLEET_PI_CHAT_DATABASE_URL is set.
• apps/web/src/lib/pii/sanitizer.ts — Input redaction before logging.
• apps/web/src/lib/logger.ts — Pino logger with redaction and requestId correlation.
• apps/web/src/lib/workspace/workspace-contract.ts — Workspace contract version and section kinds. ino logger with redaction and requestId correlation.
• apps/web/src/lib/workspace/workspace-contract.ts — Workspace contract version and section kinds.