Observability

fleet-rlm treats observability as a first-class runtime concern. Every public Runtime Event is projected as a bounded child span on the active Turn trace, streamed to clients as a typed SSE chunk, and captured in committed history. The live view, the reconciled view, and the committed view stay aligned. This page describes what you see, where it comes from, and how it is bounded. For the environment names and profile bindings, see Configuration.

​

Surfaces at a glance

​

Runtime Event projection

The centralized EventRecorder in src/fleet_rlm/observability/tracing.py projects every typed public Runtime Event as a Turn.progress.<event-kind> child span under the active Turn trace. The projection covers:

• RLM reasoning summaries.
• Generated code.
• Interpreter output.
• Tool inputs and Tool outputs.
• Status and progress events.
• Structured results.
• Streamed text.
• The committed final answer.

The projection never exports hidden provider chain-of-thought or arbitrary callback payloads. Child DSPy spans below Runtime Events remain structural only, even when the selected Root trace policy allows bounded readable previews on the Turn span.

​

SSE Runtime Event categories

Clients subscribe to Runtime Events through the SSE stream on POST /api/sessions/{session_id}/turns. The pi-tui timeline and the web UI both render the same categories:

• data-status for transient preparation, heartbeat, and cancelled parts.
• Reasoning parts, code parts, and output text or delta parts.
• Tool invocation views with bounded allowlisted metadata. No learning bodies, no provider paths, and no raw error strings are surfaced.
• data-usage for the turn-level usage summary.
• artifact.created* chunks that precede the terminal run.completed.
• finish followed by [DONE] on a normal terminal.
• abort as the only chunk after cancellation. There is no finish, no data-usage, and no checkpoint metadata after it.

See HTTP and SSE API for the exact chunk schema.

​

Cancellation observability

A cancelled attempt closes the live SSE stream with a single abort chunk. After settlement, Fleet writes a bounded tombstone to committed history so GET /api/sessions/{session_id}/turns still shows the attempt. The tombstone contains:

• The original user input.
• One assistant message carrying only a cancelled data-status part.
• Observed usage.
• The closed text Turn cancelled.

Cancelled tombstones never contain reasoning, code, output, or Tool evidence parts. The MLflow trace for the cancelled Turn is bounded the same way.

​

MLflow policy

Trace policy lives in the [mlflow] section of config/fleet.toml. This is non-secret TOML policy, not environment configuration. Ambient environment variables cannot override sampling or content bounds. Fleet enables MLflow DSPy inference autologging for the selected experiment. Compile and evaluator traces stay disabled so live Turn observability is not disturbed by offline optimization runs. FastAPI lifespan owns one explicit tracing startup attempt and one shutdown flush. Application construction performs no external MLflow probe, so an unavailable setup marks that lifespan inactive rather than poisoning later lifespans.

​

Tracking destinations by profile

The tracking target follows the selected profile. For the interactive profiles, fleet cli starts or reuses a local MLflow server. It checks that GET /version matches the installed MLflow, runs one worker, keeps SQLite metadata under .fleet_rlm/mlflow/mlflow.db, and writes artifacts to .fleet_rlm/mlflow/artifacts. It never stops a reused process. For the managed profile, set the Unity Catalog destination through these environment names:

• FLEET_MLFLOW_EXPERIMENT_NAME
• FLEET_MLFLOW_TRACE_CATALOG
• FLEET_MLFLOW_TRACE_SCHEMA
• FLEET_MLFLOW_TRACE_TABLE_PREFIX
• FLEET_MLFLOW_TRACING_SQL_WAREHOUSE_ID

Standalone fleet web and fleet-rlm serve-api do not supervise a local MLflow server. Start the configured tracking server separately before serving Turns that need to be traced. See CLI for the supervised startup contract.

​

Backend logs

Backend and owned MLflow process output land in timestamped files under .fleet_rlm/logs/. Two symlinks point to the active files:

• .fleet_rlm/logs/latest.log for the backend.
• .fleet_rlm/logs/mlflow-latest.log for the supervised MLflow server.

The rlm.verbose setting controls native DSPy host logs only. It does not control the typed Runtime Events projected through SSE or the terminal client. If you need more detail in the timeline, adjust the event projection, not rlm.verbose.

​

Health probes

Two unauthenticated probes report process status:

• GET /health returns liveness.
• GET /ready returns readiness. When a critical dependency is missing, /ready returns 503 with the same body payload so supervisors and load balancers see one shape.

​

Doctor diagnostics

uv run fleet doctor daytona runs a bounded environment probe for the Daytona profiles. It checks:

• Settings loading and profile resolution.
• Database connectivity and Alembic head alignment.
• Provider authentication for the selected models.
• Volume visibility for the configured Workspace Volume.
• Scoped mounting of that Volume.
• Interpreter execution inside a disposable Sandbox.

Doctor creates one uniquely labelled disposable Sandbox and deletes it in finally. It creates no Fleet domain rows, prints only bounded diagnostic categories, and points to corrective actions when a check fails. For guided remediation, see Troubleshooting.

​

See also