> ## Documentation Index > Fetch the complete documentation index at: https://docs.qredence.ai/llms.txt > Use this file to discover all available pages before exploring further. # Observability > How fleet-rlm projects Runtime Events to bounded MLflow spans, streams typed SSE chunks, records cancellation tombstones, and exposes health and doctor probes. `fleet-rlm` treats observability as a first-class runtime concern. Every public Runtime Event is projected as a bounded child span on the active Turn trace, streamed to clients as a typed SSE chunk, and captured in committed history. The live view, the reconciled view, and the committed view stay aligned. This page describes what you see, where it comes from, and how it is bounded. For the environment names and profile bindings, see [Configuration](/fleet-rlm/reference/configuration). ## Surfaces at a glance | Surface | Purpose | Where to look | | -------------- | ------------------------------------------------------------------- | ------------------------------------------------------------------ | | MLflow tracing | One trace per Turn with bounded child spans for every Runtime Event | Local `fleet-rlm` experiment or the managed Databricks destination | | SSE stream | Live typed Runtime Events for the pi-tui timeline and web UI | `POST /api/sessions/{session_id}/turns` | | Backend logs | Timestamped backend and MLflow process logs | `.fleet_rlm/logs/` | | Health probes | Liveness and readiness for supervisors and load balancers | `GET /health`, `GET /ready` | | Doctor | Bounded environment probe for Daytona profiles | `uv run fleet doctor daytona` | ## Runtime Event projection The centralized `EventRecorder` in `src/fleet_rlm/observability/tracing.py` projects every typed public Runtime Event as a `Turn.progress.` child span under the active Turn trace. The projection covers: * RLM reasoning summaries. * Generated code. * Interpreter output. * Tool inputs and Tool outputs. * Status and progress events. * Structured results. * Streamed text. * The committed final answer. The projection never exports hidden provider chain-of-thought or arbitrary callback payloads. Child DSPy spans below Runtime Events remain structural only, even when the selected Root trace policy allows bounded readable previews on the Turn span. ## SSE Runtime Event categories Clients subscribe to Runtime Events through the SSE stream on `POST /api/sessions/{session_id}/turns`. The pi-tui timeline and the web UI both render the same categories: * `data-status` for transient preparation, heartbeat, and cancelled parts. * Reasoning parts, code parts, and output text or delta parts. * Tool invocation views with bounded allowlisted metadata. No learning bodies, no provider paths, and no raw error strings are surfaced. * `data-usage` for the turn-level usage summary. * `artifact.created*` chunks that precede the terminal `run.completed`. * `finish` followed by `[DONE]` on a normal terminal. * `abort` as the only chunk after cancellation. There is no `finish`, no `data-usage`, and no checkpoint metadata after it. See [HTTP and SSE API](/fleet-rlm/reference/http-api) for the exact chunk schema. ## Cancellation observability A cancelled attempt closes the live SSE stream with a single `abort` chunk. After settlement, Fleet writes a bounded tombstone to committed history so `GET /api/sessions/{session_id}/turns` still shows the attempt. The tombstone contains: * The original user input. * One assistant message carrying only a `cancelled` `data-status` part. * Observed usage. * The closed text `Turn cancelled`. Cancelled tombstones never contain reasoning, code, output, or Tool evidence parts. The MLflow trace for the cancelled Turn is bounded the same way. ## MLflow policy Trace policy lives in the `[mlflow]` section of `config/fleet.toml`. This is non-secret TOML policy, not environment configuration. Ambient environment variables cannot override sampling or content bounds. | Field | Default | Purpose | | -------------------------------- | -------------------------------- | --------------------------------------------------------------------------------------------- | | `mlflow.tracing_enabled` | Enabled for interactive profiles | Master switch for the tracing pipeline on a profile. | | `mlflow.async_logging` | `true` | Keeps trace export off the Turn critical path. | | `mlflow.trace_sampling_ratio` | `1.0` | Fraction of Turns exported to MLflow. `MLFLOW_TRACE_SAMPLING_RATIO` cannot override this. | | `mlflow.trace_content_max_chars` | `10000` | Bounds each readable field: prompts, reasoning, generated code, tool payloads, and responses. | | `mlflow.expose_trace_id` | `true` | Surfaces the trace id on the Turn response for client-side deep links. | `mlflow.trace_content_mode = "safe"` is removed. `fleet.toml` files that still set the key fail validation with an unknown-key error. Delete the key. Trace content is always readable up to `mlflow.trace_content_max_chars`, and the export boundary continues to protect credentials, connection strings, private paths, and system-prompt dumps. Fleet enables MLflow DSPy inference autologging for the selected experiment. Compile and evaluator traces stay disabled so live Turn observability is not disturbed by offline optimization runs. FastAPI lifespan owns one explicit tracing startup attempt and one shutdown flush. Application construction performs no external MLflow probe, so an unavailable setup marks that lifespan inactive rather than poisoning later lifespans. ## Tracking destinations by profile The tracking target follows the selected profile. | Profile | Tracking URI | Notes | | ----------------------------------- | ----------------------- | ----------------------------------------------------------------------- | | `daytona`, `daytona-recursive` | `http://127.0.0.1:5001` | Local `fleet-rlm` experiment. `fleet cli` supervises the MLflow server. | | `daytona-managed` | `databricks` | Requires the managed environment variables listed below. | | `daytona-bench`, `daytona-bench-40` | Tracing disabled | Benchmark runs stay traceless by design. | For the interactive profiles, `fleet cli` starts or reuses a local MLflow server. It checks that `GET /version` matches the installed MLflow, runs one worker, keeps SQLite metadata under `.fleet_rlm/mlflow/mlflow.db`, and writes artifacts to `.fleet_rlm/mlflow/artifacts`. It never stops a reused process. For the managed profile, set the Unity Catalog destination through these environment names: * `FLEET_MLFLOW_EXPERIMENT_NAME` * `FLEET_MLFLOW_TRACE_CATALOG` * `FLEET_MLFLOW_TRACE_SCHEMA` * `FLEET_MLFLOW_TRACE_TABLE_PREFIX` * `FLEET_MLFLOW_TRACING_SQL_WAREHOUSE_ID` Standalone `fleet web` and `fleet-rlm serve-api` do not supervise a local MLflow server. Start the configured tracking server separately before serving Turns that need to be traced. See [CLI](/fleet-rlm/reference/cli) for the supervised startup contract. ## Backend logs Backend and owned MLflow process output land in timestamped files under `.fleet_rlm/logs/`. Two symlinks point to the active files: * `.fleet_rlm/logs/latest.log` for the backend. * `.fleet_rlm/logs/mlflow-latest.log` for the supervised MLflow server. The `rlm.verbose` setting controls native DSPy host logs only. It does not control the typed Runtime Events projected through SSE or the terminal client. If you need more detail in the timeline, adjust the event projection, not `rlm.verbose`. ## Health probes Two unauthenticated probes report process status: * `GET /health` returns liveness. * `GET /ready` returns readiness. When a critical dependency is missing, `/ready` returns `503` with the same body payload so supervisors and load balancers see one shape. ## Doctor diagnostics `uv run fleet doctor daytona` runs a bounded environment probe for the Daytona profiles. It checks: * Settings loading and profile resolution. * Database connectivity and Alembic head alignment. * Provider authentication for the selected models. * Volume visibility for the configured Workspace Volume. * Scoped mounting of that Volume. * Interpreter execution inside a disposable Sandbox. Doctor creates one uniquely labelled disposable Sandbox and deletes it in `finally`. It creates no Fleet domain rows, prints only bounded diagnostic categories, and points to corrective actions when a check fails. For guided remediation, see [Troubleshooting](/fleet-rlm/guides/troubleshooting). ## See also Profiles, MLflow policy fields, and managed destination environment names. Turn SSE contract, `abort` semantics, and cancelled tombstones. `fleet cli` supervision of the local MLflow server and `fleet doctor daytona`. Reading backend logs, health probes, and doctor output when Turns misbehave.