Skip to main content
fleet-rlm treats observability as a first-class runtime concern. Every public Runtime Event is projected as a bounded child span on the active Turn trace, streamed to clients as a typed SSE chunk, and captured in committed history. The live view, the reconciled view, and the committed view stay aligned. This page describes what you see, where it comes from, and how it is bounded. For the environment names and profile bindings, see Configuration.

​
Surfaces at a glance

​
Runtime Event projection

The centralized EventRecorder in src/fleet_rlm/observability/tracing.py projects every typed public Runtime Event as a Turn.progress.<event-kind> child span under the active Turn trace. The projection covers:
  • RLM reasoning summaries.
  • Generated code.
  • Interpreter output.
  • Tool inputs and Tool outputs.
  • Status and progress events.
  • Structured results.
  • Streamed text.
  • The committed final answer.
The projection never exports hidden provider chain-of-thought or arbitrary callback payloads. Child DSPy spans below Runtime Events remain structural only, even when the selected Root trace policy allows bounded readable previews on the Turn span.

​
Turn trace phases

A successfully prepared Turn with tracing enabled opens two fleet_turn root spans instead of one. Each root is tagged with fleet.trace_phase:
  • preparation covers deterministic scope resolution, ownership checks, environment acquisition, bounded context assembly, and Tool construction.
  • execution covers the native dspy.RLM trajectory, Runtime Event projection, and commit or failure settlement.
The execution root additionally carries fleet.preparation_trace_id, a one-way link back to the preparation root so you can pivot from the execution timeline to the setup work that produced it. Preparation roots never reference the execution trace. Only the execution trace id surfaces on the Turn response when mlflow.expose_trace_id is true. Preparation trace ids stay inside MLflow and are not exposed on SSE. Filter on fleet.trace_phase = "execution" in MLflow to see only the trajectories your clients can deep-link to. A failed preparation leaves only the preparation root; the execution root is never opened. Disabled tracing records neither root.

​
Token usage reporting

Delegation metrics report a token_usage_status field so a provider that omits usage cannot surface misleading all-zero token totals:
  • observed — at least one LM call returned normalized token fields (prompt, completion, cache aliases). Aggregate token counts are trustworthy.
  • unavailable — no LM call reported normalized token fields. Aggregate token counts are meaningless and should be ignored.
The field appears on the execution trace outputs alongside delegation_metrics and observed_lm_usage. Cost-only or cache-flag-only usage reports still count as unavailable; cache-token aliases count as observed. Use token_usage_status before charting or alerting on token totals so a silent provider does not read as a Turn with zero token use.

​
SSE Runtime Event categories

Clients subscribe to Runtime Events through the SSE stream on POST /api/sessions/{session_id}/turns. The pi-tui timeline and the web UI both render the same categories:
  • data-status for transient preparation, heartbeat, and cancelled parts.
  • Reasoning parts, code parts, and output text or delta parts.
  • Tool invocation views with bounded allowlisted metadata. No learning bodies, no provider paths, and no raw error strings are surfaced.
  • data-usage for the turn-level usage summary.
  • artifact.created* chunks that precede the terminal run.completed.
  • finish followed by [DONE] on a normal terminal.
  • abort as the only chunk after cancellation. There is no finish, no data-usage, and no checkpoint metadata after it.
See HTTP and SSE API for the exact chunk schema.

​
Cancellation observability

A cancelled attempt closes the live SSE stream with a single abort chunk. After settlement, Fleet writes a bounded tombstone to committed history so GET /api/sessions/{session_id}/turns still shows the attempt. The tombstone contains:
  • The original user input.
  • One assistant message carrying only a cancelled data-status part.
  • Observed usage.
  • The closed text Turn cancelled.
Cancelled tombstones never contain reasoning, code, output, or Tool evidence parts. The MLflow trace for the cancelled Turn is bounded the same way.

​
MLflow policy

Trace policy lives in the [mlflow] section of config/fleet.toml. This is non-secret TOML policy, not environment configuration. Ambient environment variables cannot override sampling or content bounds.
mlflow.trace_content_mode = "safe" is removed. fleet.toml files that still set the key fail validation with an unknown-key error. Delete the key. Trace content is always readable up to mlflow.trace_content_max_chars, and the export boundary continues to protect credentials, connection strings, private paths, and system-prompt dumps.
Fleet enables MLflow DSPy inference autologging for the selected experiment. Compile and evaluator traces stay disabled so live Turn observability is not disturbed by offline optimization runs. FastAPI lifespan owns one explicit tracing startup attempt and one shutdown flush. Application construction performs no external MLflow probe, so an unavailable setup marks that lifespan inactive rather than poisoning later lifespans.

​
Tracking destinations by profile

The tracking target follows the selected profile. For the shipped profile, fleet cli starts or reuses the local MLflow server. It checks that GET /version matches the installed MLflow, runs one worker, keeps SQLite metadata under .fleet_rlm/mlflow/mlflow.db, and writes artifacts to .fleet_rlm/mlflow/artifacts. It never stops a reused process. Databricks-hosted tracing is available through local policy. Declare mlflow.tracking_uri = "databricks" together with the experiment_name_env, trace_catalog_env, trace_schema_env, trace_table_prefix_env, and tracing_sql_warehouse_id_env references in a local profile. The FLEET_MLFLOW_* variables are read only when a profile declares them:
  • FLEET_MLFLOW_EXPERIMENT_NAME
  • FLEET_MLFLOW_TRACE_CATALOG
  • FLEET_MLFLOW_TRACE_SCHEMA
  • FLEET_MLFLOW_TRACE_TABLE_PREFIX
  • FLEET_MLFLOW_TRACING_SQL_WAREHOUSE_ID
Standalone fleet web and fleet-rlm serve-api do not supervise a local MLflow server. Start the configured tracking server separately before serving Turns that need to be traced. See CLI for the supervised startup contract.

​
Workspace Memory degradation

Workspace Memory preparation is fail-soft. A Turn proceeds even when Volume storage, the mounted Workspace agent, or Memory search degrades, because Memory context is optional. To keep those fallbacks visible without leaking payloads, each degraded read-side operation emits exactly one bounded, sanitized diagnostic. You see the same five fields in two places:
  • A WARNING log line from the fleet_rlm.daytona.memory_diagnostics logger.
  • fleet.memory_degradation.* attributes on the active fleet_turn MLflow span, when tracing is enabled by the selected profile.
The log line uses this shape:
Diagnostics carry only those five fields. Memory contents, search queries, file paths, exception messages, and environment values are never attached, and emission is one record per degraded operation. Degradation observability only covers the optional read-side preparation. Memory mutations (remember, edit_memory, forget) and list_memories stay strict: they still fail closed on duplicate ids, invalid records, or unavailable storage, and surface through the normal Tool error path with unavailable, full, or invalid_* codes.

​
Backend logs

Backend and owned MLflow process output land in timestamped files under .fleet_rlm/logs/. Two symlinks point to the active files:
  • .fleet_rlm/logs/latest.log for the backend.
  • .fleet_rlm/logs/mlflow-latest.log for the supervised MLflow server.
The rlm.verbose setting controls native DSPy host logs only. It does not control the typed Runtime Events projected through SSE or the terminal client. If you need more detail in the timeline, adjust the event projection, not rlm.verbose.

​
Health probes

Two unauthenticated probes report process status:
  • GET /health returns liveness: the process is serving HTTP, with no dependency checks. It answers even before startup composition completes.
  • GET /health/ready returns readiness: startup composition installed and the configured database answers one SELECT 1 round-trip. Before composition, or when a configured database is unreachable, it returns 503 with the closed service_not_ready error envelope.

​
Doctor diagnostics

uv run fleet doctor daytona runs a bounded environment probe for the Daytona profiles. It checks:
  • Settings loading and profile resolution.
  • Database connectivity and Alembic head alignment.
  • Provider authentication for the selected models.
  • Volume visibility for the configured Workspace Volume.
  • Scoped mounting of that Volume.
  • Interpreter execution inside a disposable Sandbox.
Doctor creates one uniquely labelled disposable Sandbox and deletes it in finally. It creates no Fleet domain rows, prints only bounded diagnostic categories, and points to corrective actions when a check fails. For guided remediation, see Troubleshooting.

​
See also

Configuration

Profiles, MLflow policy fields, and Databricks destination environment names.

HTTP and SSE API

Turn SSE contract, abort semantics, and cancelled tombstones.

CLI

fleet cli supervision of the local MLflow server and fleet doctor daytona.

Troubleshooting

Reading backend logs, health probes, and doctor output when Turns misbehave.
Last modified on September 4, 2026