> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qredence.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Optimization and evaluation

> Compile Fleet Reasoner's lenses and engine with MIPROv2, BootstrapFewShot, or bounded GEPA; evaluate with dspy.Evaluate against the six per-lens metrics and full-session engine trainsets.

`qlaw/optimize.py`, `qlaw/evaluate.py`, `qlaw/datasets.py`, and `qlaw/metrics.py` provide compilation, measurement, and bounded GEPA instruction optimization.

**Strategy:** measure each lens first, optimize its instruction text with held-out evidence, then cascade into whole-engine compilation. Per-lens trainsets are built from the existing `GRAPH_TEMPLATES` plus hand-authored examples. The engine trains on full-session trajectories.

## Datasets

`qlaw/datasets.py` builds per-lens and engine trainsets and devsets. Every example calls `.with_inputs(...)` — otherwise `program(**example.inputs())` crashes at evaluation time.

```python theme={null}
def decompose_trainset(n: int = 20) -> list[dspy.Example]:
    """Hand-authored + template-derived (NodeContext, list[SubNode]) pairs."""

def ontology_trainset(n: int = 20) -> list[dspy.Example]:
    """(node, list[Concept]) pairs — e.g. Quantum Computing -> Qubit / Superposition / Entanglement."""

def engine_trainset(n: int = 10) -> list[dspy.Example]:
    """Full-session trajectories: (graph, active_node_id, action) -> expected graph."""

def devsets() -> dict[str, list[dspy.Example]]:
    """Held-out devsets per lens for evaluate.py — never the trainsets."""
```

30 examples per lens, split train / val / test.

## Metrics

`qlaw/metrics.py` provides six metrics. `dspy.Evaluate` calls a metric as `metric(example, prediction)` — exactly two positional args. Optimizers may pass `trace`, `pred_name`, and `pred_trace` too, so declare defaults.

| Metric               | Checks                                                            |
| -------------------- | ----------------------------------------------------------------- |
| `taxonomy_adherence` | Output DTOs pin the right `NodeType` literal.                     |
| `node_validity`      | Non-empty labels, item counts in 3-5 where required, valid types. |
| `conciseness`        | Descriptions stay bounded; labels stay short.                     |
| `novelty`            | Batches introduce distinct children rather than paraphrases.      |
| `coverage`           | Expected concepts, sub-nodes, or trajectories are present.        |
| `grounding`          | Enrichment intel includes non-empty sources and metrics.          |

Return values from a metric may be `bool`, `float`, or `dspy.Prediction(score, feedback)`. Feedback is read only by GEPA. `dspy.Evaluate` reports `EvaluationResult.score` as a **0–100 percentage**, not 0–1.

## Compile pipeline

`qlaw/optimize.py`:

```python theme={null}
from qlaw.optimize import compile_lens, compile_engine, load_program

# One lens
compiled = compile_lens("decompose", optimizer="mipro")

# Whole engine (uses engine_trainset)
compiled = compile_engine(optimizer="mipro")

# Load a previously compiled artifact
program = load_program("artifacts/engine.mipro.json")
```

Optimizer conventions to keep in mind:

* The metric goes on the optimizer **constructor** — `dspy.MIPROv2(metric=...)`, `dspy.BootstrapFewShot(metric=...)`.
* `trainset=` is **keyword-only** at `compile()` — `train_set=` fails with `TypeError`.
* `MIPROv2(auto="light"|"medium"|"heavy")` cannot be combined with explicit `num_candidates` or `num_trials` — that raises `ValueError`.
* `compile()` returns a **new copy**; the student is not mutated.

## Command-line entrypoints

```bash theme={null}
uv run python -m scripts.compile --optimizer mipro   # compile lenses + engine into artifacts/
uv run python -m scripts.evaluate                    # eval harness on devsets -> eval_results.json
```

`scripts/evaluate.py` writes `eval_results.json` alongside `artifacts/`.

## Bounded GEPA instruction optimization

For bounded GEPA prompt optimization of the core lenses, use the wrapper:

```bash theme={null}
./scripts/optimize --lens decompose --strategy omni --engines gepa --max-evals 44
```

The default strategy is `--strategy omni` — explore all engines on a small slice, continue from the validation winner. The wrapper installs the unreleased gepa OA API at run time (DSPy pins gepa 0.1.1).

By default, both the `gepa` engine's proposals and evaluation use the `.env` model. Pass `--codex-model` to switch to the native Codex agent proposer.

## GEPA "omni" meta-optimizer

`qlaw/omni.py` composes GEPA into an "omni" meta-optimizer at the DSPy layer, distinct from — but conceptually similar to — the [GEPA Omni](/gepa-omni/introduction) plugin. `optimize_omni` and `optimize_parallel` orchestrate GEPA across lenses.

## Caching and rollout ids

DSPy caches LM calls **on** by default. For `Refine` sampling to actually sample fresh, pass a unique `rollout_id` and a non-zero temperature. Otherwise the same cached response is returned across attempts.

## Save and load

* `program.save(path)` — state only.
* `program.save(dir, save_program=True)` + `dspy.load(dir)` — full program.
* `allow_pickle` defaults to `False`. API keys are never serialized.