Optimization and evaluation

qlaw/optimize.py, qlaw/evaluate.py, qlaw/datasets.py, and qlaw/metrics.py provide compilation, measurement, and bounded GEPA instruction optimization. Strategy: measure each lens first, optimize its instruction text with held-out evidence, then cascade into whole-engine compilation. Per-lens trainsets are built from the existing GRAPH_TEMPLATES plus hand-authored examples. The engine trains on full-session trajectories.

Datasets

qlaw/datasets.py builds per-lens and engine trainsets and devsets. Every example calls .with_inputs(...) — otherwise program(**example.inputs()) crashes at evaluation time.

def decompose_trainset(n: int = 20) -> list[dspy.Example]:
    """Hand-authored + template-derived (NodeContext, list[SubNode]) pairs."""

def ontology_trainset(n: int = 20) -> list[dspy.Example]:
    """(node, list[Concept]) pairs — e.g. Quantum Computing -> Qubit / Superposition / Entanglement."""

def engine_trainset(n: int = 10) -> list[dspy.Example]:
    """Full-session trajectories: (graph, active_node_id, action) -> expected graph."""

def devsets() -> dict[str, list[dspy.Example]]:
    """Held-out devsets per lens for evaluate.py — never the trainsets."""

30 examples per lens, split train / val / test.

Metrics

qlaw/metrics.py provides six metrics. dspy.Evaluate calls a metric as metric(example, prediction) — exactly two positional args. Optimizers may pass trace, pred_name, and pred_trace too, so declare defaults.

Metric	Checks
`taxonomy_adherence`	Output DTOs pin the right `NodeType` literal.
`node_validity`	Non-empty labels, item counts in 3-5 where required, valid types.
`conciseness`	Descriptions stay bounded; labels stay short.
`novelty`	Batches introduce distinct children rather than paraphrases.
`coverage`	Expected concepts, sub-nodes, or trajectories are present.
`grounding`	Enrichment intel includes non-empty sources and metrics.

Return values from a metric may be bool, float, or dspy.Prediction(score, feedback). Feedback is read only by GEPA. dspy.Evaluate reports EvaluationResult.score as a 0–100 percentage, not 0–1.

Compile pipeline

qlaw/optimize.py:

from qlaw.optimize import compile_lens, compile_engine, load_program

# One lens
compiled = compile_lens("decompose", optimizer="mipro")

# Whole engine (uses engine_trainset)
compiled = compile_engine(optimizer="mipro")

# Load a previously compiled artifact
program = load_program("artifacts/engine.mipro.json")

Optimizer conventions to keep in mind:

The metric goes on the optimizer constructor — dspy.MIPROv2(metric=...), dspy.BootstrapFewShot(metric=...).
trainset= is keyword-only at compile() — train_set= fails with TypeError.
MIPROv2(auto="light"|"medium"|"heavy") cannot be combined with explicit num_candidates or num_trials — that raises ValueError.
compile() returns a new copy; the student is not mutated.

Command-line entrypoints

uv run python -m scripts.compile --optimizer mipro   # compile lenses + engine into artifacts/
uv run python -m scripts.evaluate                    # eval harness on devsets -> eval_results.json

scripts/evaluate.py writes eval_results.json alongside artifacts/.

Bounded GEPA instruction optimization

For bounded GEPA prompt optimization of the core lenses, use the wrapper:

./scripts/optimize --lens decompose --strategy omni --engines gepa --max-evals 44

The default strategy is --strategy omni — explore all engines on a small slice, continue from the validation winner. The wrapper installs the unreleased gepa OA API at run time (DSPy pins gepa 0.1.1). By default, both the gepa engine’s proposals and evaluation use the .env model. Pass --codex-model to switch to the native Codex agent proposer.

GEPA “omni” meta-optimizer

qlaw/omni.py composes GEPA into an “omni” meta-optimizer at the DSPy layer, distinct from — but conceptually similar to — the GEPA Omni plugin. optimize_omni and optimize_parallel orchestrate GEPA across lenses.

Caching and rollout ids

DSPy caches LM calls on by default. For Refine sampling to actually sample fresh, pass a unique rollout_id and a non-zero temperature. Otherwise the same cached response is returned across attempts.

Save and load

program.save(path) — state only.
program.save(dir, save_program=True) + dspy.load(dir) — full program.
allow_pickle defaults to False. API keys are never serialized.

Fleet-RLM

Fleet Pi

GEPA Omni

Qredence Skills

Fleet Reasoner

Qredence Plugins

Datasets

Metrics

Compile pipeline

Command-line entrypoints

Bounded GEPA instruction optimization

GEPA “omni” meta-optimizer

Caching and rollout ids

Save and load

​Datasets

​Metrics

​Compile pipeline

​Command-line entrypoints

​Bounded GEPA instruction optimization

​GEPA “omni” meta-optimizer

​Caching and rollout ids

​Save and load

Datasets

Metrics

Compile pipeline

Command-line entrypoints

Bounded GEPA instruction optimization

GEPA “omni” meta-optimizer

Caching and rollout ids

Save and load