Writing evaluators

The evaluator is where nearly all optimization quality comes from. Whichever engine you pick can only be as good as the score you compute and the feedback you return. When the Codex proposer is selected, the returned info values become materialized context for a read-only Codex subprocess. Keep feedback concrete and bounded: include the failing output, expected behavior, and a focused next change — not unrelated logs.

The contract

def evaluate(candidate: str, example) -> tuple[float, dict]:
    return score, info

score: float — higher is better. This is what the optimizer maximizes and selects on.
info: dict — free-form feedback shown to the proposer as Actionable Side Information (ASI). This is the single biggest lever on mutation quality.
For single-task runs the signature is evaluate(candidate). With dataset or valset, it is evaluate(candidate, example). Returning a bare float also works — the wrapper normalizes it — but the proposer then gets no feedback. Always return the tuple.

Batched form

When evaluations batch better than they stream — a provider batch API, one job submission per stage, or fan-out over your own infrastructure — write the batched form:

def batch_evaluate(pairs: list[tuple[str, Any]]) -> list:
    return [score_or_score_info_tuple for candidate, example in pairs]

Each evaluation stage (minibatch, valset, held-out test pass) arrives as one call with all its pairs. Everything below about feedback-rich info applies per pair. Put diagnostics in each returned info. The per-call channels (oa.log(), capture_stdio) do not apply to the grouped call.

Feedback-rich `info`

The proposer writes the next candidate by reading info. Give it specifics:

return score, {
    "score": score,
    "output": output,                # what the candidate produced
    "expected": example.get("gold"), # what was wanted, if available
    "error_type": err_type,          # compile error / wrong answer / format violation / timeout
    "error_detail": traceback_or_diff,
    "passed_checks": [...],
    "failed_checks": [...],
}

Rule of thumb: if a smart human reading only info could tell you how to fix the candidate, the proposer LLM can too. If info is {"score": 0.0}, the search is blind.

Built-in diagnostic channels

oa.log(). import gepa.optimize_anything as oa; oa.log("landing distance:", d) inside your evaluator. Same calling convention as print(); output is captured per-eval (thread-safe) and auto-included in the feedback under info["log"]. For child threads, propagate the context via oa.get_log_context() and oa.set_log_context().
capture_stdio. Set GEPAConfig(engine=EngineConfig(capture_stdio=True), ...) and any print(), stdout, or stderr during evaluation lands in the feedback under "stdout" or "stderr". This does not catch C-extension or subprocess output that bypasses Python’s sys.stdout — route that through oa.log().

LLM-as-judge scoring

For open-ended tasks (writing quality, helpfulness, tone, rubric adherence) the evaluator can call an LLM judge and use its rating as the score, then return the written critique as feedback:

def evaluate(candidate, example):
    output = run_my_model(candidate, example)
    verdict = judge_lm(JUDGE_RUBRIC.format(task=example, answer=output))
    score = verdict["rating"] / 10.0
    return score, {
        "score": score,
        "output": output,
        "critique": verdict["critique"],
        "rubric_breakdown": verdict.get("by_criterion"),
    }

Pin the judge (fixed model and temperature, ideally stronger than the model being optimized), give it a concrete rubric, and average a few judge calls if its ratings are noisy. The critique is often more valuable to the proposer than the number.

Stochastic systems

The eval server calls your function once per (candidate, example) pair. There is no samples_per_eval knob. For a temperature > 0 model, every score becomes a single-sample estimate and candidate selection then runs on noisy numbers. Average N samples inside evaluate:

def evaluate(candidate, example, N=4):
    outs = [run_my_model(candidate, example) for _ in range(N)]
    scores = [grade(o, example) for o in outs]
    score = sum(scores) / len(scores)
    return score, {"score": score, "n": N,
                   "samples": [{"out": o, "s": s} for o, s in zip(outs, scores)]}

Trade-off: N× more eval calls. Pick N to balance variance against EngineConfig.max_metric_calls.

Multi-objective optimization

GEPA can keep an objective-level Pareto front. Return per-objective metrics under info["scores"] — the adapter forwards them as objective_scores:

return score, {
    "score": score,
    "scores": {"correct": correct_rate, "speedup": speedup},
    ...
}

For the GEPA engine, pass EngineConfig(frontier_type="hybrid") inside GEPAConfig(engine=...). Hybrid is the default (instance-level and objective-level fronts combined). "objective", "instance", and "cartesian" are the alternatives. The scalar score still drives final selection; the per-objective scores shape the frontier that candidates are drawn from.

Reward-hacking-resistant scoring

The optimizer maximizes exactly what you write. A correctness-only score is gameable — e.g. the optimizer learns to emit a trivial wrapper that is “correct” but does nothing useful. Gate the score on validity and correctness, then increase it only for the real objective:

def score_fn(result):
    if not (result["compiled"] and result["correct"]):
        return 0.0
    return f(result["speedup"])  # e.g. min(speedup / target, 1.0), monotonically increasing

The only way to raise the score is to be correct and better on what you care about. See Gotchas for the full reward-hacking story.

Determinism and robustness

Make evaluate side-effect-free and resumable. It may run concurrently (EngineConfig.max_workers) and be retried.
Set a seed in the GEPA engine’s nested EngineConfig(seed=0) for reproducible search order.
Log your own per-eval record (id, score, sub-metrics, candidate hash) for analysis. A configured run_dir retains GEPA’s run log and state, and oa.log() covers in-feedback diagnostics.
Catch and return failures as low scores with info["error_*"], rather than raising. EngineConfig.raise_on_exception defaults to True, so an uncaught exception aborts the run. Setting it to False converts exceptions to score 0.0 with info["error"].

Fleet-RLM

Fleet Pi

GEPA Omni

Qredence Skills

Fleet Reasoner

Qredence Plugins

The contract

Batched form

Feedback-rich `info`

Built-in diagnostic channels

LLM-as-judge scoring

Stochastic systems

Multi-objective optimization

Reward-hacking-resistant scoring

Determinism and robustness

​The contract

​Batched form

​Feedback-rich info

​Built-in diagnostic channels

​LLM-as-judge scoring

​Stochastic systems

​Multi-objective optimization

​Reward-hacking-resistant scoring

​Determinism and robustness

The contract

Batched form

Feedback-rich `info`

Built-in diagnostic channels

LLM-as-judge scoring

Stochastic systems

Multi-objective optimization

Reward-hacking-resistant scoring

Determinism and robustness