> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qredence.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Writing evaluators

> Turn a scoring function into feedback-rich Actionable Side Information: judge-based rubrics, batching, stochastic averaging, multi-objective Pareto, and reward-hacking-resistant gates.

The evaluator is where nearly all optimization quality comes from. Whichever engine you pick can only be as good as the score you compute and the feedback you return.

When the Codex proposer is selected, the returned `info` values become materialized context for a read-only Codex subprocess. Keep feedback concrete and bounded: include the failing output, expected behavior, and a focused next change — not unrelated logs.

## The contract

```python theme={null}
def evaluate(candidate: str, example) -> tuple[float, dict]:
    return score, info
```

* `score: float` — **higher is better**. This is what the optimizer maximizes and selects on.
* `info: dict` — free-form feedback shown to the proposer as **Actionable Side Information (ASI)**. This is the single biggest lever on mutation quality.
* For single-task runs the signature is `evaluate(candidate)`. With `dataset` or `valset`, it is `evaluate(candidate, example)`. Returning a bare `float` also works — the wrapper normalizes it — but the proposer then gets no feedback. Always return the tuple.

## Batched form

When evaluations batch better than they stream — a provider batch API, one job submission per stage, or fan-out over your own infrastructure — write the batched form:

```python theme={null}
def batch_evaluate(pairs: list[tuple[str, Any]]) -> list:
    return [score_or_score_info_tuple for candidate, example in pairs]
```

Each evaluation stage (minibatch, valset, held-out test pass) arrives as one call with all its pairs. Everything below about feedback-rich `info` applies per pair. Put diagnostics in each returned `info`. The per-call channels (`oa.log()`, `capture_stdio`) do not apply to the grouped call.

## Feedback-rich `info`

The proposer writes the next candidate by reading `info`. Give it specifics:

```python theme={null}
return score, {
    "score": score,
    "output": output,                # what the candidate produced
    "expected": example.get("gold"), # what was wanted, if available
    "error_type": err_type,          # compile error / wrong answer / format violation / timeout
    "error_detail": traceback_or_diff,
    "passed_checks": [...],
    "failed_checks": [...],
}
```

Rule of thumb: if a smart human reading only `info` could tell you how to fix the candidate, the proposer LLM can too. If `info` is `{"score": 0.0}`, the search is blind.

### Built-in diagnostic channels

* **`oa.log()`.** `import gepa.optimize_anything as oa; oa.log("landing distance:", d)` inside your evaluator. Same calling convention as `print()`; output is captured per-eval (thread-safe) and auto-included in the feedback under `info["log"]`. For child threads, propagate the context via `oa.get_log_context()` and `oa.set_log_context()`.
* **`capture_stdio`.** Set `GEPAConfig(engine=EngineConfig(capture_stdio=True), ...)` and any `print()`, `stdout`, or `stderr` during evaluation lands in the feedback under `"stdout"` or `"stderr"`. This does not catch C-extension or subprocess output that bypasses Python's `sys.stdout` — route that through `oa.log()`.

## LLM-as-judge scoring

For open-ended tasks (writing quality, helpfulness, tone, rubric adherence) the evaluator can call an LLM judge and use its rating as the score, then return the written critique as feedback:

```python theme={null}
def evaluate(candidate, example):
    output = run_my_model(candidate, example)
    verdict = judge_lm(JUDGE_RUBRIC.format(task=example, answer=output))
    score = verdict["rating"] / 10.0
    return score, {
        "score": score,
        "output": output,
        "critique": verdict["critique"],
        "rubric_breakdown": verdict.get("by_criterion"),
    }
```

Pin the judge (fixed model and temperature, ideally stronger than the model being optimized), give it a concrete rubric, and average a few judge calls if its ratings are noisy. The critique is often more valuable to the proposer than the number.

## Stochastic systems

The eval server calls your function once per (candidate, example) pair. There is no `samples_per_eval` knob. For a temperature > 0 model, every score becomes a single-sample estimate and candidate selection then runs on noisy numbers.

Average N samples inside `evaluate`:

```python theme={null}
def evaluate(candidate, example, N=4):
    outs = [run_my_model(candidate, example) for _ in range(N)]
    scores = [grade(o, example) for o in outs]
    score = sum(scores) / len(scores)
    return score, {"score": score, "n": N,
                   "samples": [{"out": o, "s": s} for o, s in zip(outs, scores)]}
```

Trade-off: N× more eval calls. Pick N to balance variance against `EngineConfig.max_metric_calls`.

## Multi-objective optimization

GEPA can keep an objective-level Pareto front. Return per-objective metrics under `info["scores"]` — the adapter forwards them as `objective_scores`:

```python theme={null}
return score, {
    "score": score,
    "scores": {"correct": correct_rate, "speedup": speedup},
    ...
}
```

For the GEPA engine, pass `EngineConfig(frontier_type="hybrid")` inside `GEPAConfig(engine=...)`. Hybrid is the default (instance-level and objective-level fronts combined). `"objective"`, `"instance"`, and `"cartesian"` are the alternatives. The scalar `score` still drives final selection; the per-objective scores shape the frontier that candidates are drawn from.

## Reward-hacking-resistant scoring

The optimizer maximizes exactly what you write. A correctness-only score is gameable — e.g. the optimizer learns to emit a trivial wrapper that is "correct" but does nothing useful.

Gate the score on validity and correctness, then increase it only for the real objective:

```python theme={null}
def score_fn(result):
    if not (result["compiled"] and result["correct"]):
        return 0.0
    return f(result["speedup"])  # e.g. min(speedup / target, 1.0), monotonically increasing
```

The only way to raise the score is to be correct **and** better on what you care about. See [Gotchas](/gepa-omni/gotchas) for the full reward-hacking story.

## Determinism and robustness

* Make `evaluate` side-effect-free and resumable. It may run concurrently (`EngineConfig.max_workers`) and be retried.
* Set a seed in the GEPA engine's nested `EngineConfig(seed=0)` for reproducible search order.
* Log your own per-eval record (id, score, sub-metrics, candidate hash) for analysis. A configured `run_dir` retains GEPA's run log and state, and `oa.log()` covers in-feedback diagnostics.
* Catch and *return* failures as low scores with `info["error_*"]`, rather than raising. `EngineConfig.raise_on_exception` defaults to `True`, so an uncaught exception aborts the run. Setting it to `False` converts exceptions to score `0.0` with `info["error"]`.