> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qredence.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Gotchas and pitfalls

> Read before any real run: reward hacking, selection bias, stochastic defaults, budget sizing, stop conditions, nested PyPI config, saturated signals, and evaluator exception handling.

Every backend optimizes exactly the score and feedback your evaluator returns. Read the whole list before a real run.

## Reward hacking

A weak proxy gets gamed. A correctness-only score can reward a code candidate that wraps a reference implementation while doing none of the work that matters. Gate the score on validity and correctness, then increase it only for the real objective. Sanity-check the winning candidate against the actual goal, not only the reported score.

See [Writing evaluators](/gepa-omni/writing-evaluators#reward-hacking-resistant-scoring) for the gated-score pattern.

## Selection bias and winner's curse

In generalization mode, the selected candidate is the maximum among candidates scored on `valset`. A small or noisy validation set makes that maximum optimistic. Use a representative `valset` and average N samples inside the evaluator when the system is stochastic.

`test_set` does not reduce selection bias — it is reporting-only — but it gives an honest held-out number to report.

## Stochastic evaluation defaults to N=1

The evaluator is called once per (candidate, example) pair by default. For a temperature-bearing model, that is a single-sample estimate. Average multiple samples inside `evaluate` and return sample details in `info`. Budget for the extra calls.

## Candidate shape depends on the boundary

Direct PyPI `optimize_anything()` accepts a string, a named component mapping, or `None`. The plugin wrapper `run_optimization()` accepts a string seed. Custom proposers return `dict[str, str]` at their separate component boundary.

## The default budget may be too small

For direct PyPI GEPA, `EngineConfig.max_metric_calls` caps evaluation calls — not meaningful proposal rounds. For the `gepa` backend, size it roughly as:

```text theme={null}
generalization: max_metric_calls ≳ 15–20 × len(valset)
multi-task:     max_metric_calls ≳ 15–20 × len(dataset)
single-task:    max_metric_calls ≳ 15–20
```

Every candidate is scored on the full selection set. If a run stops after one proposal, increase the budget. PyPI config uses `EngineConfig.max_reflection_cost` for reflection spend; the plugin-native Omni wrapper separately accepts `max_token_cost`.

## Give every run a real stop condition

Use `GEPAConfig.stop_callbacks` whenever the metric has a known ceiling such as accuracy or pass rate. If evaluation caching is enabled, `max_metric_calls` counts cache misses, so a converged run can continue proposing without consuming evaluation budget. A score stop, a token cap, and a process timeout then become essential.

## PyPI configuration is nested and strict

PyPI 0.1.4 uses `GEPAConfig(engine=EngineConfig(...), reflection=ReflectionConfig(...))`. An unknown, misspelled, or stale key raises `TypeError` at construction. It does not accept top-level `engine=`, `engine_config`, `max_evals`, `max_token_cost`, or `output_dir`.

## Saturated signals return the seed

The GEPA backend learns from examples the seed gets wrong. If the seed already scores at the ceiling on the selected examples, proposals may be rejected and the seed returned unchanged. This is not necessarily a failed run: add hard examples with real failure feedback, inspect accepted proposals, or compare against `best_of_n`, which does not use the same reflective acceptance gate.

## Evaluator exceptions abort by default

`EngineConfig.raise_on_exception` defaults to `True`. Catch expected failures and return a low score with `info["error"]` or detailed `error_*` fields so the proposer can learn from them. If appropriate, set `raise_on_exception=False` to convert exceptions to score `0.0`. Do not hide unexpected failures behind a success-shaped result.