Reward hacking
A weak proxy gets gamed. A correctness-only score can reward a code candidate that wraps a reference implementation while doing none of the work that matters. Gate the score on validity and correctness, then increase it only for the real objective. Sanity-check the winning candidate against the actual goal, not only the reported score. See Writing evaluators for the gated-score pattern.Selection bias and winner’s curse
In generalization mode, the selected candidate is the maximum among candidates scored onvalset. A small or noisy validation set makes that maximum optimistic. Use a representative valset and average N samples inside the evaluator when the system is stochastic.
test_set does not reduce selection bias — it is reporting-only — but it gives an honest held-out number to report.
Stochastic evaluation defaults to N=1
The evaluator is called once per (candidate, example) pair by default. For a temperature-bearing model, that is a single-sample estimate. Average multiple samples insideevaluate and return sample details in info. Budget for the extra calls.
Candidate shape depends on the boundary
Direct PyPIoptimize_anything() accepts a string, a named component mapping, or None. The plugin wrapper run_optimization() accepts a string seed. Custom proposers return dict[str, str] at their separate component boundary.
The default budget may be too small
For direct PyPI GEPA,EngineConfig.max_metric_calls caps evaluation calls — not meaningful proposal rounds. For the gepa backend, size it roughly as:
EngineConfig.max_reflection_cost for reflection spend; the plugin-native Omni wrapper separately accepts max_token_cost.
Give every run a real stop condition
UseGEPAConfig.stop_callbacks whenever the metric has a known ceiling such as accuracy or pass rate. If evaluation caching is enabled, max_metric_calls counts cache misses, so a converged run can continue proposing without consuming evaluation budget. A score stop, a token cap, and a process timeout then become essential.
PyPI configuration is nested and strict
PyPI 0.1.4 usesGEPAConfig(engine=EngineConfig(...), reflection=ReflectionConfig(...)). An unknown, misspelled, or stale key raises TypeError at construction. It does not accept top-level engine=, engine_config, max_evals, max_token_cost, or output_dir.
Saturated signals return the seed
The GEPA backend learns from examples the seed gets wrong. If the seed already scores at the ceiling on the selected examples, proposals may be rejected and the seed returned unchanged. This is not necessarily a failed run: add hard examples with real failure feedback, inspect accepted proposals, or compare againstbest_of_n, which does not use the same reflective acceptance gate.
Evaluator exceptions abort by default
EngineConfig.raise_on_exception defaults to True. Catch expected failures and return a low score with info["error"] or detailed error_* fields so the proposer can learn from them. If appropriate, set raise_on_exception=False to convert exceptions to score 0.0. Do not hide unexpected failures behind a success-shaped result.