Static LLM pipelines can't improve: prompts don't change, failure modes aren't systematically analyzed, and performance drift is invisible. This system implements a principled self-improvement loop: an evaluator agent scores each output against a structured rubric, failures trigger a prompt optimizer that generates candidates, and the best-performing version is committed with performance attribution. The key insight is that measurable improvement requires three things: a rubric, a versioned history, and a delta you can reason about.
›Addresses the real gap: most LLM pipelines can't systematically improve without evaluation infrastructure
Read more
Hiring Summary
Hiring Summary
An autonomous agentic loop that scores its own outputs against a structured rubric, generates improved prompt candidates, validates on a held-out task set, and commits the best version with full performance attribution. Addresses a real gap: most LLM pipelines can't systematically improve because they lack evaluation infrastructure.
Evidence & Proof
Each claim is paired with evidence that a recruiter can request or verify.
“The evaluator-optimizer loop measurably improves task performance over successive iterations — with versioned prompts and performance attribution showing exactly which change produced which gain.”
writeupEvaluation Loop Architecture● coming soon✓
engineerai
“Structured rubric-based evaluation produces reproducible scores, making improvement traceable — not a subjective 'this feels better' assessment.”
benchmarkImprovement Curves● local / internal✓
engineerairecruiter
Metrics
Metrics marked with context indicate the measurement scope (local benchmark, prototype, qualitative estimate, etc.).
Each version stored with performance delta and diff from parent — enables rollback to any checkpoint
Context & Problem
LLM pipelines degrade silently. Prompts that worked last quarter perform differently as models update or task distributions shift. There's no systematic way to detect drift, diagnose the cause, or improve without manual prompt engineering on intuition. The root issue: most pipelines lack evaluation infrastructure — no rubrics, no version history, no attribution.
Challenges Addressed
Evaluation must be rubric-based (not LLM preference): reproducible scores, not vibes
Prompt versioning must track performance attribution — which version improved which metric
The improvement loop must be autonomous: evaluator → optimizer → candidate generation → validation → commit
Must avoid reward hacking: rubric dimensions must be independent and resistant to gaming
Architecture
Three agents: Evaluator, Optimizer, Validator. Evaluator receives (task, output) and scores across 6 rubric dimensions using structured output. Optimizer receives (current_prompt, failure_modes, rubric_scores) and generates 3 candidate improved prompts with rationale. Validator runs candidates on a held-out task set and scores each. Best candidate is committed to the version store (SQLite) with full attribution. A coordinator agent manages the loop, tracks convergence, and applies early stopping if scores plateau for 5 consecutive iterations.
Stack
PythonClaude APIPydanticSQLite
Technical Decisions
Decision
Rubric-based evaluation over LLM preference scoring
LLM preference scoring is inconsistent across runs and models. A structured rubric with independent dimensions produces reproducible scores. It also makes reward hacking detectable — if one dimension improves while others drop, the commit is flagged.
Decision
Versioned prompt store with full attribution
Without version history, improvement is anecdotal. A versioned store with performance deltas makes every improvement traceable: which prompt, which dimension, how much, compared to what baseline.
Decision
3-candidate generation + held-out validation
Single-step prompt optimization is high-variance. Generating 3 candidates and evaluating on a held-out set reduces the chance of overfitting to the training task distribution.
Tradeoff
Evaluation cost vs. improvement signal fidelity
Chose: Multi-dimension rubric on a small held-out task sample
Running full evaluations on large task sets is expensive. A smaller, carefully chosen held-out set with a multi-dimension rubric balances cost and signal quality. Confidence bounds are tracked to flag low-sample conclusions.
Tradeoff
Autonomous loop vs. human-in-the-loop
Chose: Autonomous with human-reviewable commit log
Full autonomy is the efficiency goal. But each committed prompt version is logged with its rationale and performance delta — making the improvement process auditable and reversible.
Future Work
01Multi-task evaluation: evaluate across diverse task distributions to prevent overfitting to a single task type
02Cross-model validation: test improved prompts across multiple LLM providers to ensure generalization
03Failure taxonomy: classify evaluation failures into types to guide targeted optimization
04Team-shared prompt library: persist versioned prompt stores across engineers for shared improvement