Home/Projects/Self-Improving Agent
Prototype
AI AgentsLLM InfrastructureEvaluation Systems

Self-Improving Agent

Static LLM pipelines can't improve: prompts don't change, failure modes aren't systematically analyzed, and performance drift is invisible. This system implements a principled self-improvement loop: an evaluator agent scores each output against a structured rubric, failures trigger a prompt optimizer that generates candidates, and the best-performing version is committed with performance attribution. The key insight is that measurable improvement requires three things: a rubric, a versioned history, and a delta you can reason about.

recruiter perspective

Built an autonomous agent that improves its own performance through structured evaluation

  • ›Evaluator → optimizer → validator loop — fully autonomous with human-reviewable commit log
  • ›Structured rubric scoring (6 dimensions) enables traceable, attributable improvement
  • ›Addresses the real gap: most LLM pipelines can't systematically improve without evaluation infrastructure
Read more
Hiring Summary

An autonomous agentic loop that scores its own outputs against a structured rubric, generates improved prompt candidates, validates on a held-out task set, and commits the best version with full performance attribution. Addresses a real gap: most LLM pipelines can't systematically improve because they lack evaluation infrastructure.

Each claim is paired with evidence that a recruiter can request or verify.

“The evaluator-optimizer loop measurably improves task performance over successive iterations — with versioned prompts and performance attribution showing exactly which change produced which gain.”
writeupEvaluation Loop Architecture● coming soon✓
engineerai
“Structured rubric-based evaluation produces reproducible scores, making improvement traceable — not a subjective 'this feels better' assessment.”
benchmarkImprovement Curves● local / internal✓
engineerairecruiter

Metrics marked with context indicate the measurement scope (local benchmark, prototype, qualitative estimate, etc.).

6
criteria
Rubric Dimensions
Correctness, completeness, efficiency, style, edge-case handling, documentation — scored 0–10 per dimension
Full
with rollback
Prompt Versioning
Each version stored with performance delta and diff from parent — enables rollback to any checkpoint

LLM pipelines degrade silently. Prompts that worked last quarter perform differently as models update or task distributions shift. There's no systematic way to detect drift, diagnose the cause, or improve without manual prompt engineering on intuition. The root issue: most pipelines lack evaluation infrastructure — no rubrics, no version history, no attribution.

Challenges Addressed
  • Evaluation must be rubric-based (not LLM preference): reproducible scores, not vibes
  • Prompt versioning must track performance attribution — which version improved which metric
  • The improvement loop must be autonomous: evaluator → optimizer → candidate generation → validation → commit
  • Must avoid reward hacking: rubric dimensions must be independent and resistant to gaming
◆Self-Improving AgentAI AGENTS/visual-placeholder

Three agents: Evaluator, Optimizer, Validator. Evaluator receives (task, output) and scores across 6 rubric dimensions using structured output. Optimizer receives (current_prompt, failure_modes, rubric_scores) and generates 3 candidate improved prompts with rationale. Validator runs candidates on a held-out task set and scores each. Best candidate is committed to the version store (SQLite) with full attribution. A coordinator agent manages the loop, tracks convergence, and applies early stopping if scores plateau for 5 consecutive iterations.

Stack
PythonClaude APIPydanticSQLite
Decision
Rubric-based evaluation over LLM preference scoring
LLM preference scoring is inconsistent across runs and models. A structured rubric with independent dimensions produces reproducible scores. It also makes reward hacking detectable — if one dimension improves while others drop, the commit is flagged.
Decision
Versioned prompt store with full attribution
Without version history, improvement is anecdotal. A versioned store with performance deltas makes every improvement traceable: which prompt, which dimension, how much, compared to what baseline.
Decision
3-candidate generation + held-out validation
Single-step prompt optimization is high-variance. Generating 3 candidates and evaluating on a held-out set reduces the chance of overfitting to the training task distribution.
Tradeoff
Evaluation cost vs. improvement signal fidelity
Chose: Multi-dimension rubric on a small held-out task sample
Running full evaluations on large task sets is expensive. A smaller, carefully chosen held-out set with a multi-dimension rubric balances cost and signal quality. Confidence bounds are tracked to flag low-sample conclusions.
Tradeoff
Autonomous loop vs. human-in-the-loop
Chose: Autonomous with human-reviewable commit log
Full autonomy is the efficiency goal. But each committed prompt version is logged with its rationale and performance delta — making the improvement process auditable and reversible.
01Multi-task evaluation: evaluate across diverse task distributions to prevent overfitting to a single task type
02Cross-model validation: test improved prompts across multiple LLM providers to ensure generalization
03Failure taxonomy: classify evaluation failures into types to guide targeted optimization
04Team-shared prompt library: persist versioned prompt stores across engineers for shared improvement