Evaluate & Benchmark
Auto prompt optimization
Closed-loop system: an optimizer LLM writes prompts and reads failure summaries, a target LLM runs batches against synthetic data, and a JSON ledger tracks every iteration until scores converge.
The problem
Prompt optimization lacks systematic validation. Teams can't tell whether a prompt change actually improves performance or just looks different.
- You tweak a prompt, it feels better, but you can't prove it's actually better.
- Two people on the team ship contradictory prompt changes because neither can measure impact.
- Regressions from a "small" prompt edit show up in production, not in review.
What NEO built
NEO built a dual-LLM loop: an optimizer model writes and revises prompts while a separate target model evaluates them against synthetic data, with every iteration logged to a JSON ledger.
The result
Dual-LLM optimization loop
Delivers a closed-loop, dual-LLM optimization system that converges on measurably better prompts with a full audit trail of every iteration.
From the blog · 8 min
AutoPrompter: Closed-Loop Autonomous Prompt Optimization
NEO built a closed-loop system that automatically refines LLM prompts through iterative testing, failure analysis, and measurable performance improvement.
Try this in your workspace
Paste this into NEO chat to kick off the same workflow on your own data.
Build a closed-loop prompt optimizer for my prompt: have one LLM propose revisions while a second LLM scores them against my test set, and log every iteration to a ledger until the score converges.
Paste it in · review the plan · get the diff
Get NEO
