Build
The product
Evaluation datasets, indexing, internal review software, defect taxonomy, dashboard, automation, and AI agent skills.
01 / At a glance
Evaluation infrastructure, not just annotation tooling.
- Project
- Agent evaluation system for AI video model improvement
- My role
- End-to-end owner, product lead, sole engineer, and review-operations lead
- System scope
- Dataset preparation, indexing, review product, defect taxonomy, automation, analysis, and reporting
- Operating team
- Four in-house reviewers working with AI researchers and software engineers
- System output
- 14,390 judgments across 105 AI model iterations
- Delivery
- 132 pull requests across four products in 10 weeks, with 98% merged
02 / The problem
When models iterate daily but quality still requires human eyes, how do you evaluate fast enough to guide the next build?
Automated checks could confirm that a model ran. They could not reliably judge editorial timing, visual continuity, awkward repetition, narrative coherence, or whether one edit simply felt better than another.
The product needed human sensitivity at engineering speed.
03 / The operating change
From outsourced annotation volume to an internal learning system.
| Old system | New system |
|---|---|
| External vendor separated from model development | In-house reviewers working close to AI and engineering |
| Manual assignment and spreadsheet coordination | Automated assignment and one operating dashboard |
| Annotations collected as output | Judgments analyzed as model-quality evidence |
| Defects relayed through coordination | Findings connected to Linear-linked issues |
| Approximately 10 graded builds per month | Approximately 26 graded builds per month |
04 / What I owned
The product, the human system, and the model-improvement loop.
Operate
The human system
Hiring, onboarding, training, assignment, quality control, tracked hours, and the finance payout workflow.
Improve
The model loop
Analysis with the AI team, recurring failure patterns, Linear-linked tickets, and 17 weekly quality reports.
05 / The system I built
The full pipeline connected a candidate model to the next engineering iteration.
The product, automation, human operation, and engineering feedback loop worked as one system.
01
Model development
AI teamCandidate PR → hypothesis → isolated deployment. An engineer opened a candidate branch, stated what should improve and why, then deployed it to an isolated Cloud Run revision.
02
Batch rendering
Render fleet · Cloud Run · GCSCandidate and baseline arms. The workflow pinned a stable render pool, generated 50 candidate outputs, reused or rendered the reference arm, and published matched video sets.
03
Evaluation batch build
Claude skill orchestration · LinearThree approval gates before assignment. The orchestrator filed the Linear request and build changelog, prepared videos and metadata, previewed capacity, assigned two randomized repetitions, and drafted the reviewer briefing.
- Gate 1: confirm the hypothesis and matched-pair count before any write.
- Gate 2: approve the dry-run showing pair volume, repetitions, and reviewer roster.
- Gate 3: approve the outward message explaining what changed and what reviewers should watch for.
04
Human annotation
Comparison Suite · Firestore · four reviewersSide-by-side editorial judgment. The approved assignment and briefing moved into the reviewer queue. Reviewers recorded preference strength, flaw severity across video, audio, text, and prompt adherence, free-text comments, and unusable outputs. Disagreements entered a third-opinion tie-break pool.
05
Result analysis
Dashboards · Claude summary · weekly reportingWin rates plus qualitative diagnosis. The Comparison Suite resolved disagreements, calculated candidate-versus-reference performance, built flaw scoreboards, summarized reviewer comments, applied a consistent tie convention, and published the weekly evaluation report.
06
Triage and root cause
Linear · hg-debugFrom visible flaw to technical cause. The analysis handed high-priority findings to Linear and hg-debug. A flaw could be promoted into a labeled bug ticket, linked back to the model session, and traced through LLM steps, tool calls, asset selection, indexing, prompts, and rendering.
07
Next model iteration
AI and engineering teamsEvidence returned to the backlog. Root causes and prioritized defects moved back to the AI team, informing the next fix, candidate branch, stated hypothesis, and evaluation cycle.
Technical output: 132 pull requests across four connected products in 10 weeks, with 98% merged, as the sole engineer.
06 / The key judgment call
Turnaround mattered more than annotation volume.
I brought evaluation in-house.
I retired two vendor-facing systems containing 57,690 legacy annotations, removed approximately 3,200 lines of obsolete tooling, and consolidated the work into one internal tool and a four-person team.
The decision saved the company $200K in vendor contract cost while creating a faster, tighter model-feedback loop.
07 / Results
A faster, less expensive, and more defensible evaluation system.
- 14,390 human judgmentsAcross 105 AI model iterations
- ~10 → ~26 graded buildsMonthly evaluation throughput
- $200K savedVendor contract cost avoided
- Four-person teamBuilt and managed in-house
- 700+ paid review hoursAnd 1,200+ tracked hours coordinated
- 17 weekly reportsPublished on model quality
- 132 pull requests98% merged across four products
- One automated workflowReplacing manual assignment and three spreadsheets
08 / What this proves
I can turn an ambiguous AI-quality problem into a working product and operating system.
AI Tooling & PrototypingBuilt the evaluation product, pipelines, automation, agents, and analysis workflow.
Zero-to-OneDefined the requirement and launched a new internal product and operating model.
Technical Project ManagementOwned delivery, hiring, quality, risk, reporting, cost, and engineering action.
Community & EcosystemBuilt and operated the distributed human network that generated the signal.