Auto-Pilot Engineering: Building a Self-Improving LLM Core with Ground-Truth Benchmarks

When designing Awareness3 (aw3), we set a core engineering requirement from day one: the core engine must be self-improving by design.

Complex LLM-assisted systems cover document ingestion, entity extraction, deduplication, and multi-step agentic search. We wanted to know if autonomous self-improvement was actually feasible, and how far an automated loop could take it.

We built an automated self-improvement loop. It combines three ingredients: an embedded, judge-free ground-truth benchmark (aw3-bench), online research capabilities for improvement agents, and automated code refactoring. Together they let the core engine continuously measure its own performance, research solutions online, and optimize itself on auto-pilot.

1. System Architecture: Where "Core" Sits

Our product, Awareness3, is a local-first, LLM based knowledge-base platform for macOS. You drop in documents, emails, notes, and receipts on your computer, our app organizes them into structured entities—people, companies, projects, schedules, and tasks.

Architecture-wise, the application is divided into two main layers:

1. Desktop & Web UI: The user-facing shell where users read notes, manage incoming mail, and search their knowledge base.
2. Headless Core Engine: The engine driving everything under the hood:
   • Sources Store: Imports raw documents (PDFs, markdown, EML emails, images, etc)
   • Entity Extraction Pipeline: Analyzes source text with local libraries & LLMs to extract structured entity cards (people, companies, dates, tasks) and reconciles duplicates.
   • Knowledge Base Store (`kb2`): Persists entities as plain markdown files with YAML frontmatter inside a git-backed repository on your Mac.
   • Agentic QA Engine: Answers user questions by querying local entities, reading original source documents, and synthesizing cited answers.

2. The Multi-Dimensional Optimization Space

We continuously optimize across four distinct dimensions:

1. Per-Task Model Routing: For different LLM task, optimize model routing.
2. Cost Efficiency: Minimizing dollar cost per document ingested and per question answered.
3. Latency & Execution Speed: Optimizing wall-clock speed, streaming responsiveness, and parallel Q&A concurrency.
4. Execution Control: Tuning sampling parameters (temperature, top-p), schema strictness, prompt layouts, and local GBNF grammar constraints.

3. Ground-Truth Benchmarking Methodology (aw3-bench)

To optimize these four dimensions automatically, we built an embedded, self-scoring benchmarking suite (`aw3-bench`).

Benchmark Design

Since we have our own JSON schemas of entities we extract, we can build gold entities and generate deterministic documents.

• Gold Truth First: Structured entity cards matching our ontology are created programmatically first.
• Deterministic Synthesis: Facts are rendered into N realistic text files, emails, and markdown notes.
• Judge-Free Scoring: Because true facts and source mappings are known in advance, evaluation uses exact ID matches, string normalization, and calendar semantics—no human or LLM judge required.

Decomposed Metrics & The QA Oracle

To isolate root causes, scoring is split into distinct layers:

• Extraction Composite: Combines detection F1 score, field accuracy, source provenance, and explicitly separates spurious extractions (hallucinations) from duplicates (failed deduplication).
• QA Composite: Measures answer correctness, refusal on fabricated decoy entities (abstention), and citation precision.
• Oracle Mode vs. Pipeline Mode: Oracle QA seeds gold entities directly to evaluate pure Q&A reasoning, while Pipeline QA asks questions over the newly extracted knowledge base. The delta prices the extraction-to-QA coupling penalty.

Proxy Metering

Every LLM call during a benchmark run passes through a local HTTP proxy. The proxy tallies token counts, tracks latency, and maps live vendor prices to calculate exact costs per task.

4. The Autonomous Improvement Loop: Benchmarking + Online Research + Refactoring

Connecting this benchmark harness to autonomous coding agents creates a closed-loop engineering pipeline. When an agent is tasked with optimizing the core, it executes benchmarks, parses metrics, researches external literature for solution and tips, and refactors implementation code:

How the Loop Operates:

1. Benchmark Execution: The improvement agent runs aw3-bench over a multi-seed matrix to collect detailed execution reports.
2. Diagnostic Analysis: The agent parses composite scores, identifying specific failure modes such as LLM deduplication failures, Q&A tool-calling protocol errors, or missing JSON citations.
3. Online Research & Synthesis: When a bottleneck is identified, the agent is prompted to conduct targeted online research. It searches academic literature, framework documentation, etc. It then synthesizes these findings into concrete experimental proposals and algorithm tweaks.
4. Implementation & Refactoring: Guided by its research, the agent modifies the codebase.
5. Multi-Seed Verification Gate: Candidate changes are re-evaluated across multiple seed configurations before being merged into main. This ensures gains are statistically consistent, not overfitted to a single document set.

5. Concrete Breakthroughs Driven on Auto-Pilot

By running this self-improving loop, our autonomous agents achieved several major architectural and performance breakthroughs:

A. Auto-Batched Pipeline Deduplication

• The Finding: Benchmark reports revealed Claude Opus produced ~10 duplicate entities per run. LLM prompts were simply unreliable at cross-document deduplication.
• Online Research & Agent Fix: Research into entity resolution patterns confirmed that prompt-level deduplication scales poorly. The agent moved deduplication into the pipeline engine using identity keys and serialized retrieve-judge-commit steps.
• Outcome: Duplicates dropped to 0–1, Claude Opus extraction composite jumped from 0.933 to 0.979, and API costs fell.

B. Native Tool-Calling & Error Taxonomy

• The Finding: Gemini derailed on 8 to 13 out of 16 questions when running agentic Q&A over prompt-based JSON instructions.
• Online Research & Agent Fix: The agent analyzed structured output patterns and implemented native function-calling, a typed error taxonomy (protocol, transport, no_answer), and a bounded corrective retry budget.
• Outcome: Zero protocol/transport errors across 9 benchmark runs (3 seeds × 3 models), with Gemini reliably answering ≥ 15/16 questions.

C. Staged Heterogeneous Model Routing

• The Finding: Running Claude Opus everywhere cost $6.15/bench run. Running Gemini 3.6 Flash everywhere cost $1.18/run but failed on complex agentic Q&A loops.
• Online Research & Agent Fix: The agent executed a 5-stage search across model candidates, incorporating provider docs and benchmark specs. Along the way it discovered that Moonshot's kimi-k3 failed 13/16 calls on transport errors, despite high public leaderboard scores.
• Outcome: Shipped a heterogeneous profile (gemini for extraction + sonnet-5 for Q&A) that achieved higher quality while cutting API costs by 45% ($1.85 vs $3.35/run).

D. On-Device Engine & GBNF Grammars

• The Finding: Local models like Qwen3-4B produced correct textual answers offline but failed to format JSON citations (yielding a citation precision of 0.00).
• Online Research & Agent Fix: Researching local inference techniques, the agent identified GBNF (GGML Backus-Naur Form) grammars as the solution. It compiled our QA schema directly into llama.cpp grammars for local execution.
• Outcome: Citation precision hit 1.00, and Qwen3-4B's QA score jumped +0.132 (to 0.855). That makes the 2.3 GB model a near-peer of the 8B model offline.

The autonomous iteration loop moved the benchmark score from 0.91 to 0.98, with minimal human intervention, and on less than 8 iterations. Since we are still at early stage of development and polishing, these gains were not particularly hard to get. However, the fact that no human intervention was needed + less than $100 in tokens, tells us that building this loop was worthwhile and justifies its existence. 

We'll report back with further findings on how this loop performs over time, when the codebase matures more.

6. What We're Building Next

While automated model routing, research synthesis, and pipeline refactoring yielded wins, our auto-pilot experiment loop is just getting started. Here is what we are working on next:

1. Automated Prompt Optimization: Opening system prompts as a continuous optimization dimension. Agents will mutate system prompts, evaluate benchmark deltas, and auto-tune phrasing per task.
2. Indexing & Retrieval Optimization: Auto-tuning entity search heuristics, candidate selection, and chunking parameters before LLM invocation.
3. Continuous LLM Provider Watcher: Building an automated monitor that tests newly released model endpoints against our aw3-bench suite. When a cheaper or higher-quality contender emerges, it will auto-propose a model route update.
4. Evolving Benchmarking Methodologies: Continuously researching state-of-the-art evaluation techniques to upgrade our benchmarking harness over time. Coverage will expand to complex multi-step reasoning, edge-case generation, and real-world failure modes.

By combining deterministic ground-truth benchmarks, targeted online research, and autonomous coding agents, the Awareness3 core engine continues to evolve on auto-pilot—fast, cost-effective, and verifiably accurate.