Self-improving agents sound like something a building-industry operator should avoid. Let the AI rewrite its own instructions, change its tools, and push the new behavior into production? No. That is not operational maturity. But the better pattern is useful: let the agent propose candidates, test those candidates, keep the evidence, and promote only a reviewed winner.

That is the practical thread running through Stanford CS329A's self-improving-agent material, CodeMonkeys-style coding-agent search, Automated Design of Agentic Systems, METR's long-task evaluation work, MIT Sloan's agentic-AI governance framing, and newer production-agent evaluation guidance from Thoughtworks and MLflow. The lesson is not that agents should mutate live work. The lesson is that AI workflows need promotion gates.

The useful loop is candidate, evidence, selection

CodeMonkeys is a software-engineering example, but the product shape transfers well. The system gathers context, generates candidate attempts, runs feedback, and selects among alternatives. In a building business, that same loop could compare vendor quote assumptions, draft two client-update variants, produce alternate scope summaries, or test a lead-routing workflow against past inquiries.

The dangerous version lets the AI pick a path and quietly overwrite the operating process. The useful version stores each candidate with the sources it read, the assumptions it made, the checks it passed, the checks it failed, and the reason a human promoted or rejected it.

Self-improvement belongs in a background job

A prompt, tool list, retrieval rule, scoring rubric, or workflow step should not change live behavior just because a model found a clever variant. It should move through a durable job with a budget, sandbox, eval set, trace, reviewer, rollout state, and rollback plan.

  • Queued: the system records what workflow is being improved and why.
  • Candidate generated: the AI proposes a prompt, tool, route, checklist, or output schema change.
  • Sandbox run: the candidate is tested against representative source packets.
  • Eval scored: source coverage, correctness, missing-field handling, tool use, latency, and cost are measured.
  • Human reviewed: the operator sees evidence, diffs, examples, failures, and downstream consequences.
  • Promoted or rejected: the decision, reviewer, version, notes, and rollback path are stored.

Review cannot be performative

MIT Sloan's agentic-AI guidance puts human oversight near the center of implementation. The weak version of oversight is a rushed approval button beside a confident AI summary. The strong version gives the reviewer enough context to disagree: source evidence, original documents, extracted fields, proposed changes, eval scores, known failures, and the specific business consequence of approval.

That matters for remodelers, builders, designers, showrooms, suppliers, distributors, and trades because the risks are not abstract. A bad agent promotion can change how a quote is compared, how a client gets updated, which lead receives follow-up, what a schedule says, or which assumption flows into a scope of work.

Long tasks need task-shaped evals

METR's long-task work and production-agent eval frameworks point away from one-shot grading. A workflow that reads files, uses tools, revises a plan, calls an API, asks for missing information, and waits for approval needs trajectory-level inspection. Did it choose the right tool? Did it recover from missing data? Did it preserve source constraints? Did it stop when the task became risky?

For Datum clients, this means evals should match the architecture. RAG workflows need retrieval and faithfulness checks. Tool-using agents need tool-selection and permission checks. Multi-step operations need state-transition checks. Workflows that affect money, client communication, contracts, or production schedules need human approval gates.

What this looks like in a building business

Take a vendor-comparison agent. A self-improving version should not quietly alter its comparison rules after one good result. It should propose a new matching rule, test it on old bid packets, show where it improved line-item matching, show where it incorrectly merged alternates or allowances, and ask an operator to promote the rule only after the evidence is visible.

The same applies to a client-update agent, a sales-intake classifier, a scope-summary generator, or an AI Search content workflow. Improvement is valuable. Unreviewed drift is not.

AI Search rewards the same discipline

Google's current guidance for AI Overviews and AI Mode is still not an AI-only schema trick. Helpful, original, technically clean, source-grounded content matters; structured data should match visible content. That is the public-web version of the same rule: keep claims tied to evidence, keep metadata honest, and do not hide unsupported assertions behind a polished surface.

Whether the audience is Google Search, an internal reviewer, or a job owner approving an AI workflow, the asset that compounds is the evidence trail.

The promotion-gate checklist

  • What exact behavior is the agent trying to improve?
  • Which source packets, documents, or past jobs are used for testing?
  • What counts as a better result, and who agreed to that definition?
  • What tool calls, retrieved sources, and intermediate steps are logged?
  • What failure cases are shown to the reviewer, not hidden?
  • What version is being promoted, and how can it be rolled back?
  • What production metric will be watched after promotion?

This is the operator-friendly version of self-improving AI. Not a magic button. A controlled promotion system.

Sources Read