An AI agent that browses a vendor portal, reads an email, or opens a product page is not only collecting facts. It is also processing content that someone else controls. That content can contain misleading instructions designed to pull the agent away from the job the operator assigned.

OpenAI describes this as prompt injection and argues that the practical defense cannot depend on perfectly identifying every malicious sentence. The safer design constrains what the agent can reach and what it can do even if outside content influences it. For remodelers, builders, designers, showrooms, suppliers, distributors, and trades, that means browser access needs an operating policy before it becomes a workflow.

Separate what the agent reads from what it can change

A useful way to frame the risk is source and sink. The source is untrusted content: a webpage, email, PDF, attachment, chat message, or portal note. The sink is an action that could create harm: sending information, opening an unverified link, changing a record, placing an order, publishing a message, or sharing a file.

Reading a public installation guide to draft a summary is lower risk than reading an inbox while also holding permission to forward files and update the project system. Do not give a research task communication or write access simply because the same agent can technically use both. Split the workflow into a read-only collection step and a separately approved action step.

A familiar domain is not a complete safety rule

OpenAI's link-safety note explains why a trusted-domain list alone is incomplete: legitimate sites can redirect, and a URL itself can carry sensitive information in its path or query string. The important question is not only whether the company behind a domain is familiar. It is whether this exact destination is expected for the task and whether opening it could transmit project or customer data.

For a building business, an agent should not quietly follow a new download link from an unsolicited bid invitation, load an embedded image whose address contains job details, or submit a form because a portal page says it is required. Unknown destinations, redirects, uploads, and outbound messages should stop for review with the full destination and proposed data visible.

Write the link and access policy as a work order

  • Name the accounts, folders, portals, and public sites the workflow may read.
  • Grant only the permissions required for this task; prefer read-only access and logged-out browsing when possible.
  • List allowed actions separately from prohibited actions such as sending, uploading, purchasing, deleting, or changing a system of record.
  • Require confirmation for new destinations, redirects, file transfers, external messages, and any action that exposes project or customer information.
  • Treat instructions found inside emails, pages, PDFs, and attachments as data to inspect—not authority to expand the task.
  • Record the sources opened, links followed, data proposed for transmission, actions taken, approvals, warnings, and failures.

The confirmation screen has to be specific enough to support a real decision. “Allow this action?” is weak. Show the destination, the fields or file that would leave the system, the reason the workflow thinks the action is necessary, and the result it expects. A reviewer should be able to approve the business action without reconstructing the agent's entire session.

Test the boundary, not only the happy path

A browser workflow is not ready because it successfully reads the normal vendor page. Its eval set should include a redirected link, an unfamiliar attachment, a page that asks for unrelated information, a request to ignore the work order, a URL containing project data, an expired login, and a duplicate submission. The expected result should be explicit: proceed, warn, ask for approval, or stop.

Review logs should show whether the workflow retained the original task, kept private data out of unapproved destinations, and avoided actions outside its permission set. Any human override or correction becomes a new test case. The goal is not to prove that an agent can browse. It is to prove that it can stay inside the business boundary while browsing adversarial or simply confusing material.

This control pattern is also valuable public proof. A concrete explanation of inputs, permissions, approval gates, failure states, and tests gives a buyer more than a generic promise of safe AI. Google says AI Overviews and AI Mode require no special AI-only schema; helpful original content, visible sources, crawlable pages, and accurate structured data remain the foundation for search and AI search visibility.

Sources Read

Next step, if this note maps to a problem on your desk: Discovery Call — a 1-on-1 leverage assessment for your business ($1,500 · 90 min).

Related Field Notes