Your Bot Rules Trust a Text Field Anyone Can Type

An AI agent asks your site for a page. Your edge has to decide, in milliseconds, whether to serve it, challenge it, or turn it away. Most sites still make that call from the user agent string, a self reported text field the client fills in itself, plus an IP allowlist. Both are claims. Neither is proof.

Throughout this article, agent means an automated HTTP client acting for a person or a company: a crawler indexing your content, an assistant fetching a page a user asked about, a shopping agent filling one of your forms. Not a human support agent, and not an internal tool call inside your own stack.

The gap between a claim and a proof is where the cost sits. Block too broadly and you cut off the assistants your buyers now use to shortlist vendors. Allow too broadly and anything that types the right string gets handled as if it were Claude or ChatGPT. Neither outcome is a decision. Both are a default you inherited from a header format designed in the 1990s.

What a Signed Agent Request Carries

A signed request replaces the claim with arithmetic. The mechanism is HTTP Message Signatures, published as RFC 9421, which lets a client sign parts of its own request with a private key. Three headers do the work.

  • Signature-Agent points to the URL where the agent publishes its public keys.
  • Signature-Input names exactly which parts of the request were signed, when, and with which key.
  • Signature carries the signature itself, produced with an Ed25519 key pair, a compact public key scheme where the private half never leaves the sender.

Your server fetches the key directory, checks the signature against the published public key, and gets back something that is not a matter of trust: this request came from the holder of that key, and nothing in it was altered on the way. A copied user agent string does not survive that check, because the forger does not have the private key.

This is a narrow claim, and the narrowness is the point. A verified signature tells you which operator sent the request. It does not tell you what the agent intends to do with the page, or which end user it is acting for. Those are separate questions, and conflating them is the first mistake teams make here.

Four stacked layers an automated request passes through before a site allows it: claimed identity, signature check, policy evaluation, and access decision, with the signature check highlighted.
What an Automated Request Passes Before It Is Allowed

The Checks Shipped Before the Standard Did

Adoption ran ahead of standardization, and that is the part most web teams have not priced in.

Cloudflare folded message signatures into its Verified Bots program and exposes the outcome as a field customers can write bot management and firewall rules against. AWS added Web Bot Auth support to AWS WAF Bot Control, tagging each request with a verified or invalid label that operators reference in custom rules. Akamai describes the same shift as moving bot management from heuristic detection toward verifiable identity, and lets operators allow, challenge, delay, or block based on the verification result.

The specification underneath all three is draft-meunier-webbotauth-httpsig-protocol, written by Thibault Meunier at Cloudflare and Sandor Major at Google. Version 02 published on August 18, 2026. It is an individual Internet Draft. No IETF working group has adopted it, which means the wire format can still change.

So the practical position is unusual but not complicated. The verification is real and it runs at three of the largest edges on the web. The format it depends on is provisional. Read the signal, and do not hard wire it. A verification result belongs in a policy you own and can rewrite in an afternoon, not baked into application code across forty services.

The Same Question Runs in Both Directions

While your edge decides which agents to trust, the agents your own company runs are arriving at infrastructure that belongs to somebody else, making the same unprovable claim.

Enterprise Management Associates surveyed 202 IT and security leaders at organizations with 1,000 or more employees for research published with Cequence on August 31, 2026. In that sample, 94% were confident their AI agents hold no more access than they need, while only 33% actually provision agents with least privilege, the practice of granting exactly the permissions a task requires and nothing beyond it. Nearly half, 47%, cannot reliably inventory all the agents they have already deployed. And 65% have had an agent take an action outside its intended scope, with 29% reporting measurable business impact from one.

An agent you cannot inventory is an agent you cannot sign for. That matters the first time a partner or a regulator asks which of your systems touched theirs, and the answer has to come from a key you control rather than a log line naming a browser that was never involved.

Four Checks Before You Allow or Block an Agent

This is a policy decision with an engineering surface, not the reverse. Four checks, in order.

  1. Inventory what is already arriving. Pull a week of edge logs, group requests by claimed identity, and mark which of those identities can be cryptographically verified today. Most teams are surprised by both columns.
  2. Decide per agent, not per category. A crawler collecting training data and an assistant fetching a page because a customer asked a question are both bots and are not the same commercial event. Blanket bot policy prices them identically, which is how sites quietly remove themselves from the surfaces their buyers use.
  3. Keep the policy in one place you can change. Verification is an input to a rule, not the rule. Put the allow, challenge, and block decisions at the edge where a draft specification changing next quarter costs you one config edit.
  4. Sign your own outbound agents. Publish a key directory, name the agents you operate, and give the other side something to verify. It is the same control, pointed outward, and it is the only version of this you fully own.

Key Takeaways

  • Most sites still sort AI agent traffic using a self reported user agent string and an IP allowlist, neither of which proves anything.
  • HTTP Message Signatures, defined in RFC 9421, let an agent prove which operator sent a request using an Ed25519 key published at a discoverable directory.
  • Cloudflare, AWS WAF, and Akamai verify these signatures in production today, while the underlying specification remains an unadopted IETF draft.
  • A verified signature identifies the operator only. It says nothing about intent or which end user the agent serves.
  • The agents your own organization runs face the same verification question in the other direction, and most enterprises cannot yet inventory them.

Frequently Asked Questions

Is this the same as blocking AI crawlers in robots.txt?

No. A robots.txt entry is a request that a well behaved client may honor and a bad one ignores. Signature verification is an assertion you can check, which is why it can support a policy that actually holds.

Do we have to implement RFC 9421 ourselves?

Only if you terminate traffic yourself. If your site sits behind Cloudflare, AWS WAF, or Akamai, verification already happens and your work is writing the policy on top of the result. If you run your own edge, the signing scheme is a small amount of code and open source libraries exist for it.

Should we wait until the IETF adopts the draft?

The inventory, the per agent decision, and your own outbound signing are all durable regardless of what the specification does. Those are worth doing now. What is worth deferring is any implementation that would be expensive to reverse if the header format changes.

Sources

  1. IETF, "HTTP Message Signatures for automated traffic," draft-meunier-webbotauth-httpsig-protocol-02, 2026. Link.
  2. Cloudflare, "Message Signatures are now part of our Verified Bots Program, simplifying bot authentication," 2025. Link.
  3. Amazon Web Services, "Authenticate legitimate AI agent traffic with AWS WAF Bot Control," 2026. Link.
  4. Akamai, "Redefine Trust with Web Bot Authentication," 2025. Link.
  5. Enterprise Management Associates and Cequence, "Agents Without Guardrails: The Agentic AI Governance Gap in the Enterprise," 2026.
  6. Security Boulevard, "Agents Without Guardrails: Why Agentic AI Governance Must Focus on Behavior," 2026. Link.

Next Steps

Deciding which AI agents reach your site is now a policy your web team owns, and most teams are running an inherited default instead. Stable Solutions audits what is arriving, builds the verification policy at the edge where it stays cheap to change, and signs the agents you send outward. Explore our App and Web Development services or contact our team to review what your edge currently allows through.