Web Bot Auth or bot detection? Recognizing agents
For years, websites have told people and machines apart with bot detection. With AI agents, that is no longer enough: a shop wants to admit legitimate agents and fend off abuse. Web Bot Auth replaces the estimate with proof.
Leutrim Miftaraj, Founder, Innopulse Consulting GmbH
Published
Key points
- Bot detection estimates from behavior, Web Bot Auth proves origin with a signature.
- Unsigned traffic still needs bot detection; signed agents can be admitted deliberately.
- Neither proves that an agent may carry out a purchase on someone's behalf.
What this is about
Bot detection works with probabilities: IP addresses, browser features, mouse movements, speed and traffic patterns. From these comes an assessment of whether a person or a machine is asking. Good systems are accurate, but they guess.
Web Bot Auth lets the operator of a bot or agent sign every request with HTTP Message Signatures. The website verifies the signature with the published key and knows for certain whose agent is asking.
The comparison
| Feature | Bot detection | Web Bot Auth |
|---|---|---|
| Basis | Behavior and features | Cryptographic signature |
| Result | Probability | Proven origin |
| Forgeability | By imitating behavior | Only with the private key |
| Agent participation | Not needed | Agent must sign |
| Unknown traffic | Is assessed | Stays unsigned and is handled as before |
| Statement on authority | None | None |
When to use what
- Bot detection remains necessary for all unsigned traffic: scrapers, attacks and bots that do not identify themselves.
- Web Bot Auth admits signed agents from known operators deliberately, instead of hindering them with human tests or blocks.
- Both together create a tiered approach: signed agents preferred, unsigned traffic by assessment.
What neither achieves
Neither bot detection nor Web Bot Auth says anything about whether an agent may carry out a specific action on a customer's behalf. A valid signature proves identity, not authority. For purchases and other actions with consequences, evidence such as Verifiable Intent or a mandate is needed in addition.
The role of KYA
KYA verifies signed requests following Web Bot Auth and RFC 9421 against Cloudflare's reference implementation. Because identity is not authority, the verdict without a mandate is “indeterminate”. The RFC 9421 signature base tool shows the signature base of a request.
Common misconceptions
- “Signed bots are automatically harmless.” A signature proves who is asking, not that every request is welcome. Signed traffic also needs limits, for example on request rates.
- “Bot detection becomes unnecessary.” As long as not all agents sign, it remains necessary for unsigned traffic.
- “A valid signature allows the purchase.” It allows nothing. Whether a purchase is on a customer's behalf is shown only by proof of authority.
Decision guide in three questions
- Do you want to treat agents of certain operators preferentially? Then verify signatures following Web Bot Auth.
- Is there a lot of unsigned automated traffic? Then bot detection remains indispensable.
- Do agents trigger purchases? Then additionally require proof of authority.
Implementation in the shop
Many CDN providers already verify Web Bot Auth signatures at the network edge and pass the result on to the shop. The shop then decides how to treat signed agents: admit them preferentially, set its own limits and require evidence for orders. Unsigned traffic continues through existing bot detection.
A practical example
A shop notices that its bot detection increasingly blocks requests from shopping assistants that bring real orders. It admits signed requests from operators it trusts without a human test and limits their request rate. Unsigned requests continue through bot detection. For orders, it additionally requires a signed purchase intent.
Frequently asked questions
Does Web Bot Auth replace bot detection?
No. It complements it for agents that want to identify themselves. Unsigned traffic still needs bot detection.
Can an attacker forge a Web Bot Auth signature?
Not without the operator's private key. They can, however, send unsigned requests; then bot detection applies again.
Should a shop admit all signed agents?
Only those from operators it trusts. For purchases, it should additionally require proof of authority.
What does KYA verify for Web Bot Auth?
The signature base, the algorithm and the binding to the operator's key. Without a mandate, the verdict stays indeterminate, because authority is missing.
How does an operator publish its keys for Web Bot Auth?
In a directory at a well-known address on its domain. Websites load the keys from there and use them to verify the signatures.
Does a shop have to implement cryptography itself?
Not necessarily. The CDN provider often takes over verification. For its own checks, a shop can integrate the KYA engine.
What does adopting Web Bot Auth cost a shop?
If the CDN provider verifies the signatures, the effort is usually limited to rules for signed agents. A shop's own verification needs key matching and verification of the signature base.
Sources
- Web Bot Auth: Architektur-Entwurf (IETF): https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/
- RFC 9421: HTTP Message Signatures: https://www.rfc-editor.org/rfc/rfc9421