Your website now needs an AI access policy

Most websites still have one front door.

Someone lands, reads a page, fills in a form, maybe buys something. Search engines crawl the same public pages and, if you are lucky, send a bit of traffic back. That was the deal. Messy and imperfect, but everyone understood it.

AI has quietly broken that deal, and most business owners have not noticed yet.

I have been watching the traffic hitting sites lately, and the thing that stands out is how many different jobs the machines are now doing. A bot visiting your site might be indexing your content for search. It might be scooping up material to train a model. It might be an agent acting for a real person who wants to compare prices, book a call, check availability, fill a basket, or pull your information into some other system.

Those are not the same thing. And treating them as one big lump called “bots” is not good enough anymore.

Cloudflare has just made this split much harder to ignore. Its new AI traffic controls let site owners manage automated visitors by behaviour: Search, Agent and Training. Search is indexing and returning references. Agent traffic is automation acting on behalf of a person in real time. Training is a crawler taking your content to improve or fine tune a model.

That distinction is the whole story.

If a search crawler indexes your service page and sends people back, that can be a fair trade. If an answer engine reads your work, gives the answer somewhere else and sends nobody, the trade has changed. If a training crawler absorbs your original content into a model, that is different again. If a customer’s agent turns up to complete a task and you block it, that might have just cost you a sale.

This is where most business websites are stuck. They were built for humans, then tuned for Google, then had a chatbot bolted onto the corner. Almost none of them were designed with a clear view of what automated systems are allowed to read, store, quote, reuse or do.

That view is about to become a normal part of the website brief. Not someday. Now.

The obvious reaction is to block everything. I understand the instinct. Nobody wants their work scraped, remixed and monetised somewhere else while they pay for the hosting and know-how. But blocking everything is blunt, and it can quietly work against you.

Small businesses still need to be found. Local firms, consultants, agencies, software companies, venues, trades, ecommerce brands, charities, publishers. They need their useful pages to show up when a customer asks an AI assistant for options. They may want search and answer systems to reference them. They might even want buyer agents to finish certain jobs for customers.

What they do not want is unlimited extraction with no attribution, no control and no commercial return.

So the real question gets sharper. Not “how do we rank in AI search” but “what do we actually want AI systems to do with this site”. Ranking is one piece. Access is the bigger one.

A workable AI access policy answers a few plain questions.

What should search systems be allowed to index? For most businesses the answer is most of the public service, product, location and educational content. If you want to be found, machines need clean, structured, current information. Clear pages, decent schema, source-backed claims, FAQs where they earn their place, and proof that survives being quoted out of context.

What should agents be allowed to do? Reading is one thing. Acting is another. An agent asking whether you serve Manchester is low risk. An agent booking a paid appointment, changing an order, requesting a refund or submitting a quote is not. Those need authentication, rate limits, confirmations, logs, and a human in the loop until you trust the pattern.

What should training systems be allowed to use? This is where you decide what is genuinely public, what is commercially valuable, and what you would rather not have absorbed into someone else’s model. There is no single right answer. A restaurant menu is not a paid research archive. A public FAQ is not your operating manual.

What content use is acceptable? Cloudflare is testing signals around immediate use, reference use and full reproduction. That is the right shape of conversation. A system briefly reading a page to answer a live question is not the same as a system storing it, summarising it, reproducing it, and never sending anyone your way.

And how will you know what actually happened? This is the boring bit, which is exactly why it matters. Logs. Bot classifications. Referral data. Tool-call records. Form submissions from automated agents. Failed attempts. Approved actions. Rejected ones. If nobody can inspect it, nobody can govern it.

This is where the agency model has to grow up.

For years the website brief has been design, copy, CMS, SEO, analytics and conversion. All still useful. None of it goes away. But it is no longer the whole job. The new brief needs a machine-access layer sitting underneath it. Which pages should be machine-readable. Which content can be referenced but not reproduced. Which automations are blocked. Which agents can act. Which actions need a login or an approval. Which data sources are authoritative. What gets logged. What can be rolled back.

That does not sound as exciting as a homepage redesign. Fine. It is more important.

The businesses that get this right will not just have nicer websites. They will have sites that can safely take part in agent-led discovery and transactions. Easier for AI systems to understand, safer for agents to interact with, and clear about where the commercial lines are.

The ones that ignore it drift into one of two bad positions. Some leave everything open and complain later when their content, offers and data are used in ways they never agreed to. Others lock everything down and then wonder why they are invisible the moment customers start using agents and answer engines as their first stop.

Neither is a strategy. It is panic in opposite directions.

For a normal business owner, the first move is not complicated. Start with an audit. List the public content that helps buyers decide, and make it accurate, structured and worth citing. List the content you do not want used for training or reproduction. Set the available controls properly, robots and content signals included. Separate low-risk information access from high-risk actions. If agents are going to submit forms, book calls, check stock or request quotes, decide what they can do on their own and what needs a human to sign off.

Then measure it, because this is not a one-off settings job. It is an operating layer. The machines visiting your site will change. Their behaviours will change. The commercial model around content, search, agents and payments will keep moving. Your rules need reviewing like any other part of the business.

At Cleo, this is the line I keep coming back to. “We need better AI SEO” is too small a brief. Yes, you need to be visible. But visibility without control is just exposure. Control without visibility is just hiding. The actual work is deciding what should be discoverable, what should be usable, what should be protected, and what should be actionable.

That is the next website conversation. Not “can people find us”. Now it is “when machines find us, what are they allowed to do”.

---

Jason Sibley is the founder of Cleo, a post-agency marketing and AI company. JasonVsTheNoise is where he writes about what is actually happening with AI, marketing, and how businesses should be thinking about both.