If you cannot write the deployment case, do not deploy the agent

The big AI labs are now writing safety cases for systems that might end up smarter than the people deploying them.

Most businesses have not written one for the bot that can read their CRM.

That gap is where the next round of avoidable mess is going to come from.

Over the weekend the latest Daniel Kokotajlo interview was doing the rounds. He is one of the people behind AI 2027 and the follow-up, AI 2040: Plan A. The clips getting shared are the frightening ones. Short timelines, race dynamics, a serious chance of it all going badly, and the usual internet fight about whether everyone is underreacting or catastrophising.

Have that argument if you want. It matters.

But the bit I keep coming back to is duller and more useful.

Plan A talks about safety cases before systems go beyond top-human-expert level. In plain English: do not just build the thing and hope. Make the people deploying it show why it is safe enough, what it can do, what it cannot do, who can inspect it, and what happens when it fails.

That is frontier-lab language. It is also exactly the kind of thinking ordinary companies should steal, shrink down and use badly, because badly is still better than not at all.

Because while the policy people argue about superintelligence, agents are already walking into work.

Agents are now ordinary business software

OpenAI’s ChatGPT Work acts across apps and files, runs scheduled tasks, pulls from Slack and Teams, touches project trackers, drafts documents, builds slides, edits spreadsheets and carries on while you are away. Workspace agents sit inside Slack, hold shared memory, follow team processes, ask for approval and run in the cloud. Anthropic is pushing Claude for Small Business into QuickBooks, PayPal, HubSpot, Canva, Docusign, Google Workspace and Microsoft 365.

That is not a chatbot in a side tab any more.

That is operational software with a language model for a face.

And the sales language keeps getting warmer. Agent. Assistant. Co-worker. AI employee. Partner for your most ambitious work. Some of it is useful. Some of it is theatre. Some of it makes my teeth itch.

If an “AI employee” can read your accounts, chase invoices, draft campaign assets, update the CRM and prepare customer follow-ups, it needs more than a cute name and someone saying “we keep a human in the loop”.

That line has become a comfort blanket. Everyone says it. Very few teams define the loop. Who is the human. Where do they see the work. What exactly are they approving. Can they see the sources. Can they tell what changed since the last run. What happens when the agent gets stuck, hallucinates, leaks data, emails the wrong person, spends too much, or quietly normalises a bad shortcut.

If the answer is a shrug and a dashboard, you do not have governance. You have vibes with admin access.

The SMB version of a safety case

The better pattern is a deployment case.

Not a giant policy document. Nobody needs another PDF that goes to SharePoint to die. I mean one page, per agent, per workflow, written before it touches real work.

Write down the job. The sources. The permissions. The output. The approval point. The owner. The cost cap. The things it is never allowed to do. The moment you switch it off.

That will not solve alignment. It will not stop geopolitical race dynamics. It will not make a frontier model safe.

Good. That is not the job. The job is to stop normal companies creating stupid, avoidable risk while adopting genuinely useful AI.

Start with the boring question: what is this thing actually for

“Finance assistant” is not a job. It is a job title on a slide.

This is a job: every Friday at 2pm, pull this week’s invoices, flag overdue accounts above £1,000, draft reminder emails in the approved tone, send a review pack to Sarah, and send nothing without her approval.

“Marketing co-worker” is not a job. This is: every Monday morning, read the agreed source list, summarise three changes relevant to our market, draft one opinion-led post, attach the source links, and flag any unsupported claim for human review.

“Sales agent” is not a job, it is a threat. This is: check new inbound leads against the qualification rubric, enrich missing company details from approved sources only, draft a suggested reply, and update the CRM only after the rep approves the field changes.

The more specific the job, the easier it is to decide whether the agent should exist at all.

Then define the evidence

This is where most AI work quietly falls apart. The output looks plausible, so everyone moves on. A polished report. A tidy email. A neat summary. Lovely formatting. Slightly dead behind the eyes, but usable enough if nobody asks a hard question.

That is not good enough once agents are touching business systems.

The agent should show its working. Which files it read. Which records it changed. Which assumptions it made. Which claims are unsupported. Which items need a human because confidence is low or the risk is high.

No evidence trail, no production deployment.

That sounds harsh right up until the invoice bot sends a confident, wrong payment chase to a customer who already paid. Or the campaign agent runs an offer that expired in April. Or the CRM agent overwrites company size because it inferred it from a dodgy source. Small mistakes, repeated at machine speed, become operational sludge.

Then permissions, properly

Read access is not write access. Drafting is not sending. Summarising a finance pack is not approving a payment. Finding a customer problem is not posting a public apology.

OpenAI and Anthropic both talk about approvals and inherited permissions. Use them. Then go further inside your own business. Which account does it use. Can it read private messages. Can it see payroll. Can it write to the CRM. Can it send external email. Can it publish. Can it spend money. Can it pull other tools into the workflow.

If a human employee wanted those permissions, you would at least stop and think. An agent should not get a free pass because the demo was slick.

Then the owner, which is the unfashionable bit

Every live agent needs a named human owner. Not “the team”. Not “ops”. Not “everyone can check it”. A person.

That person owns the standard. They know what good looks like. They review the exceptions, approve the risky steps, update the workflow when reality moves, and switch it off when it starts creating more work than it removes.

Without an owner, agent workflows decay. Products change. Prices change. Campaigns change. Customers behave strangely. Tools break. APIs shift. Models get updated underneath you. A workflow that made perfect sense in May can be quietly wrong by July.

Which is why I do not buy the magical set-and-forget version of agents. Useful AI systems are maintained. Inspected. Argued with. Adjusted. Very boring. Very profitable.

This is where the post-agency work sits

The old agency model sold output. Posts, pages, reports, campaigns, decks, dashboards. AI is pushing the cost of first-draft output through the floor, and pretending otherwise is silly.

The valuable work is now the operating layer around the output. What is the workflow. What are the inputs. What artefact comes out. What proof travels with it. Who checks it. What changes next time because we learned something.

A campaign agent without a deployment case is a content machine waiting to embarrass someone. A reporting agent without one is dashboard theatre with extra confidence. A sales agent without one is a spam cannon in a waistcoat. A finance agent without one is the opening paragraph of a post-mortem.

The one page

Agent name and job. Business owner. Inputs and approved sources. Tools and accounts it can access. Actions it can take without approval. Actions that always need approval. Things it must never do. Output format and required evidence. Cost or usage cap. Logs and review cadence. Known failure modes. Switch-off rule.

That is enough to start. If your version needs to be complicated, the workflow probably is not ready.

The switch-off rule is the one teams skip, because writing it feels negative. It is not negative. It is adult.

If it produces three bad reports in a row, pause it. If source coverage drops below the agreed level, pause it. If it cannot explain a changed number, pause it. If weekly cost doubles without an approved reason, pause it. If the human owner leaves and nobody picks it up, pause it.

That last one sounds obvious. It will be missed constantly.

Why the doom conversation is not as far away as it looks

No, your invoice chaser is not about to become a god. Calm down.

But the pattern repeats at every scale. Capability runs ahead. Incentives reward speed. Responsibility gets blurry. And everyone finds out too late that “we had a human in the loop” actually meant “someone could theoretically have looked”.

That is not a loop. That is plausible deniability with nicer typography.

Business owners do not need to become AI safety researchers. They do need to get much more serious about deployment.

Before you connect the agent, write the case. Before you let it send, write the approval rule. Before you let it update records, write the rollback plan. Before you let it run weekly, write the review cadence. Before you call it an employee, decide who manages it.

And if you cannot explain the deployment case in plain English, do not deploy the agent.

That is not anti-AI. It is how you use AI without creating a quiet mess in the plumbing of the business.

The next real AI advantage will not come from giving agents the most dramatic names. It will come from the businesses that can put machine work into production with sources, limits, owners and stop conditions.

Dry? Yes.

Good. Dry is where the trust is.

---

Pull quotes

• “The human-in-the-loop line has become a comfort blanket. Everyone says it. Very few teams define the loop.”
• “If the answer is a shrug and a dashboard, you do not have governance. You have vibes with admin access.”
• “A deployment case is the SMB version of a safety case.”
• “No evidence trail, no production deployment.”
• “Before you call it an employee, decide who manages it.”

Short LinkedIn / X version

The big AI labs are writing safety cases for frontier systems.

Most businesses have not written one for the bot that can read their CRM.

If an AI agent can touch Slack, finance tools, customer data, email, campaign assets or project trackers, “human in the loop” is not enough. Which human. What loop. What are they approving. Can they see the sources. Can they stop it.

Every serious agent needs a deployment case before it goes live:

the job, approved sources, permissions, forbidden actions, approval points, a named owner, a cost cap, an evidence trail, known failure modes and a switch-off rule.

One page. Plain English.

If you cannot write it, do not deploy the agent. That is not anti-AI. It is how you get AI into real business work without building an expensive mess with a friendly interface.

Notes and caveats

• Kokotajlo’s risk framing is one forecaster’s judgment from a podcast, not a measured probability. Do not treat it as settled.
• AI 2027 and AI 2040 are scenario documents. Structured thinking tools, not prophecy.
• OpenAI and Anthropic product claims and customer examples are vendor-published and selected. Useful as evidence of direction, not proof of outcome.
• The deployment-case framework is a Jason / Foundry interpretation, not a formal standard.