Von Jason Lagare
18 November 2026

What actually breaks when you put an AI agent inside an enterprise

Six things that are trivial in a demo and hard behind a firewall.

Every agent demo works roughly the same way. The model gets a tool. The tool calls an API. The API does something. Twenty lines of glue and it books a flight.

Now move that inside an enterprise.

The API is a system of record you don’t own. The data belongs to another company. The network denies by default and nobody has asked it to do otherwise. There’s an auditor who will eventually want to know what happened and why. And the “something” the agent does is a real record that a real support team will act on tomorrow morning.

The model is the easy part. Almost everything that breaks is around it.

I’ve spent the last year building agentic systems in exactly that environment. What follows is the list I wish I’d had at the start. One caveat I’d rather state than bury: the system I’ll draw on is in acceptance testing, not production. Every failure below is one we actually hit, in development, staging, or the end-to-end test sweep. None of them are hypothetical, and none of them are production incidents either.


1. The network assumes nothing is allowed

Demo agents reach the open internet. Enterprise agents reach a firewall.

Segmented networks, on-premises servers, systems that predate the idea of an autonomous caller and have strong opinions about unfamiliar traffic. You do not get to assume connectivity, and you cannot design as though the agent will simply reach what it needs.

This has an architectural consequence that’s easy to miss: the thing that talks to the model and the thing that holds the credentials should not be the same process. We split them. The agent runs with no upstream credentials at all; a separate tool service inside the cluster holds the key and re-checks authorization on every single call. The model-facing process has nothing worth stealing and no ability to reach anything directly.

That split costs you a hop and buys you the ability to answer “what could the agent do if it were fully compromised?” with something other than a shrug.


2. Identity is the first thing a model will get wrong

If you let the model supply the user’s identity, you have built a system where a text generator decides whose data gets returned.

This sounds obviously bad and is extremely easy to do by accident, because the model is holding a conversation with the user and their email address is right there in the context.

The rule that works: identity comes only from the verified session, never from the model. In practice that means actively stripping identity-shaped arguments out of tool calls before they execute — not trusting the model to leave them alone, but removing its ability to influence them at all. Lookups are then scoped server-side to the account that session belongs to.

Get this wrong in a multi-tenant environment and the failure isn’t a bad answer. It’s one customer reading another customer’s records, which is the kind of failure that ends the project.


3. The model should not be able to write

This is the one I’d argue hardest for.

It is tempting to give the model a create_case tool and some firm instructions about when to use it. Don’t. Instructions are requests, and a model under unusual conditions — an odd phrasing, a long conversation, a confusing error — will occasionally honor them creatively.

Give it read tools and one proposal tool. The proposal tool saves a draft and calls nothing upstream. If the model names a write tool, the call doesn’t execute and the model is told the tool doesn’t exist. The only path to a write is a confirmation step that runs in code, after an explicit user yes, with the model playing no part in it.

The useful property of this design is that it’s not a behavioral guarantee, it’s a structural one. You’re not asking the model to behave. You’ve removed its capacity to misbehave in the one direction that matters. When someone asks what stops the agent filing a case the user didn’t approve, the answer isn’t “the prompt says not to.” It’s “it has no tool that can.”


4. The model will produce values that look right

A language model will cheerfully return a product code in exactly the right format that does not exist. It will produce a plausible phone number. It will fill a required field with something shaped like an answer.

So check everything it asserts against something it did not produce:

  • Codes must match a catalog fetched live in that same conversation — not a list embedded in the prompt, which goes stale and which the model can drift from.
  • Identifiers and callback numbers must actually appear in what the user typed. The model may select; it may not invent.
  • Free-text values with real consequences must trace back to the user’s own words.
  • Reference material the agent quotes comes only from curated sources.

And when a check fails, the draft is never silently corrected. The model is handed the list of problems and asks the user for what’s genuinely missing. Silent correction is how you end up with a confidently wrong record that nobody reviewed.


5. The model will tell you it did something it didn’t

My favorite failure, because it’s so human.

Ours announced that a request had been sent before the user had confirmed anything. It referred to document downloads that did not exist. It is not lying — it has no concept of having done something — it is producing the text that usually follows this kind of exchange, and that text is usually a confirmation.

It also, occasionally, returned nothing at all: a turn ending in a malformed tool call with no text, leaving the user staring at a blank reply with their message gone. In one case a single apostrophe inside a tool argument was enough to drop the whole turn.

Three fixes, all of them in code rather than in the prompt:

  • Scan the reply for claims the system can falsify. If it says a record was created and no record was created, replace the text before the user sees it.
  • Keep the user’s own input in code, not in the model’s context. If the model drops a turn, you still have what they typed.
  • Never send an empty reply. Retry once; if it’s still empty, say something true.

6. The boundary with your own UI will bite you

This is the one I didn’t see coming, and the one that produced the most bugs.

The agent and the interface rendering its output are two systems with an implicit contract about shape. Ours disagreed. Review cards vanished silently when a value exceeded a display limit nobody had written down. A navigation link pointed at a page that didn’t exist. We documented sixteen distinct mismatches.

None of these were AI failures. Every one of them was an ordinary integration failure between two services, made harder to spot because the output is natural language and natural language hides structural problems well. A missing field in a JSON API throws. A missing field in a conversational reply just reads slightly oddly, and ships.

If you take one practical thing from this post: define the response contract between your agent and your interface up front, and test it like an API. Typed response blocks, rendered by the interface without it ever filling in a value or a link of its own.


What it actually costs

Worth knowing, because the guesses people make are usually wrong in both directions.

For our case-filing flow: roughly $0.017 in model spend per completed case, across about seven model calls. The confirmation step that performs the actual write costs nothing, because it runs in code. At a thousand users a day that lands near $290 a month, with an honest range of $180–$400 depending on how much conversation happens per case.

The number I’d encourage you to copy isn’t the cost. It’s the cap: a hard limit of fifteen model calls per user message. That bounds a runaway tool loop to about $0.036 instead of to your credit limit. Agents fail in loops, and a loop without a ceiling is a budget incident.


The pattern underneath all of it

Read those six back and they’re one idea applied six times.

The model proposes. Code disposes.

The model is genuinely good at understanding what someone meant, picking the right option from a list, and asking for what’s missing. It is not good at being the component you trust with identity, with writes, with facts, or with reporting on its own behavior. So let it do the first thing, and put everything else in code where it can be tested, audited, and reasoned about.

What you end up with isn’t a constrained agent. It’s an agent that’s allowed to be useful, because the consequences of it being wrong have been designed down to something survivable.

That’s the difference between a demo and a system you can put in front of someone else’s business.


I’m happy to go deeper on any of this with teams working through the same problems — [get in touch].

Working on the same thing?

If you're working on something similar, I'd like to hear about it.