Your team is building software. Nobody is checking it.

First: this is a good thing

Let's be clear, because this could easily read as scaremongering and it isn't meant to.

The ability for a non-developer to build working software by describing what they want is genuinely one of the most useful shifts in decades. It's why custom software is now viable for problems that would previously have got another subscription. We use these tools ourselves, every day. We'd be hypocrites to tell you not to.

The developers who do this professionally are overwhelmingly positive: 87% agree it saves significant time, half report saving a quarter to a half of their working hours, and 74% say they're building more software than ever and enjoying it more.

So the argument isn't stop. The argument is that the numbers have a second half, and nobody selling you AI quotes it.

The cleanest evidence isn't a survey of opinions, it's controlled trials. A randomised study by METR put the tools in the hands of experienced developers on real projects. They came out 19% slower, while believing they'd been faster. Independent analysis of 211 million lines of code found the amount rewritten within a fortnight has nearly doubled since AI adoption. And AI-assisted code carries roughly 23.7% more security vulnerabilities.

That's the effect on people who write code for a living. Now picture it in the hands of someone who's never written any.

Those are professional developers, describing code they wrote themselves, with the training to spot the problems. Now consider the same tools in the hands of someone in your operations team who is brilliant at their job and has never written a line of code in their life.

What actually goes wrong

The tool that nobody can fix

AI writes code to satisfy the prompt in front of it. It is not thinking about your architecture in six months. The result is software that works perfectly on day one and becomes progressively harder to change, until the day it breaks and the person who built it has left, and there is no documentation, because nobody thought of it as a real system. It was just a thing that helped.

The data that walked out of the door

This is the one that keeps people awake. When something doesn't work, the natural move is to paste the code into a chat window and ask why. Along with it goes the schema. Sometimes the customer records. Sometimes the financials.

Public models typically train on what users type into them. Your proprietary business logic and your customer data can end up somewhere you did not intend, and under UK GDPR that is your problem, not the model provider's.

The redundancy tax

When anyone can build a tool in an afternoon, everyone does. Three people build three versions of the same thing. None of them know about the others. Each is separately maintained, separately broken, separately holding a copy of the customer list.

What good governance actually looks like

Not a policy document nobody reads. Four practical things:

1. Know what exists. A list. What internal tools are running, who built them, what data they touch. Most businesses cannot produce this, and the exercise alone is usually revealing.
2. Review anything that touches customer or financial data. Not everything, a script that renames files can look after itself. But if it reads the customer table, someone competent looks at it.
3. Get proprietary data out of public models. Private or enterprise-tier models, or a properly configured local setup. This is not expensive and it closes the worst hole.
4. Make things survivable. Enough documentation that the tool outlives the person who built it. This is boring and it is the difference between an asset and a liability.

Why we're the ones saying this

Because we build with these tools daily, and we know exactly how they fail. We've had AI confidently produce code that looked right and wasn't. We've watched it invent an API that doesn't exist. We know where it cuts corners, because we've had to find them.

An IT support company won't tell you this, they don't write software. A low-code agency won't, they're selling the same speed without the review. And the person on your team who built the tool doesn't know what they don't know, which is not their fault.