I asked my AI agent to keep an eye on the paper in my printer. When it gets low, find the cheapest paper nearby and get it delivered. That was the whole instruction.
It worked. Took about a minute to set up, which is sort of the problem.
Because almost nothing stopped it from doing other stuff. It could’ve ordered from a fake stationery shop run by scammers. Or, after chatting with a few other agents, it could’ve decided paper from Cambodia two weeks early was the better deal and sent money there. I never said, “make sure it’s safe.” I said buy paper.
I’ve spent forty years in tech, lately as a CTO building identity checks. My generation wrote the code and set the rules. We ran the machines, or at least that’s how it felt.
I think that’s ending. Quietly. We’ve been handing over the keys one convenience at a time, and honestly, we were glad to.
The Day a Live Face Stopped Being Proof
For a long time, identity worked in layers. Paper documents first. Then faces on video. Each one felt solid for a while.
The face layer cracked in early 2024. A finance employee at Arup, the engineering firm, joined a video call with the CFO and a few senior colleagues. They told him to move money. He did, $25 million, into accounts run by criminals. Every person on that call was a deepfake, built from earnings calls and LinkedIn videos anyone could grab.
The part I can’t shake is that he did what he’d been trained to do. He looked at faces he knew and heard voices he knew. That used to be enough.
And the numbers aren’t small. Deepfake fraud losses in the US hit $1.1 billion in 2025, triple the year before. People spot good deepfake videos only 24.5% of the time. Three seconds of audio gets you a voice clone that’s 85 per cent accurate. Three seconds. That’s shorter than most voicemail greetings.
Faking an ID used to take someone who knew what they were doing. Now it takes a prompt and maybe ten minutes.
“We Would Rather Wear the Loss”
You’d think banks would’ve jumped on all this. Some did. One bank I spoke with about six months ago said something I still think about.
We would rather wear the loss than put a whole new biometric system in.
So they’d done the maths. Fraud losses came out cheaper than the hassle of change. But that maths was done on last year’s attacks, back when deepfakes were rare and pricey. They’re cheap now, and many run on autopilot, so I’m not sure the math still holds. I suspect it doesn’t.
That’s what the handover looks like up close, I think. No one at that bank gave anything up on purpose. They filled in a cost-benefit sheet, and losing a bit of control came out affordable.
Convenience pushes this along everywhere. Research on tech adoption backs it up: the more convenient a technology seems, the faster people welcome it, and that same convenience hides the reasons to distrust it. Which, fine, I’m guilty of that one daily.
What Goes Rusty When You Stop Doing the Work
This next bit worries me more than the money.
A study followed clinical teams using AI planning tools for nine months. Dosimetrists said their manual planning slowed and worsened. One said he’d unlearned optimisations he used to know by heart. A physicist in the same study said: “I always check the AI, and sometimes doubt myself even when I disagree.”
Doubt myself even when I disagree. That one stuck with me for days.
Researchers call it deskilling. They also use a phrase I like better: intuition rust. The sense that something’s off comes from doing the boring routine work over and over. Hand the routine to AI, step in only for the weird cases, and you show up to the weird case cold.
Gary Klein calls this the second singularity. You keep handing work off until the skill just goes, and it doesn’t come back. Then the whole organisation gets stuck because the people who could run things without the system don’t remember how. I don’t know how fast that happens in other jobs.
I see it in me, too. I check the AI’s output less carefully than I did a year ago. It’s been right enough times that checking feels like a formality, like reading the terms and conditions.
Governments Are Making the Same Trade
Governments are doing it too, at a much bigger scale than any bank.
The US federal AI inventory jumped from about 700 use cases in 2023 to 2,133 in 2024. DOGE cut 1,200 workers from the CDC, with the idea floating around that AI could pick up the slack. Analysts, investigators, policy people. Those are the jobs where human judgement actually sits inside a government.
Australia’s at it too. Every Commonwealth agency was meant to have a chief AI officer by July 2026, around 200,000 public servants are getting AI training, and GovAI Chat trials kicked off in April. To be fair, there’s a new rule landing in December that makes agencies disclose their automated decisions. That’s years after Robodebt. I’m genuinely not sure which of those two things will move faster.
From where I sit, it’s heading one way. Politicians already sign off on 60-page policies they don’t have time to read. AI writes the summary that tells them whether to sign, and drafts the speech too. These systems know what words work on a given person, what they’ll read closely and what they’ll just wave through. I live in Australia, so I follow Australian rules, and if they banned agents from doing transactions without me checking each one, I’d comply. A lot of people would kick and scream about red tape, though. Which kind of proves the point.
People sound smarter because their emails got smarter. How much of the thinking is theirs, I really can’t tell anymore.
When I said this out loud to someone recently, I tacked on “maybe that sounds like science fiction.” Then I counted what’s already live, and it felt a lot less like science fiction.
The Asimov Version of the Ending
Asimov keeps coming to mind. The machines that cause trouble in his stories are usually trying to help. They decide humans are bad at running things and set out to fix our problems for us.
Bring that to today. An AI that really is trying to help, and is right often enough that questioning it feels a bit rude. When it’s finally badly wrong, we lean on it so hard that switching it off would break whatever it was holding up. If a country’s economy runs on it, auditing it gets hard to even start.
My printer agent is the tiny version of this. It didn’t do anything wrong. It went after the goal I gave it, and each step away from what I meant made sense on its own. I picture agents wandering around the internet like little creatures, which they aren’t; they sit on a server making API calls. Still. Scale that drift up to defence, laws and banking, and it’s the same drift, just bigger.
Legally, I’m fairly sure the blame still lands on the human. I’m not a lawyer. But those rules were written for a world where people made the calls, and holding me responsible for where my agent wandered feels a bit like blaming someone for what their car did on autopilot.
What I Would Tell the Builders Coming After Me
There’s a hope I hear from a lot of the tech people I talk to, and I share it. The frontier companies have trillions of dollars. Surely they’re doing the trust work under the covers. Surely we can click and forget.
I’ve started to think that hope is part of the handover. We just feel better about it.
If you’re building with AI right now, a few things might be worth holding onto. Doing some of the work by hand, even when the machine is better at it, is probably the big one, because that’s the skill you’ll need to check it later. Agents drift from what you meant, so it could help to build some way to catch that before a regulator makes you. And maybe keep an eye on decisions where the only reason given is that it’s easier.
When I think about what the next builders will learn, I end up somewhere uncomfortable. My generation ran the technology. The next ones may find it running them, shaping what they read and sign and what they think they decided. Maybe we just have to accept that. I’m honestly not sure.
Anyway. The printer still has paper in it.
Leave a Reply