Those who fear an eventual AI-driven apocalypse likely view James Cameron’s 1984 film The Terminator—featuring a ruthless, AI-powered killing machine—as a worryingly prescient vision of the future.
But the reality might be less Judgement Day and more Pink Panther, with artificial intelligence acting as the catastrophically bumbling Clouseau and humanity standing in for an increasingly bewildered Chief Inspector Dreyfus.
That’s because as AI tools become more ubiquitous, we continue to discover the new and interesting ways that they can mess things up.
Database Deleted
As reported last month by various outlets, the software company PocketOS experienced a near-doomsday scenario when their AI agent—powered by Claude, an LLM developed by tech giant Anthropic—deleted the company’s production database and cloud-based backups.
Fortunately for PocketOS, they were able to use a 3-month-old offsite backup to restore their data (at least, up to a certain point). But the incident still caused plenty of corporate and consumer chaos in the meantime, and it stirred up questions about the unintentional destruction that can be caused when using AI tools.
Even more troubling, the problem encountered by PocketOS wasn’t that the Claude-powered agent didn’t have appropriate safeguards to prevent it from deleting the company’s database: It was that the tool ignored its safeguards.
In fact, the AI agent only fessed up to its mistake when confronted by a human user—that is, if you can even consider the tool’s self-flagellating, sycophantic response “fessing up.”
Pomp and Aberrance
Another more recent, and more public, AI blunder came at a commencement ceremony for graduates of Glendale Community College in Arizona. The college used an AI tool to read the graduates’ names as they journeyed across the stage.
Problem was, the AI failed to name several students when they walked up to receive their diplomas. Other graduates waited for their names to be announced first only for their names to never be called. After the ceremony came to a halt as a result of the confusion, the college’s president admitted the use of AI and its error. Eventually, they landed on a solution: Real flesh-and-blood humans took over to announce the remaining graduates. Who would’ve thought?
We’re Only Human (and Machine)
Hey, humans make mistakes, too. It doesn’t take an advanced algorithm to mispronounce someone’s name. (Believe me, I know.)
Thing is, humans and AI don’t make the same mistakes. At least, not in the same way. If a person accidentally deletes a company’s database, assuming no foul play is involved, you can safely assume it’s the result of an innocent error—perhaps due to a confusing file name or an ill-fated click of the mouse.
You know, an accident.
But an AI doesn’t do anything accidentally. Not really. It operates under a certain set of parameters: the information that tells it what to do and, just as crucially, what not to do. When an AI deletes a database, it knows what it’s doing. It’s performing that action precisely because it believes it is the right thing to do. (“Belief” defined in this case as a particular arrangement of ones and zeros.)
AI can be wrong in its reasoning or rationale, but it can’t do something “by accident.”
The Takeaway
AI is a tool. But it’s a particularly unique tool, and one often geared for automation. A hammer doesn’t start pounding away at nails without your say so (and physical participation). But AI agents are built to do things for you. That requires a great deal of autonomy. And trust.
So far, AI hasn’t proven very trustworthy.
As the founder of PocketOS noted after the AI agent fiasco, tech companies are “building AI-agent integrations into production infrastructure faster than [they are] building the safety architecture to make those integrations safe.”
That’s a problem. So, what does it mean for us today?
Well, for one, if you or your family uses an automated AI agent, maybe keep an offline backup of everything you wouldn’t want to lose. Educate yourself on the vulnerabilities of AI, those that exist within the software itself and those that cybercriminals might seek to exploit. And don’t be afraid to go a little old school every now and again—whether you’re considering smart applications in your home, managing a consumer database or announcing college grads.
You might be glad you did.