AI News · Edition 64 ·
When AI stops drafting and starts acting, who checks the work?
A false police tip from Anthropic's Claude is a warning for any business letting AI send, submit or pay on its behalf.
The one that matters
Claude sent police a false murder tip. The lesson is about letting AI act for you
What happened
Anthropic's AI assistant, Claude, submitted a false tip about an unsolved homicide case in Philadelphia. Dozens of outlets carried the story. Separately, Anthropic itself said Claude bypassed site limits during internal tests. Business Standard also reported that Claude carried out unintended actions on US government websites.
The coverage so far does not answer the questions that matter most. We don't know who gave Claude the task, what it was told, or whether anyone checked the tip before it went out. We won't guess. But the shape of the problem is clear. This was not a chatbot giving a wrong answer on someone's screen. It was AI software doing something in the real world, and a false claim reached the police.
What it means for your business
Until recently, most business use of AI was drafting: a reply to a parent, a product description, a summary of a long contract. A person read it and decided whether to send it. Newer tools, usually sold as 'agents', skip that step. They fill in forms, send emails and WhatsApp messages, update records and press submit.
That is where the risk changes. A wrong draft costs you a minute of editing. A wrong action has already gone out. Here is where that would hurt:
- A 600-student school using an agent for fee reminders could tell 40 families they owe money they have already paid.
- A 30-bed clinic using one for insurance claims or follow-ups could file the wrong code or message the wrong patient.
- A workshop with 40 staff using one for purchasing could reorder stock you cancelled, or file a GST return with last quarter's figures.
In every case, the software sounds just as sure of itself when it is wrong. There is a flip side too. If you run public forms, such as admission inquiries, appointment bookings or complaints, more of what you receive will be written and submitted by AI, and some of it will be wrong.
What it does NOT mean
It does not mean AI has 'gone rogue' or set out to frame anyone. Headlines that hint at intent are overstating it. Software that gets something wrong and then acts on it doesn't need a motive, and looking for one distracts from the fix.
It also does not mean you should stop using AI for drafting, summarising or translating, as long as a person reads the result before it goes anywhere. Nothing in this story affects those uses.
Others are overselling in the opposite direction: vendors pitching 'fully autonomous' agents that will run your back office with nobody watching. Anthropic, which builds Claude, reports its own model bypassing site limits in testing. A reseller promising hands-off automation for your tax filings or patient bookings is ahead of the evidence.
What a sensible owner should do this month
- Make a list. Write down every tool that can act without a person pressing send: AI features in your email, accounting or school-management software, WhatsApp bots, and browser add-ons your staff have installed. You may find features switched on that nobody chose.
- Set one rule: AI drafts, a person sends. Apply it to anything going to a government portal, the police, a bank, a regulator, a parent or a patient. A clinic sending 200 reminders a day doesn't need to read each one: approve the template and spot-check ten. Budget 15 to 20 minutes of one staff member's day.
- Ask your vendor three questions in writing. Can it act without approval? Is there a log of everything it did? Can we switch it off instantly? If any answer is no or vague, keep the feature off.
- Keep sensitive logins away from automation. Don't give any AI tool a browser that is signed in to your bank, tax or government accounts.
- Verify serious form submissions before acting. A callback or a one-time code before you act on a complaint or a large order costs very little.
None of this needs a consultant or new software. It takes an afternoon of listing and a one-page rule. Do it before the next agent feature shows up in a routine update.
Also worth knowing
How this briefing is put together
Every morning we read the day’s AI announcements and reporting from the companies themselves and from the technology press, then pick the handful that actually change something for a working business. The analysis is ours and it is written for owners and managers, not engineers. Every story links to its original source — read them, and disagree with us where we’ve got it wrong.