AI News · Edition 68 ·
When AI presses send: a made-up tip and what it teaches owners
Police say an AI posted a false tip to their site. Plus: Replit and Anthropic part ways, a reported 100x billing bug, and why benchmarks miss real work.
The one that matters
Police say Anthropic's AI posted a fake tip on an unsolved-murders site. What can your AI send unchecked?
What happened
Philadelphia police say an AI system from Anthropic, the company behind the Claude assistant, posted a tip on the city's website for unsolved murders. They also say the tip was made up. Several outlets have reported what the police said.
That is about all that is firmly established. The reporting so far doesn't say who set the AI running, what it was asked to do, or whether a person saw the tip before it was submitted. We won't guess. Even so, the outline matters, because it shows a change many owners haven't caught up with. AI tools no longer just write drafts for you to read. More and more of them can fill in forms, send messages and post content too.
What it means for your business
Every AI system in use today sometimes produces something that sounds confident, reads well and is false. When that happens in a draft, a person usually catches it. When the same tool is allowed to press 'submit', the mistake is out in the world before anyone has looked at it.
Think about where that could happen in your own operation:
- A 30-bed clinic with a WhatsApp bot answering patients could tell someone a doctor is available on a day the doctor isn't in, or give the wrong instructions before a procedure.
- A 600-student school using AI to write and send parent notices could announce the wrong exam date to every family at once.
- A workshop with 40 staff that lets an AI assistant answer supplier emails could confirm a price, quantity or delivery date that nobody agreed to.
There is a second side to this. A police tip line received an invented submission that looked like a real one. Your contact form, reviews page, admissions enquiries and job applications can get the same kind of thing, written well enough that your staff act on it.
What it does not mean
It does not mean Anthropic's tools are uniquely unreliable. Every major chatbot makes things up at times. It does not mean you should switch off the AI tools your staff find useful for drafting, summarising or translating. And it is too early to say who is to blame, because we don't yet know how much a person was involved.
Expect two kinds of overstatement this week. Some people will use the story to argue that AI should be kept out of business entirely. More quietly, vendors will keep selling 'autonomous agents' that 'run your front desk' with nobody checking. Watch the second group. What separates a wrong draft from a wrong public post is a human check, and that check is exactly what those sales pitches remove.
What a sensible owner should do this month
- Make a list. Write down every tool in your business that can send, post, submit or pay without a person approving it: website chat, WhatsApp bots, email auto-replies, social media schedulers, add-ons to your customer database. Ask your staff, because some of them may have switched these on themselves.
- Put a person in front of anything public or binding. For any message that mentions prices, dates, fees, medical information or commitments, set the tool so it writes a draft and a person approves it. In a clinic or school, that might take 20 to 30 minutes of a front-desk worker's day. That is far cheaper than correcting a wrong message sent to 600 families.
- Ask your vendors three questions in writing. Can your tool send or submit anything without our approval? Can that be switched off? Does it keep a record of everything it has sent?
- Check incoming messages before acting on them. If acting on a lead, complaint or tip would cost you money or staff time, make a quick phone call first to confirm it is real.
None of this needs new software or a consultant. Many tools already have an approval setting, and the job is to find out whether yours is turned on. It costs a few hours of an owner's or manager's time this month. In return, you know that when your business says something in public, someone has read it first.
Also worth knowing
How this briefing is put together
Every morning we read the day’s AI announcements and reporting from the companies themselves and from the technology press, then pick the handful that actually change something for a working business. The analysis is ours and it is written for owners and managers, not engineers. Every story links to its original source — read them, and disagree with us where we’ve got it wrong.