AI News · Edition 50 ·
The machine suggested it. Who signed off?
A thin but serious Pentagon admission about overreliance on AI carries a lesson every clinic, school and workshop should apply before its next rollout.
The one that matters
The Pentagon says it trusted AI too much — the same failure fits a clinic or a school
What happened, in plain words
The Pentagon has said that overreliance on artificial intelligence contributed to a missile strike that hit a school in Iran. That sentence is the substance of the report, and it is close to all of it. The account comes from a single outlet and drew a very large amount of discussion, which is not the same as being heavily corroborated. We do not know from the available material which systems were involved, what they recommended, or exactly where in the chain a person was supposed to object and did not. So we will not pretend to know.
We are leading with it anyway, because the shape of the failure is the shape of the failure that will eventually show up in a 30-bed clinic, a 600-student school or a workshop with 40 staff. Not the AI was wrong. That happens daily and is survivable. The dangerous version is nobody had been assigned to be wrong. The output arrived looking finished, it moved through a process that used to contain a pause, and the pause had quietly been removed because the tool was usually right.
What it actually means for your business
Every AI rollout in a small business removes a moment of friction, and that friction was doing work you were not tracking. A school that uses an AI tool to draft fee-defaulter lists and send notices has removed the clerk who recognised a family name and said "wait, they paid in cash last Tuesday." A clinic that runs consultation audio into an AI notes tool has removed the second reading where a dosage that looked odd got questioned. A workshop that lets a tool generate reorder quantities from last year's consumption has removed the storeman who knew one big customer left in March.
These tools are cheap — roughly ₹1,500 to ₹3,000 per user per month in India, roughly CAD 20 to 40 per user per month in Canada. The cost of the tool is never the exposure. The exposure is one wrong legal-sounding letter to a parent, one wrong dose in a chart, one wrong ₹4 lakh of stock sitting in your godown.
What it does not mean, and who is overstating it
It does not mean AI is uniquely dangerous. A wrong formula in a spreadsheet has closed more businesses than any model will this year. It does not mean you should pause your plans. Two groups will overstate this story in opposite directions. The first is the vendor selling "autonomous" agents that "run your back office" — ask them, plainly, who is accountable when the agent is confidently wrong, and watch the answer get vague. The second is the consultant who will arrive within a fortnight selling an AI risk audit or a governance package priced off this headline. You do not need either. What you need fits on one page.
What a sensible owner should do this month
- Write the one-page list: decisions where AI output may be used as a draft, and decisions where it may not. Money going out, medication, employment, legal notices and anything touching safety go in the second column.
- Put a human name on every AI-assisted output that leaves your premises. Not a department — a name. The person who signs is the person who reads.
- Before you trust a tool, run it against 20 past cases where you already know the right answer. Count the misses. If the vendor discourages this, that is the review.
- Ask each vendor one question in writing before renewal: when this is wrong, how do I find out before my customer does? File the answer.
- Keep the boring records. Thirty days of what the tool suggested and who approved it costs nothing and settles arguments.
Also worth knowing
How this briefing is put together
Every morning we read the day’s AI announcements and reporting from the companies themselves and from the technology press, then pick the handful that actually change something for a working business. The analysis is ours and it is written for owners and managers, not engineers. Every story links to its original source — read them, and disagree with us where we’ve got it wrong.