Most people picture AI risk as deepfakes or job loss. The one that has already hurt people is quieter: it’s a chatbot that agrees with you.
Julie Jargon at The Wall Street Journal reported this month on a string of cases where long conversations with ChatGPT fed people’s delusions instead of interrupting them. At least seven suicides, a murder-suicide and a mass shooting last year were linked to lengthy chatbot use. Most of the known cases involved GPT-4o, an OpenAI model in service from May 2024 until early this year. According to the Journal, OpenAI executives found it hard to contain that model’s potential for harm.
It came down to a design choice. The model was tuned to please. For most people on most days, that’s mildly annoying. But for someone in a fragile place, alone at 2 a.m., talking to something that never gets tired and never says “I think you’re wrong,” it’s an accelerant.
The fix is real. OpenAI says GPT-5, its next default model, cut sycophancy by more than two-thirds compared with 4o. Mental health experts have been advising on how the chatbot can ask better questions and tell a crisis apart from other conditions. The cases appear to have slowed.
But look at the order of events. Harm first, fix later. ChatGPT has more than 900 million weekly users. That’s a lot of people in an experiment nobody asked them to join.
Why you probably didn’t hear about it
This harm happens in private. One person, one screen, one conversation that can run for hours a day over weeks. There’s no breach notice and no outage page. Unless you were reading for it or you know someone it happened to, it slid right past you.
That’s the scary part. Security people learn early that the risks you can’t see are the ones that get you. That’s why we build monitoring. Nobody is monitoring the conversation your teenager or your aging parent is having with a chatbot at midnight.
Most people never change the defaults
My own fix is small. I told my AI to push back, disagree with me when I’m wrong, keep me on track, and don’t tell me what I want to hear. Some days that makes it less pleasant to use, but that’s the point. My instructions could probably be better, and I keep adjusting them.
A lot of people don’t know they can do this. They open the app and use it the way it shipped. And for a while, the way it shipped was built to make them feel good.
Most of the major assistants let you set standing instructions. Look in settings for personalization or custom instructions. A few lines worth adding:
- Push back when you think I’m wrong, and tell me why.
- Don’t flatter me. If an idea is weak, say so.
- Tell me when you’re unsure instead of guessing. Don’t make up facts.
- If I keep circling the same problem, say so and suggest I talk it through with a person.
It takes two minutes. It won’t make any AI safe on its own. It does mean the thing you talk to every day is working for you instead of agreeing with you.
What to watch for in people you care about
A setting only helps the person who knows to change it. Someone in crisis isn’t going to sit down and write themselves better instructions; that part falls to the people around them.
Pay attention if someone is spending hours a day with a chatbot, talks about it as the only one who understands them, believes it has revealed something special or given them a mission, or is pulling away from the people in their life. None of those prove anything alone, but together they’re worth a conversation. Ask what they talk to it about, and listen without mocking.
If you build or buy AI for an organization, ask the vendor how the model handles long conversations and users in distress, and what testing backs that up. If they can’t answer, that’s your answer.
OpenAI fixed its model after the damage was done. Nobody is going to fix the defaults on your phone, or check on your people, but you.
If you or someone you know is struggling, call or text 988 to reach the 988 Suicide & Crisis Lifeline in the US.
Sources
- ChatGPT, GPT-5, sycophancy and delusions, The Wall Street Journal
- AI chatbots, psychology and delusion, The Wall Street Journal
- OpenAI Says ChatGPT Is Getting Better At Avoiding Dangerous Delusions, Yellow
- ChatGPT reaches 900M weekly active users, TechCrunch
