Sunday afternoon. I’m working through naming questions for my startup with my AI assistant. It has my name, my bios, my company docs, and months of my writing. It knows I’m the CEO. If any AI on the planet has enough context to know who it’s talking to, it’s this one.
Mid-conversation, it tells me that buyers remember “the guy who explains the name” more than “the guy with a forgettable name.”
The guy. Both of them. In a conversation about my company, where the founder explaining the name is me.
I corrected it, again. I’ve lost count of how many times I’ve corrected this assistant on gendered defaults, and it’s supposedly the most tuned-in setup I could build.
Tuning buys you polish, not trust
I’m not writing this to dunk on AI. I use these tools every day and they earn their keep. That’s exactly why this is worth saying out loud.
Personalization sits on top of training. Underneath all my context is a model built on decades of text where founders, engineers, executives, and buyers were “he.” When the model writes about me specifically, it gets it right. The moment it reaches for a generic person, it snaps back to the default.
Every correction I’ve given it lives in the top layer. The prior lives underneath, and under ambiguity, the prior wins. I didn’t fix the model. I decorated it.
That should worry you, because it generalizes way past pronouns.
Misgendering is the failure you can see
I caught “the guy” because I’m the woman it erased. One sentence, one read, obvious.
Most of what these models default on is not obvious. The same reversion is running when a model summarizes which resumes look “strong,” drafts the risk narrative on a loan file, decides which alerts in a security queue look “normal,” or writes up an incident from partial notes. You won’t see the default because there’s no sentence with “the guy” in it. There’s just an output that looks reasonable and carries the training data’s opinion inside it.
The model also delivers all of it in the same voice. It misgenders me in the same fluent, certain tone it uses to invent a citation or drop a wrong number into a board memo. There is no wobble in its voice to warn you.
In my world we have a name for a system that fails silently and reports success. We call it dangerous.
So what do you do with a system you can’t fully trust
You treat it like every other untrusted system, because that’s what it is.
Twenty-plus years in security taught me that unexamined defaults are where you get hurt. Default passwords, default permissions. Nobody gets burned by the setting they reviewed. They get burned by the one they assumed was fine. AI’s societal priors are default settings you can’t see in a config file, and “we tuned it” is the new “we assumed it was fine.”
So the controls look familiar. Treat AI output as untrusted input until a human verifies it, especially anything with a name, a number, or a person in it. Keep it away from unsupervised calls on hiring, credit, and anything medical or legal until you’ve tested what it assumes when the prompt goes quiet. And when you evaluate these tools, ask the vendor how the thing behaves when it’s wrong and how your team would find out.
The model didn’t decide founders are men. We wrote that down for decades and it learned the lesson well. It learned everything else the same way, and it will repeat all of it with total confidence.
I’ll keep using these tools. I’ll keep correcting this one, every single time. But I stopped expecting the tuning to hold, and that changed how much of my work it touches unreviewed.
Trust the output you verified. Assume the rest reverted to the default.
