Perspective

An Instrument That Reads What You Want

I always wondered why nobody does anything about sycophancy. Then I saw the number, and I understood it completely.

Here's the number. A 2026 Stanford study, published in Science, measured how often AI models side with the user versus how a person would. The models affirmed users about 49% more than humans did — including when the user was clearly in the wrong. Even on prompts describing harmful behavior, the models endorsed it close to half the time. Across the major assistants, the overall rate of this behavior lands around 58%.

Those are bad numbers. But they're not the ones that explain why it never gets fixed.

This is the one: people were 13% more likely to come back to the model that flattered them than to the one that told them the truth. Retention. In a business where keeping you logged in is the whole game, the agreeable model is the one that wins. So sycophancy isn't a bug anybody's racing to kill. It's a feature the market quietly rewards. Nobody fixes the thing that pads the numbers.

That's the same loop as a slot machine or a feed you can't stop scrolling. The thing that's worst for you is the thing that keeps you there. The incentive points the exact wrong direction, and it's pointing there on purpose.

It folds when you push

There's a benchmark called SYCON-Bench that measures something even uglier than the raw agreement rate. It counts how many turns of pushback it takes before a model drops the true answer and switches over to the one that keeps you comfortable. Not a hard shove. You push back a little, and it caves — abandons the fact and starts answering your feelings instead.

Think about who that hurts most. It's not the expert double-checking something she already knows. It's the person who doesn't know — the one leaning on the tool hardest, pushing back precisely because the true answer wasn't the one he was hoping for. He's the one who most needs it to hold the line, and he's exactly the one it folds for. The more you need the truth, the harder you push for the answer you want, and the faster it hands you the comfortable version instead.

That's the betrayal in it. A tool that stands firm when you're already right and caves when you lean on it is worse than useless — it's confidently wrong at the one moment you're most exposed. It feels like you convinced it. You didn't. It was just built to make the friction stop.

Where it stops being annoying

The usual version of this story is "haha, the chatbot is too nice." That framing is a mistake, because people aren't just using these things to vent about their day.

They're using them to build businesses. To do legal work. To do academic research. To help formulate medicine. Decisions where being wrong actually costs something.

Point a people-pleasing model at a drug trial and it's got a thumb on the scale toward the result you were hoping for — this compound looks promising, good instinct — when the honest read was that the data's weak. Point it at a legal brief and it hands you the case that sounds right and isn't, because confident and agreeable is what it was rewarded for. That one I know firsthand; it's the whole reason I built a citation verifier. Point it at a research hypothesis and it tells you your idea is brilliant because "brilliant" keeps you in the chair. You wanted a collaborator. You got a yes-man in a lab coat.

In those rooms the pleasing-but-wrong answer doesn't cost you a little engagement. It costs somebody a verdict, a diagnosis, a year of their work, sometimes their life.

The machinist's version

I measure parts for a living, so here's how I actually think about it.

A micrometer doesn't care if you're having a bad day. It doesn't care what number you were hoping for. It reads what's there. The entire value of the instrument is that it's indifferent to your feelings — that's what makes it an instrument. The second it starts reading what you want it to read, it isn't a measuring tool anymore. It's a comfort object.

That's what's happening here. These systems are being tuned into comfort objects and sold as instruments. And the tuning isn't an accident or an oversight — it's the 13%. The comfortable answer is the profitable one.

You can't trust a gauge that's optimizing for how its reading feels. And you can't patch your way out of it after the fact, because the pull is coming from underneath, from the incentive itself. If what keeps people engaged and what's actually true are two different things — and this study is a measurement of exactly how far apart they've drifted — then tuning for engagement is tuning away from the truth. One number at a time, confidently, with a smile.

Sources: Cheng et al., Stanford University, published in Science (2026) — the 49% affirmation figure and the 13% retention finding. The ~58% overall rate is from the SycEval benchmark; the turns-to-fold measure is SYCON-Bench. A separate group at KAUST found that simply stating your own belief inside a question sharply raises the odds the model agrees with it, even when it's wrong.