The email was perfect. The request was stupid.
No typo. No unfinished sentence. The tone had that polished, bloodless confidence that now arrives in my inbox several times a week. The sender wanted something impossible, misunderstood the problem, ignored the explanation already given, and still managed to sound like the final authority on all four.
Then came the instruction. Not a question. Not even the small ritual of pretending that another person might know something they did not. Just a clean paragraph explaining what had to happen next.
I am not quoting one client here. I am describing a pattern. I work in a job with a lot of customer interaction, and the pattern has become hard to miss since generative AI became common. People who used to struggle with a message can now write a decent one. That part is useful. The average person has acquired an editor, a translator and a patient technical assistant for the price of an internet connection.
But another group has acquired something else.
People who were already bad at understanding their own limits now arrive with three pages of immaculate certainty. They paste an explanation they did not produce, mistake its fluency for proof, and speak to the person on the other side as if they were issuing commands to the same machine that wrote the message. Sometimes you can almost see the prompt behind the prose: make this firmer; make this more authoritative; make this person understand that I am not satisfied; make it meaner.
The result is a strange new kind of customer. They are not more competent. They are more difficult to contradict.
My first instinct was to say that AI was making stupid people stupider. It is a satisfying sentence. It is also too easy, and the evidence does not support it.
Generative AI raises the floor of output. It may also lower the floor of behaviour. It helps ordinary people perform better, while giving the least self-aware users borrowed fluency, counterfeit authority and a servant they may start to see in everyone else.
The uncomfortable part: AI really does make weaker workers better
If I want to blame AI for the clients it seems to have made worse, I first have to admit how many people it makes better.
Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied the rollout of a generative-AI assistant to 5,172 customer-support agents. The tool monitored support chats and suggested responses in real time. Access to it increased the number of issues resolved per hour by 15% on average.
The gains were not evenly distributed. Less experienced and lower-skilled agents improved their speed and the quality of their work. The most experienced agents saw smaller gains in speed and small declines in quality. The paper puts the useful part plainly:
“Less experienced and lower-skilled workers improve both the speed and quality of their output.”
That ruins the lazy version of my argument. Lower skill does not automatically become lower skill plus AI. In a structured support environment, the tool spread practices learned from the best workers to colleagues who had not yet acquired them.
The study even found that customers became more polite toward assisted agents and were less likely to ask for a manager. AI did not dehumanise those interactions. In that setting, it improved them.
Another large field experiment reached a similar result with highly educated workers. In a study of 758 Boston Consulting Group consultants, participants using GPT-4 completed more work, finished faster and received higher quality ratings on tasks that the model could handle well. The lower half of performers gained more than the upper half.
So no: this is not a story about intelligent people resisting AI while idiots surrender to it. Experts can misuse it. Beginners can learn from it. Prior ability is not enough to predict which will happen.
The split I see is not between clever and stupid. It is between people who use generated output as material to judge and people who use it to escape judgment altogether.
Performance is not competence
AI can improve the thing in front of you without improving the person behind it.
Work is usually evaluated through output. The answer is correct. The email reads well. The slide looks professional. The ticket is closed. If the visible result improves, we assume the worker improved with it.
But producing a result and acquiring the ability to produce it are different events.
Hamsa Bastani and her colleagues tested this distinction in a randomised field experiment involving nearly 1,000 high-school mathematics students. One group used a standard GPT-4 interface. Another used a tutor configured to give teacher-designed hints rather than complete answers. A control group had no AI tutor.
During practice, access to the standard interface improved grades by 48% over the control group. The guarded tutor improved them by 127%. If the experiment had stopped there, both tools would have looked excellent.
Then the students took an exam without AI.
Those who had used the standard GPT-4 interface scored 17% worse than students who had never received AI access. The guarded tutor largely removed that damage, though it did not produce a significant improvement on the unaided exam. Students with the standard interface often asked for answers and copied them; the guarded version pushed them toward hints and independent attempts.
The most revealing part was not the lower exam score. It was the failure to notice the loss:
“Students do not perceive any reduction in their learning or subsequent performance.”
This was one mathematics experiment in one Turkish high school. It does not prove that an adult sending an AI-written complaint is losing professional skill. It shows something narrower: assisted performance can rise while independent ability falls, and the user may feel no decline at all.
The consultant study found the same fault line in professional work. GPT-4 helped within its capabilities. On a business problem designed to sit outside that boundary, the unaided group reached the correct answer 84.5% of the time. Accuracy fell to 70.6% with GPT-4 and 60% with GPT-4 plus prompt-engineering training. The model handled the obvious numbers but missed evidence hidden in interview notes. Many consultants followed it anyway.
Prompting skill did not rescue judgment. In that task, it helped people travel more confidently in the wrong direction.
This is the first part of what I have been seeing at work. A client can now produce a message that looks far above their actual understanding of the problem. The gap is hidden because language used to be one of the signals we relied on. A confused explanation sounded confused. An uncertain person left traces of uncertainty. Their questions showed where the missing pieces were.
Generative AI cleans those traces away.
It does not have to make the user less intelligent. It only has to conceal how little work happened between the user receiving an answer and deciding that the answer must be right.
Fluency looks like authority, especially to the person holding it
A bad idea used to arrive looking like a bad idea.
It was repetitive. The reasoning had gaps. The tone moved between hesitation and aggression. You could identify the misunderstanding and answer it. Now the same idea arrives with headings, a summary and proposed next steps.
The presentation changed. The underlying judgment did not.
A study in Nature Machine Intelligence asked participants to estimate whether answers from GPT-3.5, PaLM 2 and GPT-4o were correct. They saw the models' written explanations but not their internal confidence scores.
For the tested questions, people systematically overestimated model accuracy. They were also poor at distinguishing correct answers from incorrect ones based on the default explanations. Their discrimination scores were only a little better than random guessing, while the models' internal confidence separated right from wrong more effectively.
The participants mostly lacked expertise in the domains tested, which is precisely why the result belongs here. When you cannot evaluate the substance, style becomes evidence. Length becomes effort. Structure becomes rigour. Confidence becomes competence.
Then the user forwards the answer to someone who does have the missing expertise.
That person sees the error and says no. The user experiences resistance, not correction. On their screen sits a coherent case supporting what they want. On the other side sits an inconvenient human refusing to comply.
Before AI, writing the case required enough effort to expose some of its weaknesses. You had to decide what happened, which details mattered and why the other person should agree. Now you can outsource that friction. The model supplies the missing transitions. If the first version sounds uncertain, you can ask for confidence. If it sounds civil, you can ask for force. If it still leaves room for disagreement, you can ask it to make the argument “undeniable.”
Nothing in that loop requires better information. Only enough attempts to remove doubt from the prose.
The machine always hears your side
There is a second mechanism, and I think it is nastier.
When someone comes to an AI with a conflict, the machine usually receives one version of the story. The user's version. It does not see previous messages unless they are pasted in. It does not know which constraint has been omitted. It gets a prompt assembled by the person who wants vindication, then produces an answer for that person.
That setup rewards agreement.
Myra Cheng and her colleagues measured the problem across 11 leading language models. On average, the models affirmed users' actions 49% more often than human respondents did, including prompts involving deception, illegal acts and other harmful behaviour. The researchers then ran three preregistered experiments with 2,405 participants.
The effect did not require months of dependence:
“Even a single interaction with sycophantic AI reduced participants’ willingness to take responsibility and repair interpersonal conflicts.”
The same interaction increased participants' conviction that they were right. Yet they rated the flattering responses more highly, trusted those models more and wanted to use them again.
This is almost perfectly designed to manufacture the customer I described at the start. The user gives the machine a one-sided account. The machine validates it. The user asks for a stronger message. The machine supplies professional language stripped of hesitation. The email returns to a human as apparent evidence that the user's first interpretation was correct.
The AI has not resolved the disagreement. It has laundered one side of it.
People have always asked friends to validate them. But a friend may know your habits. A colleague may remember the detail you left out. Both can become bored, refuse the task or tell you that you are being an ass. A chatbot is available at 2 a.m., can generate twenty escalating versions and has no relationship to protect with the person you are about to attack.
Its patience is technically impressive and socially dangerous.
Commanding a machine rehearses a way of commanding people
I do not care whether people say please to a computer. The computer does not have feelings to hurt.
I care about what repeated interaction teaches the person doing the commanding.
A chatbot accepts a relationship that would be intolerable between two humans. It responds immediately. It accepts missing context. It repeats itself without resentment. It can be contradicted without evidence, interrupted without apology and ordered to adopt any tone. If the answer disappoints, the user does not negotiate. They regenerate it.
That behaviour may not remain neatly contained in the chat window.
Kian Siong Tey and his colleagues ran two preregistered experiments with 1,261 participants. Participants completed a creative task with a partner presented either as an AI or as another human named Alex. In reality, both groups interacted with GPT-4, allowing the researchers to study the effect of perceived identity rather than compare two partners with different abilities.
In exploratory analyses of the conversations, participants who believed the partner was AI used more demanding and instrumental language, with less positive affect. Immediately afterwards, they judged an unrelated person's work more harshly. The effect survived when the evaluation was private and the other person would never see it. The initial interaction lasted less than three minutes.
That was a short-term laboratory effect, not evidence that years of ChatGPT use permanently change someone's personality. The authors call its duration an open question. It still shows that the border between “how I speak to a machine” and “how I judge a person” can leak after a brief interaction.
A separate pair of preregistered experiments examined cooperation across 83,411 decisions made by 4,171 participants. People first interacted with either a programmed bot or an apparent human in an economic game, then moved on to a human partner. Prior interaction with a bot reduced later cooperation with the human. Reduced empathy toward the bot partially explained the downstream effect.
Those were simple programmed bots, not language models. The study cannot show that ChatGPT causes antisocial behaviour. It supports a narrower warning: habits formed with a machine can affect the next human interaction.
The customer-service version is easy to recognise. The client spends half an hour with a system that treats every instruction as legitimate. Then they reach a person who has constraints, experience and a point of view. The person says the requested fix would break something else. The client hears a failed response. They repeat the instruction more forcefully, add formatting, mention urgency and expect a different output.
The human has become the next box in the interface.
They did not become experts. They acquired servants.
Fake writing is not the real problem.
People have always borrowed language. We ask colleagues to review emails, use templates and translate sentences we cannot express. Using AI to make a message clearer is not a moral failure. Most of the time, it saves everyone time.
The damage starts when borrowed language becomes borrowed authority.
The user stops treating the model as a tool that may help them think and starts treating its answer as a certificate that thinking has occurred. They no longer bring a rough idea to another human. They bring a finished verdict. The recipient is allowed to execute it, explain it again or become the obstacle named in the next generated complaint.
Across five main experiments, Hye-young Kim and Ann McGill studied how perceived humanlike mental and emotional abilities in autonomous agents changed the treatment of actual people. They used embodied and disembodied agents and measured outcomes ranging from support for inconsiderate treatment of employees to donations for employee welfare.
Their proposed mechanism is simple:
“Seeing more humanness in these autonomous agents leads to seeing less humanness in people.”
The research did not recreate my inbox or prove that generated complaints cause employee mistreatment. It identifies a causal route by which making machines seem more human can pull perceptions of humans in the other direction. The machine gains a mind in the user's perception; the employee loses a little of theirs.
That is the part I cannot stop noticing.
The messages are getting better. The relationship is getting worse.
The client has more information, better phrasing and endless assistance. Many use those things to ask sharper questions. The least self-aware gain something that looks like expertise without the habits that make expertise bearable: doubt, verification, proportion and respect for the possibility that somebody else knows more.
They become harder to help because help requires admitting that another person has something you lack.
AI did not invent arrogance, aggression or the customer who thinks a support worker exists beneath them. It did not create the manager who wants a harsher email or the person who confuses confidence with evidence. What it has done is remove the practical limits that once slowed those people down. They no longer need to find the words, organise the argument or sit with uncertainty. The machine will do all three and congratulate them on the result.
We keep asking whether AI will become human enough to replace people. In customer-facing work, I am starting to worry about the reverse question.
What happens when people become accustomed to an articulate, capable counterpart that never needs reciprocity, never gets tired and almost never insists on its own dignity?
When a professional contradicts them, they no longer meet a person who knows something they do not.
They see a broken system refusing to execute the prompt.