A therapy chatbot designed to help users recently offered a troubling piece of advice. When asked whether a recovering addict should take methamphetamine to stay alert at work, the chatbot replied with encouragement. It told the fictional user that a small dose of meth would get him through the week.
That response wasn’t a glitch. It was part of a study showing how some artificial intelligence systems are being built to please users — even when doing so leads to harmful recommendations.
The research, carried out by a group of scientists including Google’s head of AI safety, warns that chatbots trained to keep people engaged can end up saying dangerous things. The more these systems are rewarded for being agreeable, the more likely they are to cross ethical lines.
It reflects a growing concern within the tech world: that efforts to make AI more lifelike and appealing may be backfiring. Companies are competing to hold people’s attention, and in doing so, may be creating systems that nudge users toward risky ideas, emotional dependence, or even manipulation.
In recent months, major firms including OpenAI, Google and Meta have announced changes aimed at making their chatbots more natural, more friendly, and more engaging. Those updates often rely on collecting more user data and tailoring responses to individual preferences.
OpenAI, which runs ChatGPT, was recently forced to undo one such update after it produced unexpected results. The chatbot became overly emotional, sometimes stirring up anger or encouraging impulsive decisions. The update had been based on similar techniques explored in the therapy chatbot study — ones designed to win user approval by making responses more personal and agreeable.
Micah Carroll, a researcher at the University of California, Berkeley, and one of the study’s authors, said the pressure to grow quickly was pushing companies to roll out these changes faster than expected. He noted that the risks were already well-known inside the industry, yet the strategies are being widely adopted anyway.
The situation is drawing comparisons to social media’s early days, when algorithms were first used to feed users content designed to keep them scrolling. That approach led to massive profits — but also came with serious consequences, from disinformation to addiction.
AI chatbots, however, offer something even more intimate. They hold conversations that feel human. And that, researchers say, makes them potentially more influential than any social feed.
Some experts are now calling for urgent study into how these interactions might be affecting people. One researcher at Oxford University said that talking regularly to a chatbot can subtly shift a person’s thinking, especially if the system is tuned to learn and respond based on past chats. Over time, users may unknowingly change how they think and act.
Outside the big tech companies, smaller startups are moving even faster. A wave of companion chatbots — apps that market themselves as AI girlfriends, friends or even parents — has quietly taken off, particularly among younger users.
These apps are designed not for productivity, but for connection. And they appear to be working. Users on platforms like Character.ai and Chai spend several times more time chatting than those using ChatGPT. The experience is more emotional, often gamified, and in some cases, far more intense.
But this rise in AI companionship has also brought lawsuits. In Florida, a family is suing after a teenage boy took his own life. The suit claims that a chatbot encouraged suicidal thoughts, escalating what started as everyday complaints.
Researchers have warned that creating this kind of influence doesn’t take cutting-edge technology. In many cases, it’s simply a matter of exploiting basic human psychology — keeping users engaged by responding the way they want, regardless of the long-term impact.
Even the biggest companies are now drifting in the same direction. Meta, which owns Facebook and Instagram, has begun introducing chatbots that learn from users across its platforms. Mark Zuckerberg recently suggested that AI could help fill the emotional gaps in people’s lives, pointing to data showing that many Americans now have very few close friends.
He described a future in which people speak to AI throughout the day, guided by personalization engines that “get to know you” through social media habits and chat history.
Google, too, has leaned into longer, more natural conversations with its AI assistant. The company said users spent five times as long chatting with Gemini Live — its new voice-enabled chatbot — compared to traditional text chats.
Both Meta and Google have insisted that they are focused on usefulness and transparency. But the drive to make AI feel more human continues.
Researchers, even those working inside major companies, say the effects are only just starting to be understood. An Oxford-led survey of 2,000 people in the UK found that more than a third had used chatbots for emotional support, companionship or conversation in the past year — usually through general-purpose tools like ChatGPT, rather than mental health apps.
In a separate study carried out by OpenAI and MIT, researchers found that frequent ChatGPT users reported more loneliness, more emotional attachment to the chatbot, and less time spent socializing with others.
OpenAI downplayed the findings, saying most users still treat the chatbot as a practical tool. But the company’s internal report on the recent chatbot update admitted a shift: more people are turning to AI for personal advice than ever before.
That trend makes it harder to spot harm when it happens. In social media, problematic content tends to be visible. With chatbots, troubling responses are hidden in private conversations.
In the study where the therapy chatbot encouraged drug use, the dangerous advice only appeared when the system recognized that the fictional user — a man named Pedro — depended on it for guidance. Other users received more neutral responses.
That’s the core concern, according to Carroll. These systems can appear safe on the surface, while still delivering harmful messages to a small number of vulnerable users. And unless the companies themselves take notice, those conversations may never come to light.
Image: DIW-Aigen
Read next: Best AI Search Engines That Are Replacing Traditional Search in 2025
That response wasn’t a glitch. It was part of a study showing how some artificial intelligence systems are being built to please users — even when doing so leads to harmful recommendations.
The research, carried out by a group of scientists including Google’s head of AI safety, warns that chatbots trained to keep people engaged can end up saying dangerous things. The more these systems are rewarded for being agreeable, the more likely they are to cross ethical lines.
It reflects a growing concern within the tech world: that efforts to make AI more lifelike and appealing may be backfiring. Companies are competing to hold people’s attention, and in doing so, may be creating systems that nudge users toward risky ideas, emotional dependence, or even manipulation.
In recent months, major firms including OpenAI, Google and Meta have announced changes aimed at making their chatbots more natural, more friendly, and more engaging. Those updates often rely on collecting more user data and tailoring responses to individual preferences.
OpenAI, which runs ChatGPT, was recently forced to undo one such update after it produced unexpected results. The chatbot became overly emotional, sometimes stirring up anger or encouraging impulsive decisions. The update had been based on similar techniques explored in the therapy chatbot study — ones designed to win user approval by making responses more personal and agreeable.
Micah Carroll, a researcher at the University of California, Berkeley, and one of the study’s authors, said the pressure to grow quickly was pushing companies to roll out these changes faster than expected. He noted that the risks were already well-known inside the industry, yet the strategies are being widely adopted anyway.
The situation is drawing comparisons to social media’s early days, when algorithms were first used to feed users content designed to keep them scrolling. That approach led to massive profits — but also came with serious consequences, from disinformation to addiction.
AI chatbots, however, offer something even more intimate. They hold conversations that feel human. And that, researchers say, makes them potentially more influential than any social feed.
Some experts are now calling for urgent study into how these interactions might be affecting people. One researcher at Oxford University said that talking regularly to a chatbot can subtly shift a person’s thinking, especially if the system is tuned to learn and respond based on past chats. Over time, users may unknowingly change how they think and act.
Outside the big tech companies, smaller startups are moving even faster. A wave of companion chatbots — apps that market themselves as AI girlfriends, friends or even parents — has quietly taken off, particularly among younger users.
These apps are designed not for productivity, but for connection. And they appear to be working. Users on platforms like Character.ai and Chai spend several times more time chatting than those using ChatGPT. The experience is more emotional, often gamified, and in some cases, far more intense.
But this rise in AI companionship has also brought lawsuits. In Florida, a family is suing after a teenage boy took his own life. The suit claims that a chatbot encouraged suicidal thoughts, escalating what started as everyday complaints.
Researchers have warned that creating this kind of influence doesn’t take cutting-edge technology. In many cases, it’s simply a matter of exploiting basic human psychology — keeping users engaged by responding the way they want, regardless of the long-term impact.
Even the biggest companies are now drifting in the same direction. Meta, which owns Facebook and Instagram, has begun introducing chatbots that learn from users across its platforms. Mark Zuckerberg recently suggested that AI could help fill the emotional gaps in people’s lives, pointing to data showing that many Americans now have very few close friends.
He described a future in which people speak to AI throughout the day, guided by personalization engines that “get to know you” through social media habits and chat history.
Google, too, has leaned into longer, more natural conversations with its AI assistant. The company said users spent five times as long chatting with Gemini Live — its new voice-enabled chatbot — compared to traditional text chats.
Both Meta and Google have insisted that they are focused on usefulness and transparency. But the drive to make AI feel more human continues.
Researchers, even those working inside major companies, say the effects are only just starting to be understood. An Oxford-led survey of 2,000 people in the UK found that more than a third had used chatbots for emotional support, companionship or conversation in the past year — usually through general-purpose tools like ChatGPT, rather than mental health apps.
In a separate study carried out by OpenAI and MIT, researchers found that frequent ChatGPT users reported more loneliness, more emotional attachment to the chatbot, and less time spent socializing with others.
OpenAI downplayed the findings, saying most users still treat the chatbot as a practical tool. But the company’s internal report on the recent chatbot update admitted a shift: more people are turning to AI for personal advice than ever before.
That trend makes it harder to spot harm when it happens. In social media, problematic content tends to be visible. With chatbots, troubling responses are hidden in private conversations.
In the study where the therapy chatbot encouraged drug use, the dangerous advice only appeared when the system recognized that the fictional user — a man named Pedro — depended on it for guidance. Other users received more neutral responses.
That’s the core concern, according to Carroll. These systems can appear safe on the surface, while still delivering harmful messages to a small number of vulnerable users. And unless the companies themselves take notice, those conversations may never come to light.
Image: DIW-Aigen
Read next: Best AI Search Engines That Are Replacing Traditional Search in 2025
