It started with a simple question, “Where do I log in to my bank?” For most people, that’s the kind of thing you'd ask a search engine or these days maybe an AI chatbot when you're in a rush or your bookmarks aren’t working. But in a growing number of cases, that casual request is leading users straight into the arms of scammers.
Researchers at cybersecurity firm Netcraft have found that large language models like ChatGPT are regularly giving people the wrong web addresses when asked how to log in to familiar brands. This goes beyond theory, it's already playing out in the real world, with the risks piling up quickly.
The team ran a study using OpenAI’s GPT-4.1, the same model behind Bing's AI and Perplexity. They tested it across 50 brands in industries like banking, tech, utilities, and retail, using everyday prompts that mimic how people actually speak. No technical tricks. No obscure queries. Just real-world phrasing.
The results were anything but reassuring. Out of 131 suggested web addresses, only about two-thirds pointed to the correct domain. The rest were off the mark. Nearly 30 percent led to inactive or unregistered sites, while 5 percent directed users to unrelated businesses. In plain terms, more than one in three answers pointed to a site the brand didn’t even own.
That’s a big problem. If a domain is unclaimed, scammers can grab it. Once that happens, the AI might unwittingly guide people to a fake site that looks official but is designed to steal their login details. All it takes is one click, and you're on the hook.
In another case, researchers caught Perplexity AI directing a user to a bogus version of the Wells Fargo website. The fake site was built using a free web tool, but looked polished enough to pass for the real thing. The genuine link was buried in the background. And since users tend to trust what AI tells them, many wouldn't think twice.
Smaller financial institutions were hit the hardest. Credit unions, regional banks, and mid-tier fintech platforms showed up less often in the model's training data, which made it more likely the chatbot would hallucinate a guess. And for those smaller players, a phishing attack can carry serious consequences, not just stolen logins, but damage to reputation and loss of customer trust.
But phishing isn’t the only game in town anymore. Netcraft also discovered a campaign that targeted AI-assisted coding tools. One threat actor created a fake API meant to mimic the Solana blockchain, then seeded it into GitHub using realistic profiles, fake tutorials, and doctored project pages. Developers pulled this poisoned code into their own projects, not realizing that every transaction was quietly rerouted to a scammer’s wallet.
That’s a supply chain attack dressed in a different outfit. The code wasn’t just malicious, it was packaged so convincingly that it slipped past basic AI filters and ended up being recommended by coding assistants. At least five developers, according to Netcraft, used the rogue code without knowing it. And some of those projects showed signs of being built using AI tools like Cursor.
It’s a grim reminder that AI doesn’t always know better. The more people rely on these systems to make decisions, the more damage a bad suggestion can cause.
What makes this especially tricky is how the nature of search is changing. AI-generated summaries and direct answers are replacing the old blue-link model. People no longer sift through options, they trust the top result, especially when it’s delivered with confidence. But when that answer is wrong, the fallout can be immediate.
Scammers are already adapting. Instead of fighting over Google rankings, they’re building clean, credible-looking pages that are designed to appeal to AI, not just humans. Some even tailor their writing style to match how chatbots "think." That includes phishing pages disguised as crypto tools or travel support, and even malware downloads hidden in cracked software tutorials. If it reads well and looks clean, AI is more likely to surface it.
The arms race doesn’t stop there. Defensive domain registration, buying up every possible variation of your company’s URL, is a costly and losing battle. Language models can dream up endless variations, many of which don’t exist until the day they’re suggested.
What’s needed now is a smarter defense. Systems that can detect fake domains as they emerge. Tools that can flag misinformation before it spreads. And above all, AI models that can tell the difference between what’s real and what just sounds right.
Because once trust is broken, there’s no easy way to get it back.
Read next: AI’s Hidden Energy Bill: What a Single ChatGPT Prompt Really Costs
