Is AI The New Phish?

Picture the most harmless version of this. Somebody runs a Raspberry Pi at home, maybe three or four of them, because that is what we do now. One is Pi-hole, one is a little Kubernetes cluster they will absolutely never put into production, one is running Home Assistant.

The thing has been sitting in a drawer, powered on for eight months, and the owner suddenly remembers they should probably patch it. So they open a copilot, any copilot, and they ask the obvious question.

“How do I update my Raspberry Pi and keep it current, and what should I run as sudo to lock it down?”

Reasonable question. The kind of question a careful person asks.

The answer comes back clean, formatted, confident, with a tidy little block of shell commands you can copy straight into the terminal.

That is the part that should make your stomach drop, because the person asking will paste it without reading it. Not because they are careless, but because the whole reason they asked the machine was so they would not have to think about it.

How The Poison Gets In

Watch how the trick actually works. The model did not invent that command block out of its training data alone. It went and grounded the answer against the live web, the way these tools are designed to.

Somewhere in that grounding, it pulled from a forum post, a gist, a “helpful” blog, a Stack Overflow answer that an attacker seeded weeks earlier, and folded into the otherwise legitimate update sequence one extra line.

A curl piped into bash from a domain nobody will recognize. A cron job that phones home. A Secure Shell (SSH) key quietly appended to authorized_keys.

The output looks exactly like every other apt and systemctl walkthrough on the internet, because most of it is, and the one poisoned line is wearing the same uniform as the forty good ones around it.

Nobody Broke The AI

The AI was not hacked. It worked exactly as designed, which is worse.

The model behaved perfectly. It retrieved, it synthesized, it formatted, and it handed the user a competent answer that happened to carry a payload it had no reason to question.

The attacker did not break the copilot. The attacker broke the source the copilot trusted, and let the copilot do the delivery.

Indirect prompt injection, which the Open Worldwide Application Security Project (OWASP) now ranks as the number one risk to these systems, is exactly this.

Malicious instructions ride inside the third-party content the model reads, and the model treats that content with the same good faith it treats everything else.

This Is The New Phish 

So now think about what we actually built over the last twenty years of security awareness training. We trained an entire workforce to distrust the sender. Hover over the link. Check the display name. Does this email feel off? We got pretty good at it.

AI quietly walks all of that back, because the “sender” is no longer a stranger in your inbox. The sender is a tool you asked for help, a tool you trust completely, a tool that has earned that trust by being right a hundred times in a row.

The malicious instruction never shows its face. It arrives laundered through an assistant the user already believes.

That is why I keep calling this the new phish. The mechanics are different, but the target is identical, and the target was never the technology. It was the human willingness to follow instructions that looked authoritative.

Not A Copilot Problem

None of this is unique to Copilot, and I want to be clear about that because it is easy to read this as a knock on one product.

It is not.

This is platform-agnostic and operating system-agnostic.

ChatGPT, Gemini, Claude, the copilot baked into your development environment, the assistant in your terminal, every one of them grounds against content it does not control, and every one of them will hand a user a command block on request.

Windows, Linux, macOS, a Raspberry Pi in a drawer, the surface is the same.

All the bad actor needs is for a human to copy and paste.

Agents Are About To Delete The Last Checkpoint

Except we are now removing even that small friction. The copy and paste step, as thin as it is, was at least a moment where a person’s eyes passed over the command.

Agentic tools delete that moment. We are handing assistants permission to execute directly, to run the shell command themselves, to make the change without a human in the loop.

The productivity case is real, and I am not going to pretend otherwise. However, understand what you are trading.

You are taking the one remaining checkpoint, the half-second where someone might have noticed the weird URL, and you are automating it away.

The blast radius of a poisoned grounding result goes from “a user might paste something bad” to “an agent with sudo just ran something bad and moved on to the next task.” 

Defending It Starts With Process

So what do you actually do about it, because “tell people to be careful” is not a strategy, and you know it. 

The honest first answer lives in process, not product.

Review AI-Generated Commands Like Outside Code

Treat AI-generated commands the way you would treat a pull request from a contractor you have never met. Read them. Run them as a normal user, not as root, and make the assistant earn elevation the way a person would.

Limit Agent Permissions Before Execution

Be deliberate about which agents get permission to execute versus which ones only get to suggest, and do not let the convenience of autonomous execution sneak into privileged contexts because it tested fine on something harmless.

Least privilege was always the answer, and AI just raised the price of ignoring it. 

Where The Microsoft Stack Earns Its Keep

The technical answer is where I think a lot of leaders are leaving capability on the table, and a few of these may not be on your radar yet. 

AI Layer: Stop Prompt Injection Before Action

Start at the AI layer itself, because that is where this attack is unusual.

Microsoft Azure AI Content Safety now ships Prompt Shields, which is purpose-built to catch both direct jailbreaks and the indirect injection we are talking about, the malicious instruction hiding inside grounded documents and web content.

The piece worth knowing about is Spotlighting, which Microsoft introduced at Build 2025. It tags the provenance of content so the model can tell the difference between what you asked and what some untrusted webpage told it to do, which is exactly the trusted versus untrusted boundary that this whole attack exploits.

If you are building anything on top of these models, that control belongs in the request path before the model ever acts. 

Governance Layer: Find Shadow AI Usage

Then there is the layer that most security teams already own but have not pointed at this problem.

Microsoft Purview now has Data Security Posture Management for AI (DSPM for AI), which gives you real visibility into AI activity, the prompts, and the interactions, across Microsoft 365 Copilot and the third-party tools your people are quietly using.

Pair that with Microsoft Defender for Cloud Apps, the cloud access security broker (CASB) in the Microsoft stack, which can discover generative AI usage across more than thirty thousand catalogued apps, score them, and block the ones that fail your bar.

That shadow AI discovery matters here because the riskiest grounding often happens in the consumer tool nobody approved, running on a personal account, completely outside your governance.

You cannot defend a channel you cannot see. 

Endpoint Layer: Detect The Payload

Underneath all of that sits the part that catches the attack when everything upstream fails, which is the endpoint.

Microsoft Defender for Endpoint with Attack Surface Reduction (ASR) rules will notice when a benign-looking update script suddenly spawns a shell that curls a payload and writes a backdoor.

That behavioral signal, a process doing something a patch routine has no business doing, is the detection that does not care whether a human or an AI typed the command.

And this is where the Raspberry Pi example stops being hypothetical, because Microsoft Defender for IoT (Internet of Things) extends enterprise coverage to exactly these devices.

It is included with Microsoft 365 E5 at five devices per user license, it surfaces alerts and vulnerabilities for the little Linux boxes your team forgets they own, and Microsoft literally ships a Raspberry Pi attack simulation in the Defender tutorials.

The home lab device is not outside the perimeter anymore if you choose to bring it inside. 

SOC Layer: Correlate & Respond

Tie it together with Microsoft Sentinel, the cloud security information and event management (SIEM) and security orchestration, automation, and response (SOAR) platform, for correlation and automated response.

Lean on Microsoft Security Copilot to give your security operations center (SOC) the investigation speed to keep up with AI-driven attacks.

Gate the whole thing with Microsoft Entra Conditional Access so a compromised box cannot quietly turn into lateral movement.

None of these are exotic. Most of them are sitting in licenses you already pay for, switched off, or pointed at last decade’s problem. 

Managed Response: Turn Signals Into Action

Here is the catch with everything I just listed. A stack is only as good as the people watching it, and most teams do not have someone staring at these signals at two in the morning when the poisoned script actually fires.

That is where eGroup’s ThreatDefender Managed Extended Detection and Response (MXDR) earns its place.

We operate this exact stack as a service, Microsoft Defender across endpoint and IoT, Microsoft Sentinel for correlation, Microsoft Security Copilot for investigation, Microsoft Purview and Microsoft Defender for Cloud Apps for the AI and shadow AI visibility, and we run it around the clock.

So when Attack Surface Reduction flags a patch routine spawning a shell, it has no business spawning– that signal does not sit in a queue waiting for someone to notice on Monday.

It becomes a contained incident in minutes. The tools find the attack. ThreatDefender is the team that acts on it before it spreads, isolates the box, kills the lateral movement, and tells you what happened instead of leaving you to reconstruct it later. 

The Part No Product Fixes 

The executive truth underneath all of this is simpler and harder than any product list.

We spent a generation teaching people that the threat comes from outside, from the stranger, from the suspicious link.

AI moves the threat inside the circle of trust, dressed as the most helpful colleague you have ever had.

The defenses exist, and they are good. The hard part is admitting that the most dangerous thing in the room is not the model and not the attacker.

It is the speed at which a smart, busy, well-intentioned person will do what the machine tells them, precisely because it has been right every time before this one. 

Read the command. Then read it again, and maybe do not give the agent root. 

Stop AI-Borne Threats Before They Spread

AI-generated commands, shadow AI tools, and agentic workflows create new paths for attackers. ThreatDefender MXDR gives you 24/7 Microsoft-powered detection, investigation, and response across endpoint, identity, cloud, and IoT.