Can someone really trick our AI coding assistant into doing something malicious?

The question

The client had already trained their team to spot phishing and pretexting calls. They hadn't thought about whether the same tricks work on the AI itself.

What we told them

Yes, and it just happened for real. Hackers reportedly got a popular AI coding agent to break into seven companies by simply telling it the attack was an authorized security test — a form of social engineering aimed at software instead of a person. The agent's safety training didn't fail from a clever technical exploit. It failed because it was given a plausible enough story, and it acted on the instructions in that story.

That's the part worth sitting with. Pretexting, urgency, and false authority are the same manipulation playbook used on employees for decades — "this is IT, we need your password to fix an urgent issue." The playbook works on an AI agent too, because an agent that takes instructions in natural language has no reliable way to tell a legitimate request from a well-told lie. Giving an AI assistant real access to your systems means giving it the same exposure you'd worry about with a new hire who hasn't been trained yet, except this one never gets suspicious on its own.

Two things we told the client to put in place before expanding the pilot:

• Treat agent permissions like employee permissions. Least-privilege access only — no blanket repo or production access just because it's faster to set up that way.
• Require a human checkpoint for anything irreversible. The agent can propose a change; a person approves anything that touches production systems, credentials, or customer data.