Phishing emails now targeting AI assistants with hidden prompt injections

Infosecurity Magazine reports that Barracuda has identified phishing emails specifically designed to manipulate both human users and AI assistants. The emails contain hidden prompt injection attacks, which are instructions embedded in the message content intended to influence the behaviour of AI tools that process or summarise the email. For example, an AI assistant reading the email on behalf of a user might be instructed to ignore security warnings, approve a transaction, or extract and forward sensitive information. This represents a new variation on phishing that exploits the growing use of AI-powered email filtering, summarisation and response tools in enterprise environments. The technique is particularly concerning because it targets the tools that organisations are deploying to improve productivity and reduce the burden of email management.

Why this matters for UK organisations

For UK businesses, this development is significant because many organisations are now deploying or evaluating AI assistants to help manage email workflows, summarise messages, draft responses or automate routine tasks. These tools are designed to save time and improve productivity, but they also introduce a new attack surface. If an AI assistant can be manipulated by carefully crafted input, it may take actions that a human user would not, or it may present information in a way that makes malicious content appear legitimate. This is particularly concerning in environments where AI tools have access to sensitive data, can initiate transactions, or are trusted to make decisions on behalf of users. The risk is compounded by the fact that many organisations are deploying these tools quickly, without fully understanding the security implications or establishing governance frameworks to manage how they are used, what data they can access, and what actions they are permitted to take.

What to review

Review what permissions and access your AI assistants have, how they handle untrusted input, and whether your security controls account for the possibility that AI-generated summaries or recommendations may themselves be influenced by attacker-controlled content. Ensure there is clear ownership of AI tool governance, including who approves deployment, who monitors usage, and who is responsible for reviewing security incidents involving AI systems. Check that your AI tools are configured to operate with the principle of least privilege, and that they cannot take high-risk actions without human approval. Review whether your email security controls can detect prompt injection attempts, and whether your security awareness training includes guidance on the limitations of AI-generated content. Consider implementing logging and monitoring for AI assistant activity, particularly where these tools have access to sensitive data or can initiate transactions. Finally, ensure your incident response plan includes a process for investigating incidents involving AI systems, including how you would identify what actions an AI tool took, what data it accessed, and whether it was manipulated by external input.

Source: Infosecurity Magazine