What is prompt injection?
Prompt injection is an attack in which crafted input changes how a language model behaves or what it outputs, in ways the system owner did not intend. OWASP ranks it LLM01, first in the Top 10 for LLM Applications 2025, published 17 November 2024.
The reason it is hard is architectural rather than incidental. NIST's adversarial machine learning taxonomy explains that generative models combine the data and instruction channels. A system prompt written by the developer and a paragraph pasted in by a user arrive as the same kind of thing: text in a context window. A prompt injection exploits the concatenation of untrusted input onto higher-trust instructions, so content the system was meant to read gets treated as a command it should obey.
That is why the usual software analogy only half works. SQL injection has a clean fix, because parameterised queries separate code from data. There is no equivalent separation in a language model, which is exactly why OWASP says that, given the stochastic influence at the heart of the way models work, it is unclear if there are fool-proof methods of prevention.
Who does prompt injection affect, and how does it reach them?
NIST splits the attack into two forms with very different threat models.
Direct prompt injection is mounted by the primary user of the system, through the query access they already have. Someone types instructions designed to override the higher-trust instructions the application designer wrote, usually to circumvent restrictions on what the model will output. A jailbreak is the familiar case. The person taking the risk and the person bearing it are the same person.
Indirect prompt injection is mounted by a third party through resources the system consumes. NIST describes it as being enabled by resource control, where an attacker plants instructions in something the model will later read, without ever touching the application directly. A web page, an email, a shared document, a calendar invitation, a code comment. The critical asymmetry, which NIST states directly, is that in many cases it is the primary user of the model who is harmed by the compromise.
This is not theoretical. CVE-2025-32711, published 11 June 2025, records an AI command injection vulnerability in Microsoft 365 Copilot that allowed an unauthorised attacker to disclose information over a network. Microsoft scored it 9.3 critical, NIST scored it 7.5 high, and it is classified under CWE-74, the same weakness class as the older injection families.
Where does prompt injection fit in Australian regulated work?
The exposure follows the data, not the tool. Any assistant that reads material an outsider can influence is reachable, and in regulated work that material is often the most sensitive thing in the building. A claims assistant summarising correspondence. A compliance tool reading vendor documentation. A recruitment screen reading uploaded CVs.
Two Australian consequences are worth naming. First, an indirect injection that causes personal information to be disclosed is a privacy incident with the ordinary notification analysis attached, regardless of the fact that a model was the mechanism. Second, for APRA-regulated entities an AI assistant with access to information assets sits inside the information security obligations of CPS 234, and where it supports a critical operation the service provider and continuity expectations of CPS 230 come with it.
The risk compounds with autonomy. A model that only writes text can be wrong. An agentic system that acts on injected instructions can send, delete, pay or publish before anyone reads the output. OWASP's agentic list opens with agent goal hijack for that reason. It also compounds with retrieval, because a RAG knowledge base is a durable place to hide an instruction that fires whenever the right question is asked.
What should practitioners do about prompt injection?
Design for containment rather than prevention. OWASP's mitigations are the sensible baseline: constrain model behaviour with specific system instructions, define and validate expected output formats, filter input and output, enforce least privilege on the model's own access, require human approval for high-risk actions, segregate and clearly identify external content, and run adversarial testing.
Translate that into procurement questions. Ask a vendor which content sources the model treats as untrusted, what the model can do without a human approving it, whether tool permissions are scoped to the requesting user or to a shared service account, and how an injection attempt would appear in logs. A supplier who cannot answer the last one has no detection at all.
Then set the internal rule that matters most. Never let a model's output authorise an irreversible action on its own, and never give an assistant standing access broader than the person who invoked it. Prompt injection turns the model's privileges into the attacker's privileges, so the size of the blast is decided before the attack, when someone chooses what the assistant can reach.
Bottom line
Prompt injection is an attack where crafted input changes how a language model behaves, and it is hard because generative models combine the data and instruction channels, so content the system was meant to read gets treated as a command it should obey. There is no parameterised-query equivalent that separates code from data here, which is why the realistic goal is containment rather than prevention: constrain behaviour, validate output formats, filter input and output, enforce least privilege on the model's own access, segregate external content and require human approval for high-risk actions. In Australian regulated work the exposure follows the data rather than the tool, so an indirect injection that discloses personal information is a privacy incident with the ordinary notification analysis attached, and an assistant reaching information assets inside an APRA-regulated entity brings the prudential information security and service provider expectations with it. The internal rule that matters most is the simplest one, never let a model's output authorise an irreversible action on its own and never give an assistant standing access broader than the person who invoked it, because the size of the blast is decided when someone chooses what the assistant can reach.
TheAICommand. Intelligence, At Your Command.*
TheAICommand. Intelligence, At Your Command.