In SQL, instructions are something the database engine does.
Data is something that is stored or used in a query.
Similar is true in cross-site scripting and buffer overflows, in that data and instructions have inherent differences in how they are processed.
Mitigations to all these issues enforce this separation between data and instructions. For example, using parameterised queries in SQL means that regardless of the input, the database engine can never interpret it as an instruction. The right mitigation solves the data/instruction conflation at its root. For example, Memory Tagging Extension (MTE) in ARM processors tags memory as to what its purpose is, and enforces that separation.
Under the hood of an LLM, there’s no distinction made between ‘data’ or ‘instructions’; there is only ever ‘next token’. When you provide an LLM prompt, it doesn’t understand the text it in the way a person does. It is simply predicting the most likely next token from the text so far. As there is no inherent distinction between ‘data’ and ‘instruction’, it’s very possible that prompt injection attacks may never be totally mitigated in the way that SQL injection attacks can be.
However, attempting to mitigate prompt injection is a vibrant area of research, including approaches such as:
-
detections of prompt injection attempts
-
training models to prioritise ‘instructions’ over anything in ‘data’ that looks like an instruction
-
highlighting to a model what is ‘data’
All of these approaches are trying to overlay a concept of ‘instruction’ and ‘data’ on a technology that inherently does not distinguish between the two.
