Prompt Injection: When Hackers Hijack LLMs

Prompt Injection: An Attack on LLM-Based Systems

In recent years, Large Language Models (LLMs) such as ChatGPT and Claude have made enormous progress and are now used in a wide range of applications. But as with any new technology, there are new security risks. One of these risks is prompt injection — an attack on LLM-based systems in which an attacker uses specially crafted prompts to overwrite the system prompt or bypass safety guardrails.

What Is Prompt Injection?

Prompt injection is a specific attack on LLM-based systems in which an attacker, by entering specially crafted prompts, overwrites the system prompts or bypasses the system's security mechanisms. This can cause the system to perform unwanted or harmful actions. Prompt injection is not to be confused with adversarial attacks on audio or image systems, where an attacker attempts to manipulate a system's input to achieve a specific behavior.

Indirect Prompt Injection

A further risk is indirect prompt injection, where an attacker does not access the LLM system directly but instead via a website or plugin that interacts with the system. This can happen, for example, by entering malicious data into a form on a website that is then forwarded to the LLM system. Indirect prompt injection can be especially dangerous because it allows attackers to hide their attacks and make them harder to detect.

Jailbreaking vs. Prompt Injection

Jailbreaking and prompt injection are two different attacks on LLM-based systems. Jailbreaking refers to attempts to bypass a system's security mechanisms to gain access to functions or data that are normally inaccessible. Prompt injection, by contrast, refers specifically to the manipulation of prompts to induce the system to perform certain actions. While jailbreaking often targets the circumvention of security mechanisms, prompt injection targets the manipulation of the system prompts.

Countermeasures

To prevent prompt injection attacks, various countermeasures can be taken:

• Input sanitization: Checking and cleaning input data to ensure it is not harmful.
• Prompt isolation: Separating system prompts from user prompts to prevent an attacker from manipulating the system prompts.
• Least privilege: Restricting the system's rights and access to prevent an attacker from causing damage.

By implementing these countermeasures, developers can protect their LLM-based systems from prompt injection attacks and increase the security of their applications.