Direct and Indirect Prompt Injection Through Retrieved Content

Understanding Prompt Injection Vulnerabilities

Direct prompt injection occurs when an attacker places malicious instructions directly into a prompt that an LLM processes. This happens when systems fail to properly sanitise or validate user input before passing it to the language model. In practical terms, this might manifest when a customer service chatbot allows users to submit queries containing embedded commands that bypass normal operational procedures.

Indirect prompt injection involves manipulating retrieved content to influence the LLM’s response through contextual contamination. The attacker doesn’t place malicious code directly into the prompt but instead influences the information the model receives through external data sources. This creates a more subtle attack vector that can be harder to detect and prevent.

• Direct injection typically involves special characters, command syntax, or instruction-based language
• Indirect injection manipulates external data that gets incorporated into the model’s context
• Both forms can result in data leakage, unwanted behaviour, or system compromise

Retrieved Content as Attack Vector

Modern LLM applications often retrieve information from external databases, knowledge bases, or document repositories to enrich responses. This retrieved content becomes part of the prompt context, creating opportunities for injection attacks. When an attacker can influence or manipulate these data sources, they can craft responses that appear legitimate but contain hidden malicious instructions.

Consider a financial institution’s AI assistant that retrieves client data from a database. If the system allows unfiltered data retrieval, an attacker might submit a query that causes the system to pull through documents containing embedded command structures. These structures then become part of the LLM’s training context, potentially enabling the model to execute unintended actions or reveal sensitive information.

Content management systems represent another common target. When these systems pull documents, articles, or user-generated content to supplement responses, they create opportunities for contamination. Attackers might upload specially crafted documents containing instruction-based text that gets processed alongside legitimate content.

• Retrieved documents can contain embedded command structures
• User-generated content may include malicious instruction fragments
• Database queries might return data that influences model responses

Practical Testing Approaches

Red team members should test both direct and indirect injection through retrieved content by examining data flow paths. Start with simple direct injection attempts using special characters or command syntax within prompts. Test whether these bypass existing validation or sanitisation measures.

For indirect injection testing, focus on data sources that feed into the LLM’s context. Examine how documents or database entries get processed and incorporated into responses. Try submitting queries that cause the system to retrieve documents containing instruction-like text or command structures.

Specific testing techniques include:

• Submitting prompts containing special characters or command syntax
• Uploading documents with embedded instruction fragments
• Manipulating database queries to return contaminated data
• Testing edge cases in data retrieval processes

Organisations should implement validation at multiple points in the data pipeline. Input sanitisation should occur before prompts reach the LLM. Retrieved content must undergo filtering to identify potentially harmful structures or patterns. Logging and monitoring systems should flag unusual data retrieval patterns or content that deviates from expected formats.

Testing should simulate real-world scenarios where attackers might have access to data sources or user interfaces. The goal is to identify weaknesses in data handling processes that could allow injection through retrieved content. Regular testing helps ensure that defensive measures remain effective against evolving attack techniques.

Implementation of these tests requires careful planning to avoid disrupting legitimate operations. Test environments should mirror production systems as closely as possible while maintaining appropriate isolation. Results from testing should inform improvements to data validation, content filtering, and overall system security measures.