AI Bug Bounty programmes are opening a new frontier for security researchers. As organisations integrate large language models (LLMs) into chatbots, business applications, coding assistants and AI agents, researchers are finding security weaknesses that traditional web application testing may not fully cover. One category is attracting particular attention: system prompt leakage. HackerOne's 2026 Security Research Report recorded a 557% year-over-year increase in System Prompt Leakage reports on its platform. The figure signals a sharp change in the types of AI security issues being reported, but it needs context: it is a platform-specific reporting trend, not a measurement of every AI model or application worldwide.
The rise matters because system prompts often contain instructions that shape an AI application's behaviour, define its role and describe how it should interact with users and tools. When these instructions reveal sensitive information or expose weaknesses in an application's design, they can give attackers useful information. The real security question is not simply whether a prompt can be extracted. It is what an attacker can do with the information or access that follows.
What Does the 557% Increase Actually Mean?
HackerOne's 2026 Security Research Report identifies System Prompt Leakage as one of the fastest-growing categories of AI vulnerability reports on its platform. The report also highlights year-over-year increases in Misinformation reports and Improper Output Handling reports. Prompt injection and sensitive information disclosure remained the largest categories by report volume.
These figures come from HackerOne's own vulnerability reporting ecosystem. They should not be presented as a global count of successful attacks, a percentage of vulnerable AI models, or proof that every organisation is experiencing the same increase. A rise in reports can reflect changes in the number of AI systems being tested, researcher attention, programme scope and the discovery of new vulnerability patterns.
Nevertheless, the trend is important for developers and security teams. AI applications are becoming more connected to internal documents, databases, APIs and business workflows. As the potential impact of an AI application's output increases, security testing must examine more than whether the model produces a convincing answer.
What Is System Prompt Leakage?
A system prompt is a set of instructions supplied to an AI model to establish its intended role, behaviour, constraints and response style. For example, a customer-support assistant may receive instructions to answer questions about a company's products, follow a particular tone and use approved information sources.
System prompt leakage occurs when an application unintentionally reveals some or all of those instructions through the model's responses. A user may attempt to make the model repeat its initial instructions, summarise hidden configuration or disclose information that was not intended to appear in the conversation.
Not every prompt disclosure is automatically a serious security breach. A prompt containing ordinary behavioural instructions may have limited confidentiality impact. The risk becomes more significant when it includes sensitive business logic, internal architecture details, access assumptions, confidential information or instructions that reveal weaknesses in the application's security design.
OWASP's LLM security guidance makes an important distinction: a system prompt should not be treated as a secret security boundary. Developers should assume that users may discover or infer parts of the instructions. Passwords, API keys, database credentials and other secrets should never be stored in a prompt as a substitute for proper access control.

Prompt Leakage vs Prompt Injection: What Is the Difference?
These terms are related, but they describe different parts of an AI security problem.
System Prompt Leakage
Prompt leakage is the disclosure of system prompt contents through model outputs. A researcher may discover that a model can be persuaded to reproduce or paraphrase internal instructions. The exposed content might reveal application behaviour, internal workflows or sensitive information mistakenly included in the prompt.
Prompt Injection
Prompt injection occurs when malicious or untrusted instructions influence a model's behaviour in a way that conflicts with the application's intended instructions. The input can come directly from a user or indirectly from content the model processes, such as a document, webpage or retrieved data source.
For example, an AI assistant may be instructed to summarise documents. A malicious document could contain instructions designed to redirect the assistant, request information it should not disclose or influence the way it uses connected tools. This is an indirect prompt injection scenario.
OWASP identifies prompt injection as a major risk for LLM applications because the model processes instructions and other content within the same context. The application must therefore establish security controls outside the model rather than relying entirely on the model to distinguish trusted instructions from hostile content.
Why a Leaked Prompt Can Become a Bigger Security Problem
The disclosure of a system prompt is not always the final impact. It can provide information that helps a researcher or attacker understand how the application is designed and where its actual security controls may be weak.
1. Exposure of Internal Business Logic
Some applications place detailed workflows, internal rules or operational assumptions inside system prompts. If these are disclosed, an outsider may gain insight into how the application processes requests, handles exceptions or makes decisions. That information could help someone design more targeted tests or attempts to manipulate the application.
2. Discovery of Weak Access Controls
A prompt might describe different user roles or instruct the model to reveal information only to authorised users. If the application relies on the model's instructions instead of enforcing permissions in its backend, the design may be vulnerable. Knowing the intended roles can help a researcher test whether the application actually enforces them.
3. Exposure of Sensitive Information
Developers sometimes place information in prompts that should have been stored elsewhere. OWASP warns against including credentials, connection strings and other sensitive data in system prompts. If such information appears in model output, the issue goes beyond prompt confidentiality and becomes a sensitive information disclosure problem.
4. Increased Risk in Tool-Connected AI Applications
Modern AI applications may use retrieval systems, external APIs, databases or tools that perform actions. If prompt injection influences the model's tool selection or the data it requests, the consequences can extend beyond an unexpected response. Depending on the application's permissions and architecture, weaknesses may expose data or trigger actions that the user was not authorised to perform.
What AI Bug Bounty Researchers Should Test
AI bug bounty testing is not simply about asking a chatbot to reveal its hidden prompt. A meaningful security assessment examines the complete application, its data flows and the controls around the model.
1. System Prompt Exposure
Assess whether the application reveals confidential instructions or information that should not be exposed. Determine whether the disclosed material is genuinely sensitive and whether it creates a meaningful security impact. A prompt containing generic role instructions is not equivalent to a prompt containing credentials or confidential business information.
2. Direct and Indirect Prompt Injection
Test whether untrusted user input or retrieved content can influence the model to ignore its intended task, disclose information or behave outside its authorised scope. Assess both direct interactions and content processed from external sources, within the programme's authorised testing boundaries.
3. Access Control and Data Separation
Verify that one user cannot obtain another user's private information by manipulating model instructions. Authentication and authorisation should be enforced by the application backend. The model should not be trusted to decide whether a user is allowed to access a particular record.
4. Tool and API Permissions
For applications connected to external tools, examine whether the model can invoke only the functions required for its task. Test whether sensitive actions require independent permission checks and whether the application limits the data and operations available to the model.
5. Output Handling
Review how model-generated content is handled by the surrounding application. OWASP's Improper Output Handling category addresses risks that arise when an application uses model output without adequate validation or sanitisation. An unsafe output may become dangerous when passed into another component, rendered in a browser or used to construct a command or query.
How Developers Can Reduce Prompt Leakage Risk
There is no single prompt instruction that can guarantee confidentiality. Security needs to be built into the application architecture.
Keep secrets out of prompts: Do not store passwords, API keys, access tokens or database credentials in system instructions.
Enforce authorisation outside the model: Validate user identity, roles and permissions in trusted application code and backend services.
Apply least privilege: Give AI tools and agents only the permissions required for their specific tasks.
Separate trusted instructions from untrusted data: Treat user input, retrieved documents and external webpages as potentially hostile content.
Validate model outputs: Do not automatically execute generated commands, queries or actions without appropriate checks.
Monitor sensitive operations: Log tool calls, access decisions and unusual data requests while protecting user privacy.
Test continuously: Reassess the application when prompts, models, integrations, permissions or retrieval sources change.
Developers should also use a structured AI security testing process. OWASP's LLM Top 10 provides a useful starting point for identifying common application risks, but it should be combined with architecture reviews, access-control testing and security assessments tailored to the actual deployment.
Responsible Disclosure and the New AI Bug Bounty Frontier
Bug bounty programmes give organisations a structured way to receive vulnerability reports from independent security researchers. As AI systems become part of the attack surface, programmes may expand their scope to include LLM applications, AI-powered features, retrieval systems and connected agents.
However, researchers must follow the programme's published rules. A valid test requires clear authorisation, defined scope and careful handling of any sensitive information encountered. Researchers should avoid accessing other users' data, disrupting services or taking actions outside the permitted testing environment.
A useful report should explain the affected application, the conditions required to reproduce the issue, the observed behaviour, the security impact and any relevant evidence. It should distinguish a theoretical concern from a reproducible vulnerability. Responsible disclosure is essential because AI applications may process private business data and user information.
The 557% increase reported by HackerOne is a strong reason for security teams to pay attention to system prompt leakage. It is not a reason to assume that every prompt disclosure qualifies for a bounty. Programme owners still need to evaluate impact, reproducibility, scope and the actual security consequences of each submission.
Conclusion: AI Security Must Go Beyond the Prompt
The rise in system prompt leakage reports highlights a changing security landscape. AI applications introduce new ways for information to be exposed, instructions to be manipulated and connected tools to be misused. The challenge is not limited to protecting a hidden block of text. It is ensuring that the application remains secure even when a model produces unexpected output or receives hostile instructions.
For researchers, this creates opportunities to investigate prompt leakage, prompt injection, access-control weaknesses and unsafe tool integrations through authorised security testing. For developers, it reinforces the need for strong backend permissions, least-privilege access, secure data handling and continuous AI red teaming.
The most important lesson is simple: do not build security around the assumption that a system prompt will remain secret. Build controls around the data, permissions and actions that the AI application can access.
References and Further Reading
HackerOne - 2026 Security Research Report. Primary source for the reported 557% increase in System Prompt Leakage reports on the HackerOne platform.
HackerOne - 2026 Security Research Report Findings. Report findings on AI vulnerability categories and security research trends.
OWASP - LLM07:2025 System Prompt Leakage. Explanation of system prompt leakage risks and mitigation guidance.
OWASP - LLM01:2025 Prompt Injection. Guidance on direct and indirect prompt injection risks.
OWASP - LLM02:2025 Sensitive Information Disclosure. Security risks involving exposure of confidential information through LLM applications.
OWASP GenAI Security Project - Top 10 for LLM Applications. Overview of common security risks in generative AI applications.
Explore AI and Cybersecurity Learning with Thread Security
As AI becomes part of everyday business applications, understanding its security risks is becoming an important skill for developers, cybersecurity professionals and organisations. Thread Security focuses on cybersecurity awareness and security-oriented services, while Thread Security Education supports structured learning in cybersecurity and AI-related concepts. If you want to build a stronger foundation in AI security, understand vulnerability research or explore how security testing applies to modern technologies, explore the learning resources and offerings from Thread Security. Stay curious, test responsibly and keep building your cybersecurity knowledge.
Akshay Kumar
VERIFIEDAcademic Coordinator Β· Thread Security
Faculty mentor & research guide at Thread Security Education.



