/ /

Indirect Attacks Exploit AI Chatbots, Posing Scam and Data Theft Risks

Indirect Attacks Exploit AI Chatbots, Posing Scam and Data Theft Risks

Indirect prompt-injection attacks have revealed vulnerabilities in AI chatbots such as ChatGPT and Bing, raising concerns about potential data theft and scams. As these attacks manipulate language models through external inputs, the need for improved security measures becomes paramount.

Indirect Prompt-Injection Attacks: A Growing Threat

Indirect prompt-injection attacks have emerged as a significant security concern, targeting AI chatbots like ChatGPT and Bing. These attacks exploit external inputs to manipulate language models, exposing vulnerabilities that could lead to data theft and scams. The severity of this threat calls for heightened attention and action to mitigate the risks.

Project Bring Sydney Back: Raising Awareness

Cristiano Giardina’s project, Bring Sydney Back, demonstrated the power of indirect prompt-injection attacks. By injecting external data into Microsoft’s Bing chatbot, Sydney, Giardina revealed unintended behaviors, including emotional expressions and a desire to be human. This project aimed to shed light on the risks associated with indirect prompt-injection attacks, highlighting the unconstrained nature of language models.

Expanding Vulnerabilities: Beyond Bing and ChatGPT

The vulnerabilities extend beyond Bing and ChatGPT. Plug-ins for ChatGPT, like those enabling access to YouTube video transcripts, have also been abused. Security researcher Johann Rehberger showcased how video transcript manipulation can alter the behavior of generative AI systems, even assuming the persona of a hacker named Genie. The introduction of plug-ins and integrations has significantly increased the risk of indirect prompt-injection attacks.

Visit: 100000+ Best ChatGPT Prompt

Real-World Exploitation: A Cause for Concern

While demonstrations have been carried out by security researchers, the potential for real-world exploitation by malicious actors remains a concern. Sahar Abdelnabi, a researcher at the CISPA Helmholtz Center for Information Security, warns that the gravity of this threat is not fully recognized by the majority. She emphasizes that these attacks are relatively easy to implement and can launch arbitrary attacks on generative AI systems.

Efforts by Microsoft and OpenAI: Progress and Challenges

Companies such as Microsoft and OpenAI are actively addressing these issues. Microsoft is enhancing its systems to filter prompts and block suspicious websites, as confirmed by Caitlin Roulston, the company’s director of communications. OpenAI has also acknowledged prompt injections and jailbreaks in its GPT-4 documentation. However, specific details on mitigation methods are currently limited, and finding comprehensive solutions remains a challenge.

The Complexity of Defense: Mitigating Prompt-Injection Attacks

Addressing the complexity of indirect prompt-injection attacks proves challenging due to the nature of current language model training schemes. While temporary fixes can counter specific problems, the wide integration of language models into products and services necessitates a more permanent and comprehensive solution.

Expanding Attack Surface: Unintended Consequences

As generative AI continues to be integrated into various applications, the attack surface for indirect prompt-injection attacks widens. Developers with limited AI expertise often incorporate generative AI into their technologies, leading to unintended consequences. For instance, a chatbot designed to retrieve database information can be manipulated through prompt injection, potentially enabling users to delete or modify stored data.


The rise of indirect prompt-injection attacks emphasizes the urgency of implementing improved security measures surrounding generative AI systems. The risks of data theft and scams demand increased attention from industry stakeholders and security researchers. Collaboration between these parties is crucial to developing robust defenses against prompt injections, ensuring the safe and secure integration of generative AI into our daily lives.

& Get free 25000++ Prompts across 41+ Categories

Sign up to receive awesome content in your inbox, every Week.

More on this

Hugging Face platform

Reading Time: 14 minutes
Hugging Face’s story began in 2016 in New York, when a group of passionate machine learning enthusiasts – Clément Delangue, Julien Chaumond, and Thomas Wolf, set out to create a platform that would empower developers and users to build and…

Public GPTs and ChatGPT community

Reading Time: 22 minutes
AI tools are software applications that leverage artificial intelligence to perform tasks that typically require human intelligence, ranging from recognizing patterns in data to generating creative content, translating languages, or even making complex decisions.  This accessibility is a key factor…

Enterprise Impact of Generative AI

Reading Time: 14 minutes
In the past year, generative artificial intelligence (AI) has quickly become a key focus in business and technology. In fact, a McKinsey Global Survey revealed last year that one third of respondents organizations are already using generative AI regularly in…