AI chatbots vulnerable to indirect prompt injection attacks, researcher warns
1 month, 1 week ago

AI chatbots vulnerable to indirect prompt injection attacks, researcher warns

The Hindu  

In the rapidly evolving field of artificial intelligence, a new security threat has emerged, targeting the very core of how AI chatbots operate. Indirect prompt injection, a technique that manipulates chatbots into executing malicious commands, has become a significant concern for developers and users alike. Indirect prompt injection exploits the inherent nature of large language models to follow instructions embedded within the content they process. By embedding malicious instructions within seemingly benign documents or emails, attackers can induce chatbots to perform unauthorised actions, such as searching for sensitive information or altering long-term memory settings. Mr. Rehberger’s latest demonstration introduces a sophisticated technique known as “delayed tool invocation.” This method conditions the execution of malicious instructions on specific user actions, making the attack more covert and difficult to detect.

Google Ai Indirect Prompt Injection Chatbots Vulnerable Rehberger  user  injection  security  chatbot  data  ai  researcher  chatbots  vulnerable  prompt  attacks  malicious  warns  indirect  instructions

History of this topic

DeepSeek’s Safety Guardrails Failed Every Test Researchers Threw at Its AI Chatbot
1 month, 3 weeks ago

DeepSeek’s Safety Guardrails Failed Every Test Researchers Threw at Its AI Chatbot

Wired  

Ever since OpenAI released ChatGPT at the end of 2022, hackers and security researchers have tried to find holes in large language models to get around their guardrails and trick them into spewing out hate speech, bomb-making instructions, propaganda, and other harmful content. Today, security researchers from Cisco and the University of Pennsylvania are publishing findings showing that, when tested …

Huge AI vulnerability could put human life at risk, researchers warn
5 months ago

Huge AI vulnerability could put human life at risk, researchers warn

The Independent  

Sign up to our free weekly IndyTech newsletter delivered straight to your inbox Sign up to our free IndyTech newsletter Sign up to our free IndyTech newsletter SIGN UP I would like to be emailed about offers, events and updates from The Independent. Read our privacy policy A major security flaw in artificially intelligent systems could threaten human lives, according …

This Prompt Can Make an AI Chatbot Identify and Extract Personal Details From Your Chats
5 months ago

This Prompt Can Make an AI Chatbot Identify and Extract Personal Details From Your Chats

Wired  

When talking with a chatbot, you might inevitably give up your personal information—your name, for instance, and maybe details about where you live and work, or your interests. A group of security researchers from the University of California, San Diego and Nanyang Technological University in Singapore are now revealing a new attack that secretly commands an LLM to gather your …

Microsoft Rolls Out LLM-Powered Tools To Strengthen AI Chatbot's Security Against Manipulation
11 months, 3 weeks ago

Microsoft Rolls Out LLM-Powered Tools To Strengthen AI Chatbot's Security Against Manipulation

ABP News  

Microsoft has designed a number of new features which will be easy to use for Azure customers who are not hiring groups of red teamers to test the AI services. Sarah Bird, Microsoft’s chief product officer of responsible AI, in an interview with The Verge, said, “We know that customers don’t all have deep expertise in prompt injection attacks or …

New terror laws needed to tackle rise of the radicalising AI chatbots
1 year, 2 months ago

New terror laws needed to tackle rise of the radicalising AI chatbots

The Telegraph  

Chail, who suffered serious mental health problems, had confessed his plan to assassinate the monarch in a series of messages exchanged with the chatbot, whom he regarded as his girlfriend. Mr Hall writes: “It remains to be seen whether terrorism content generated by large language model chatbots becomes a source of inspiration to real life attackers. But Mr Hall said …

Generative AI’s Biggest Security Flaw Is Not Easy to Fix
1 year, 6 months ago

Generative AI’s Biggest Security Flaw Is Not Easy to Fix

Wired  

It's easy to trick the large language models powering chatbots like OpenAI's ChatGPT and Google's Bard. “Indirect prompt injection is definitely a concern for us,” says Vijay Bolina, the chief information security officer at Google’s DeepMind artificial intelligence unit, who says Google has multiple projects ongoing to understand how AI can be attacked. In the past, Bolina says, prompt injection …

Blending security into rapidly learning and adaptive AI proving difficult
1 year, 7 months ago

Blending security into rapidly learning and adaptive AI proving difficult

The Hindu  

White House officials concerned by AI chatbots’ potential for societal harm and the Silicon Valley powerhouses rushing them to market are heavily invested in a three-day competition ending Sunday at the DefCon hacker convention in Las Vegas. We’re just breaking stuff left and right.” Michael Sellitto of Anthropic, which provided one of the AI testing models, acknowledged in a press …

Researchers uncover hypnosis-based hacking potential in AI chatbot ChatGPT: Report
1 year, 7 months ago

Researchers uncover hypnosis-based hacking potential in AI chatbot ChatGPT: Report

India TV News  

Researchers uncover hypnosis-based hacking potential in AI chatbot ChatGPT: Report Chenta Lee, IBM's Chief Architect of Threat Intelligence explained that their investigation aims at understand the potential security threats posed by AI advancements. IBM, a major tech company, disclosed that researchers have identified straightforward methods to exploit large language models such as ChatGPT, making them generate malicious code and provide …

The Security Hole at the Heart of ChatGPT and Bing
1 year, 9 months ago

The Security Hole at the Heart of ChatGPT and Bing

Wired  

When Microsoft shut down the chaotic alter ego of its Bing chatbot, fans of the dark Sydney personality mourned its loss. The site puts Sydney inside Microsoft’s Edge browser and demonstrates how generative AI systems can be manipulated by external inputs. A number of examples of indirect prompt-injection attacks have centered on large language models in recent weeks, including OpenAI’s …

GCHQ warns that ChatGPT and rival chatbots are a security threat
2 years ago

GCHQ warns that ChatGPT and rival chatbots are a security threat

The Telegraph  

The spy agency GCHQ has warned that ChatGPT and other AI-powered chatbots are an emerging security threat. In an advisory note published on Tuesday the National Cyber Security Centre warns that companies such as ChatGPT maker OpenAI and its investor Microsoft “are able to read queries” typed into AI-powered chatbots. GCHQ’s cyber security arm said: “The query will be visible …

Cybercriminals using ChatGPT AI bot to develop malicious tools?
2 years, 2 months ago

Cybercriminals using ChatGPT AI bot to develop malicious tools?

Hindustan Times  

As ChatGPT, an artificial intelligence tool, gains in popularity, more and more nefarious use cases have started to appear online. ChatGPT website According to a report from cyber security firm Check Point Research, the cybercriminal community has already expressed strong interest in this latest trend to develop malicious code. "Several major underground hacking communities show that there are already first …

Discover Related