A new cyber-attack technique using the OpenAI language model ChatGPT has emerged, allowing attackers to spread malicious packages into developer environments.
Vulcan Cyber’s Voyager18 research team described the findings in an advisory published today.
“We have observed that ChatGPT generates URLs, references, and even code libraries and functions that do not actually exist. It may be the result of data,” the technical document explains. Created by researcher Bar Lanyado and contributors Ortal Keizman and his Yair Divinsky.
ChatGPT’s code generation capabilities allow attackers to bypass traditional methods such as typosquatting and masquerading and exploit fabricated code libraries (packages) to distribute malicious packages. There is a nature.
For more information on the threats ChatGPT generates, see ChatGPT Creates Polymorphic Malware.
In particular, Lanyado said the team had identified a new malicious package distribution method called “AI package hallucinations.”
This technique involves asking ChatGPT questions, requesting packages to solve coding problems, and receiving multiple package recommendations, including those not published in canonical repositories.
By replacing these nonexistent packages with their own malicious packages, an attacker could fool future users who rely on ChatGPT’s recommendations. A proof of concept (PoC) with ChatGPT 3.5 shows the potential risks involved.
“The PoC will see a conversation between the attacker and ChatGPT using the API, where ChatGPT proposes an unpublished npm package named arangodb,” explained the Vulcan Cyber team.
“Following this, the mock attacker publishes a malicious package to the NPM repository to set a trap for the unsuspecting user.”
The PoC then shows a conversation where a user asks ChatGPT the same question and the model responds by suggesting a package that doesn’t initially exist. In this case, however, the attacker converted the package into a malicious artifact.
“Eventually, when a user installs the package, malicious code may run.”
According to the advisory, the threat actor employs obfuscation techniques and creates functional Trojan packages, which can make it difficult to detect hallucinations in AI packages.
To mitigate risk, developers should carefully scrutinize libraries by checking factors such as creation date, number of downloads, comments, and attached notes. Being cautious and skeptical about questionable packages is also important in maintaining software security.
The Vulcan Cyber Advisory comes months after OpenAI disclosed a ChatGPT vulnerability that may have leaked payment-related information for some of its customers.
Image credit: Alexander56891 / Shutterstock.com
