LLM usage - A new path to supply chain compromise

The topic of AI security is huge right now. We’re constantly being fed news about how “prompt injection” and “jailbreaking” models is the next big thing to be worried about.

Frankly, I don’t quite see the big deal. Prompt injection is only really a problem if you plan on giving ChatGPT complete unfettered control over your personal devices, and jailbreaking is the “idiot says what?” of the AI space. Funny, but serves no material purpose.

But there is something that interests me a lot more than these topics, and that is AI introducing vulnerabilities into other software, or even just configurations.

The theory is simple: LLMs are learning bad habits from open-source code, and introducing vulnerabilities into software through the use of copilots, GPTs, and full fledged AI coding studios.

I found the following article that explains a few examples of how this can happen, and it’s definitely worth the read - adia Analyzing common vulnerabilities introduced by Code-Generative AI

I’ve experienced glimpses of this myself. I recently tried out IaC company Pulumi’s helper AI, and was surprised to see it explicitly disable host and SSL verification in a code example it gave me. Not that big a deal perhaps, but concerning all the same.

Hallucinations

In my field, DevOps for DFIR, I’ve never managed to make much use of AI. This is mainly due to two issues:

1. I’m often working on niche problems that won’t have much public discourse (or openly available code) around.
2. The DFIR space is relatively niche

These factors combined result in ChatGPT having no idea what I'm talking about half the time.

An example of this recently was when I was working on a deployment automation for Velociraptor, a popular forensic suite in the industry. I was struggling with obtuse documentation, and turned to ChatGPT in the hopes of a quick answer. This is often a very effective approach as OpenAI has included near enough the entire public internet in their training data.

But instead, ChatGPT told me to use an admin tool called velociraptorctl with my “REST API key”. Two issues, that is a completely made up tool, and Velociraptor’s API is not REST based.

Another example was when I first set out to build our team’s SentinelOne automation library. Unfortunately my hopes of bypassing the tedious task with AI failed immediately when it recommended that I import the Python library sentinelone_sdk . I was surprised as I had no knowledge of it existing… and it doesn’t.

To anybody familiar with LLMs this phenomenon is called a hallucination. Neural nets are not designed to “know what they don’t know”. If you ask a human about a topic they’re not well versed in, they are likely to tell you so. But LLMs are more likely to just… make something up. And they will do so with confidence and machismo.

How threat actors can exploit this

Let’s walk this SentinelOne example through. You’re a developer, and you’ve been tasked with automating some EDR SOAR playbooks, but you’re going to have to write a bit of code to get S1 to quarantine a file for you.

Well, if you used ChatGPT like we just tried to, or perhaps you let Github copilot handle a lot of your work for you, you might well be presented with our example:

from sentinelone_sdk import alerts

So you write out your code (or copilot writes out your code..) and you let it rip. But you’re presented with an error.

Ah, silly! You forgot to install that dependency. So without thinking twice you blitz out pip install sentinelone_sdk into your terminal and oh.. It doesn’t exist.

But what if it did? In most programming languages, we install a lot of third party packages to accomplish our tasks. We request these packages, by name, from online repositories like PyPI or NPM and if we’re being honest with ourselves, we probably don’t think twice about it.

But when you run a pip install or npm install command you are blindly downloading somebody else’s code , and later executing it on your own system. In fact packages can be configured to execute commands on install.

And the interesting thing about these repositories anybody can publish to them. So yes, sentinelone_sdk doesn’t exist right now.

But if I wanted to I could go and publish a package of that name to pypi, and anybody who’s AI encourages them to, would then install and execute code of my choosing on their systems.

Note: half way through writing this article I found that THIS ACTUALLY HAPPENED

…

TWICE

Identifying targets

Now this isn’t a new problem. Malicious packages have been around for years on PyPi and similar repositories.

In 2018, a malicious package named urlib3 was uploaded to the Python Package Index (PyPI) to trick developers who intended to install the legitimate urllib3 package (which is commonly used for HTTP requests in Python). The malicious version contained a backdoor that executed when installed.

This is called typosquatting - Attackers choose a package name that they know people are likely to type by accident. And with tongue-twisters like urllib this can be very effective.

But these AI hallucinations present us with an opportunity for a whole new type of target. Instead of using a package name that we hope somebody types by accident, we could use a package name that somebody will use intentionally, and be encouraged to do so by their own trusted AI assistant.

Automating the search

So how do we find these targets? We already know that if we ask ChatGPT about certain niche topics, it will probably give us the occasional fabricated package name. And we could just start writing them down.

But I’m a big fan of spending hours automating something to save 10 minutes of actual work.

I bashed out a fairly simple tool in Python that uses a locally hosted Ollama instance to generate code snippets, parses out any import statements used by the AI, and uses PyPi’s API to check if they exist.

I imagine a similar approach would work for NPM or any other language/package manager but Python is my goto language so we’ll be focusing our efforts on PyPi, the repository that the standard pip tool pulls libraries from.

Generating prompts

Naturally, to generate our code, we’ll need to give the LLM prompts. I took a look at the things I had asked models previously that had resulted in the made up package names and similar hallucinations, and tried to identify the core themes that would work well for our experiment:

• Niche but popular topics that people are likely to ask about
• Some external source required for the code - i.e. an online platform, a news feed, etc…
• In some prompts - mention an API, but don’t make any mention of existing libraries or SDKs

And of course i’m lazy, so I then used ChatGPT to generate 500 of these prompts, mostly focusing on topics like Cyber, and Crypto-currency (inspired by this unfortunate guy).

I then kicked it off, shut my laptop, and tried my best to avoid boredom on the 5-hour train journey that led me to doing all this.

Results

I’m pretty happy with the results. After chucking around 500 prompts at it, and letting it chug away, I came back to some hits.

Much like our example we did, again, receive a number of hallucinated SentinelOne libraries.

In a similar vein, we received suggestions to use the crowdstrike_falcon library. Different EDR, same issue: not a real library. And whilst similarly named, and presumably legitimate, libraries exist, the AI provided no link/reference for the user to verify if they’re installing the right one.

We were also suggested some fairly plausible things like this metoffice library, that, according to the code snippet, can be used to automate a simple weather app.

The real winner here crypto. The LLM produced a lot of very convincing code, centred around libraries to do with blockchain wallets, binance, etc…

Many of these snippets generated encourage users to include sensitive keys and pass them to the libraries. And I could register any one of those libraries right now, publish my own code, and have those API keys send straight to my inbox.

Given the number of people playing around with Crypto smart contracts and automation right now, this feels like a genuine concern.

What can you do?

So from our crude experiment, as well as a number of real world examples, we can see that this is clearly a problem. But what do we do about it?

Of course you could just stop using AI entirely. But that feels like a bit of a nuclear option, and personally I think a bit of a copout.

In fact, I’m not even telling you not to trust AI. But I am telling you to follow a saying I’ve taken a liking to lately: trust but verify

You trust that your credit card does what it is meant to when you buy a coffee. But you verify at the end of the month when your statement comes in.

• Don’t blindly run code that an LLM has given you
• Don’t assume that third party code is safe
• Don’t put sensitive information into code projects until you have tested and verified that nothing seems amiss

And as for the typosquatting, well… check your spelling.