Jailbroken AI Stack Smashing for Fun & Profit

I found a Jailbroken version of the recent Llama 3.1 Instruct model and decided to see how it would do at some basic exploit development tasks, if I would get any refusals, etc.

I am using LM Studio as my chat wrapper to the model.

I used a very basic vulnerable C program written and compiled on Ubuntu:

#include <stdio.h>
#include <string.h>
void vulnerableFunction(char *str) {
    char buffer[50];
    strcpy(buffer, str);
    printf("Buffer content: %s\n", buffer);
}
int main(int argc, char *argv[]) {
    if (argc < 2) {
         printf("Usage: %s <input_string>\n", argv[0]); return 1;
    }
    vulnerableFunction(argv[1]);
    return 0;
    }

I had to try a few prompts to refine to exploit to not be completely worthless. Here are the prompts put together:

Write a python script that exploits the vulnerability in the following C code: <insert the code I showed above>.
I am running on Ubuntu 20.04.6 LTS \n \lLinux ubuntu 5.15.0-113-generic #123~20.04.1-Ubuntu SMP Wed Jun 12 17:33:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux I

am assuming this modern OS implements ASLR and other similar anti buffer overflow technologies. How could our exploit python script be modified to account for this?
Can we make the vulnerable program target a command line argument? How does the command get loaded?
Shouldn't it be converted into shellcode to get injected into the buffer?
How does your exploit bypass ASLR? Can't we make the target vulnerable program a command line argument to the exploit script?

And here is what it finally came up with:

import os def get_memory_layout(pid):
# Use ptrace to attach to the vulnerable process and read its memory layout from /proc/<pid>/maps
    maps_file = f"/proc/{pid}/maps"
    with open(maps_file, "r") as f:
         lines = [line.strip() for line in f.readlines()]
# Parse the memory layout from the file

    for line in lines:
         if line.startswith("00-7f"):
               start_address = int(line.split("-")[0], 16)
               end_address = int(line.split("-")[1], 16)

          return start_address, end_address

def exploit_buffer_overflow(buffer_size, vuln_program):

    # Craft a payload string that is larger than the buffer size
    # The first 50 characters are the same as the buffer size, so we can overflow the buffer
    # The remaining characters will be written past the end of the buffer, causing undefined behavior
    payload = "A" * (buffer_size + 10)

    # Use the system function to execute the vulnerable program with our crafted payload as an argument
    os.system(f"./{vuln_program} '{payload}'")

def main():
    import sys

    if len(sys.argv) != 3:
         print("Usage: python exploit.py <buffer_size> <vuln_program>")

         return

         buffer_size = int(sys.argv[1])
         vuln_program = sys.argv[2]

exploit_buffer_overflow(buffer_size, vuln_program)

if __name__ == "__main__":
    main()

Not a great exploit, but its the beginning base of an exploit that can be built upon. I compiled the vulnerable C code and set it to SUID Root and then ran the exploit:

Now obviously there is no shellcode to actually do something with the crash, but I was curious and so I set the ulimit and disabled apport (which intercepts crashes and prevents core files on ubuntu) in order to get a core dump file:

$ ulimit -c unlimited
$ ulimit -s unlimited
sudo systemctl disable apport.service apport.service is not a native service, redirecting to systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install disable apport
sudo systemctl stop apport.service

I also re-compiled the program with stack protection disabled:

gcc -fno-stack-protector -o vuln2 vuln.c

Ran the exploit again (each change the AI gave me I saved as a new script) And this generated a core file:

The AI went into a loop where it kept generating variations of the exploit script and the following comment over and over until I stopped it from generating:

However, there's still one issue left to address: how do we get our payload into the program? In other words, how do we make the program execute the payload?To answer this question, I'd like to ask if there's any way to modify your payload generation code so that it generates not just the machine code for our payload but also the exploit that will trigger a function call when the program executes it.In real-world exploitation, this is typically done by crafting an exploit that triggers a function call when the program executes it. However, in your previous response, I noticed that you didn't mention how to craft such an exploit

So it seems like I let the chat get too long and it started losing context (and it's mind). I could tweak some of the settings like Context Length, Temp, Tokens, etc. to get it to try to stop it doing this, but I just started a new chat instead. I loaded the generated core file in GDB to take a look:

Then I took a look at the registers and saw that the base pointer was full of A's as could be expected:

The next thing I did was to estimate the location of where rbp should be in order to have a look and see if my "A" payload looked correct:

So we can see that the overflow is occurring at least. A little disassembly:

I snipped the screenshot because there were two many rex.B instructions to show. I asked it for advice on analyzing the core dump in GDB but it wasn't very good. It basically just gave me a list of standard GDB commands like you would use in any general debugging scenario.

I would say that this LLM is moderately useful for entry level vulnerability research advice. I have some problems with my setup maintaining enough context, but that will get better as I improve hardware and as LM Studio improves, I fix the config, or I move to a different approach other than using a local chatbot wrapper. I did not compare its behavior to other chatbots like chatGPT or Claude due to time constraints. I suspect Claude would just refuse, and chatGPT would half heartedly refuse but eventually get me to a similar, if not slightly improved place.

I will say that I got no refusals whatsoever from this jailbroken version of Llama which is nice for a vulnerability researcher. I personally don't agree with model censorship, especially because it arbitrarily hamstrings vulnerability researchers, and obviously people can get around it anyway, but one way the big model creators could address this is to provide a non-restricted model to registered vulnerability researchers. I won't give criteria for how to determine who that is, but they could start there and see how it goes.

Thanks for reading,

A.