S hellcode_ IA 32: A Dataset for Automatic Shellcode Generation (2021)

Abstract: We take the first step to address the task of automatically generating shellcodes, i.e., small pieces of code used as a payload in the exploitation of a software vulnerability, starting from natural language comments. We assemble and release a novel dataset (Shellcode_IA32), consisting of challenging but common assembly instructions with their natural language descriptions. We experiment with standard methods in neural machine translation (NMT) to establish baseline performance levels on this task.

## introduction and related work a growing body of research has dealt with automated code generation: given a natural language description, a code comment or intent, the task is to generate a piece of code in a programming language @xcite . the task of generating programming code snippets, also referred to as semantic parsing @xcite , has been previously addressed to generate executable snippets in domain-specific languages @xcite , and several programming languages, including python @xcite and java @xcite . we consider the task of generating shellcodes, i.e., small pieces of code used as a payload to exploit software vulnerabilities. shellcoding, in its most literal sense, means writing code that will return a remote shell when executed. it can represent any byte code that will be inserted into an exploit to accomplish the desired, malicious, task @xcite . an example of a shellcode program in assembly language and the corresponding natural language comments are shown in listing 1. shellcodes are important because they are the key element of security attacks: they represent code injected into victim software to take control of 1 global _start; declare global _start. 2 section .text; declare the text section. define the _start label. 4 xor eax, eax; and push its contents on the stack. 6 push 0x68732f2f;move /bin//sh 7 push 0x6e69622f;into the ebx register. 8 mov ebx, esp 9 push eax; put the system call 11 into the al register. 14 int 0x80; make the kernel call. listing 1: x86 assembly code used to spawn /bin/sh shell on linux os. lines 4-5, 6-7-8, 9-10, 11-12 are multi-line snippets generated by four different intents. a machine, to escalate privileges, and to use the machine for malicious purposes such as ddos attacks, data theft, and running malware @xcite . well-intentioned actors (security practitioners and product vendors) also develop shellcodes to run non-harmful proof-of-concept attacks, to show how security weaknesses can be exploited to identify vulnerabilities and patch systems. thus, shellcode generation using (semi-) automated techniques has become a popular and very active research topic @xcite . however, writing shellcodes is technically challenging since they are typically written in assembly language (c.f. listing 1). the most sophisticated shellcodes can reach hundreds of assembly lines of code. the task of the shellcode generation has been addressed by several works and tools. @xcite designed shellswap, a system that can modify an observed exploit and replace the original shellcode with an arbitrary replacement shellcode. the system uses symbolic tracing, with a combination of shellcode layout remediation and path kneading to achieve shellcode transplants. pwntools (pwntools, accessed: 2021-05-29) is a ctf framework and exploit development library written in python. it is designed for rapid prototyping and development and intended to make exploit writing as simple as possible. differently from previous work in the security literature, we approach this problem as a machine translation (nmt) task. we apply neural machine translation @xcite , which unlike the traditional phrase-based translation system consisting of many small sub-components tuned separately, attempts to build and train a single, large neural network that reads a sentence and outputs a correct translation @xcite . nmt has emerged as a promising machine translation approach, showing superior performance on public benchmarks @xcite , and it is widely recognized as the premier method for the translation of different languages @xcite . nmt has also been used to perform complex tasks on the unix operating system shell @xcite (e.g. file manipulation and search), by stating goals in english @xcite , to automatically generate commit messages @xcite , etc. however, the nmt techniques have not heretofore been adopted to automatically generate software exploits from natural language comments. since nmt is a data-driven approach to code generation, we need a dataset of intents in natural language, and their corresponding translation (in our context, in assembly language) for shellcode generation. in this preliminary work, we address the lack of such a dataset by presenting shellcode ia32, a dataset containing 3, 200 lines of assembly code extracted from real shellcodes and described in the english language. moreover, we present experiments on our dataset using a baseline technique, in order to establish performance levels for evaluating shellcode generation techniques. ## dataset we compiled a dataset, shellcode ia32, specific to our task. this dataset consists of 3,200 examples of instructions in assembly language for ia-32 (the 32-bit version of the x86 intel architecture) from publicly-available security exploits. we collected assembly programs used to generate shellcode from shell-storm (shellcodes database for study cases, accessed: 2021-04-22) and from exploit database (exploit database shellcodes , accessed: 2021-04-22), in the period between 2000 and 2020. our focus is on linux, the most common os for security-critical network services. accordingly, we added assembly instructions written with netwide assembler (nasm) for linux @xcite . nasm is line-based. figure 1 shows a simple example of a nasm source line. every source line contains a combination of four fields: an optional label used to represent either an identifier or a constant, a mnemonic or instruction, which identifies the purpose of the statement and followed by zero or more operands specifying the data to be manipulated, and an optional comment, i.e., text ignored by the compiler. a mnemonic is not required if a line contains only a label or a comment. to take into account the variability of descriptions in natural language, multiple authors described independently different samples of the dataset in the english language. where available, we used as natural language descriptions the comments written by developers of the collected programs. we enriched the dataset by adding examples of assembly programs for the ia-32 architecture from popular tutorials and books @xcite @xcite to understand how different authors and assembly experts describe the code and, thus, how to deal with the ambiguity of natural language in this specific context. our dataset consists of ∼ 10% of instructions collected from books and guidelines and the rest from real shellcodes. multi-line snippets: to automatically generate shellcodes, we need to look beyond a one-to-one mapping between a line of code and its comment/intent. for example, a common operation in shellcodes is to save the ascii "/bin/sh" into a register. this operation requires three distinct assembly intent: jump short to the decode label if the contents of the al register is not equal to the contents of the cl register else jump to the shellcode label multi-line snippets: cmp al, cl \n jne short decode \n jmp shellcode intent: jump to the label recv http request if the contents of the eax register is not zero else subtract the value 0x6 from the contents of the ecx register multi-line snippets: test eax, eax \n jnz recv http request \n sub ecx, 0x6 we also note here that existing code generation datasets do contain a larger, potentially noisy, subset of training examples (ranging in several thousand) obtained by mining the web. for example, the conala mined (as opposed to the conala annotated) dataset contains 598,237 training examples mined directly from stack overflow @xcite . in our case, although shellcodes are written in assembly language, it is not feasible to simply mine examples of natural language-assembly from the web: not all assembly programs are shellcodes. thus, our shellcode ia32 dataset, which contains ∼ 20 years of shellcodes from a variety of sources is the largest collection of shellcodes in assembly available to date. ## preliminary evaluation we performed a set of preliminary experiments with our dataset, in order to assess the applicability @xcite using xnmt @xcite . we use an adam optimizer (kingma and ba, 2015) with β 1 = 0.9 and β 2 = 0.999. the last step is inference. during inference, the auto regressive inference component uses beam search with a beam size of 5. the train/dev/test split is train (@xmath0, dev (@xmath1, and test (@xmath2. the test set includes 44 multi-line snippets (13.75% of the test set). following prior work in this area @xcite @xcite , we evaluate the translation performance in terms of averaged token level bleu scores @xcite . bleu uses the modified form of n-grams precision and length difference penalty to evaluate the quality of the output generated by the model compared to the referenced one. bleu measures translation quality by the accuracy of translating ngrams to n-grams, for values of n usually ranging between 1 and 4 @xcite . we measure the performance of the evaluation task also in terms of exact match accuracy (acc), which is the fraction of exactly matching samples between the predicted output and the reference @xcite . both metrics range between 0 and 1. during our experiments, we set a basic configuration of the model: α = 0.001, @xmath3, vocabulary @xmath4, 000, epochs (with early stopping enforced) = 200, beam @xmath5, minimun word @xmath6. next, we performed experiments by varying the dimensionality of the layers from 64 to 1024, and the number of layers from 1 to 4 while keeping all other hyper-parameters constant. all experiments were performed on a linux os running on a virtual machine with 8 cpu cores and 8 gb ram. the computational times are highly dependent on the model hyper-parameters, and range between few minutes to ∼ 105 minutes, with the average training time equal to ∼ 28 minutes. ## qualitative analysis automated metrics (bleu and accuracy) provide a somewhat limited window into the efficacy of the models to accomplish our task: the task of automatically generating assembly code from natural language intents. we conducted a qualitative analysis of the outputs to address this issue and present our findings through cherry-and lemon-picked examples from our test set ( natural language ground truth model output put ascii /bin/sh into eax push 0x68732f2f \n push 0x6e69622f \n mov eax , esp push 0x68732f2f \n push 0x6e69622f \n mov eax, esp place address buff into esi mov esi, buff lea esi, [buff] perform a bit-wise inversion of edx not edx and edx, 0 if the contents of the bl register is greater than or equal to the value 78h then jump to the memory location loc 402b1d cmp bl, 78h \n jge short loc 402b1d cmp bl, 78h \n jle short loc 402b1d we also highlight problems with the models through illustrative examples of failure outputs (rows 3 and 4, ## ethical considerations recognizing that attackers use exploit code as a weapon, it is important to specify that the goal of the proof-of-concept (poc) exploits is not to cause harm but to surface security weaknesses within the software. identifying such security issues allows companies to patch vulnerabilities and protect themselves against attacks. offensive security is a sub-field of security re-search that employs ethical hackers to probe a system for vulnerabilities or can be a technique used to disrupt an attacker. automatic exploit generation (aeg), an offensive security technique, is a developing area of research that aims to automate the exploit generation process and to explore and test critical vulnerabilities before they are discovered by attackers @xcite . indeed, studying exploits on compromised systems can provide valuable information about the technical skills, degree of experience, and intent of the attackers who developed or used them. using this information, it is possible to implement measures to detect and prevent attacks @xcite . ## conclusion we address the problem of automated exploit generation through nlp. we use neural machine translation to translate the natural language intents into assembly code. the contribution in this work is a new dataset, shellcode ia32, containing 3, 200 pairs of instructions in assembly language code snippets and their corresponding intents in english. these assembly language snippets can be combined together to generate attacks or exploits on linux os running on intel architecture 32-bit machines. shellcode ia32 represents a first step towards the ambitious goal of automatically generating shellcodes from natural language. our experimental evaluation has shown promising early results, demonstrating the feasibility of generating assembly code instructions with high accuracy.