LLM -Supported Natural Language to Bash Translation (2025)

Abstract: The Bourne-Again Shell (Bash) command-line interface for Linux systems has complex syntax and requires extensive specialized knowledge. Using the natural language to Bash command (NL2SH) translation capabilities of large language models (LLMs) for command composition circumvents these issues. However, the NL2SH performance of LLMs is difficult to assess due to inaccurate test data and unreliable heuristics for determining the functional equivalence of Bash commands. We present a manually verified test dataset of 600 instruction-command pairs and a training dataset of 40,939 pairs, increasing the size of previous datasets by 441% and 135%, respectively. Further, we present a novel functional equivalence heuristic that combines command execution with LLM evaluation of command outputs. Our heuristic can determine the functional equivalence of two Bash commands with 95% confidence, a 16% increase over previous heuristics. Evaluation of popular LLMs using our test dataset and heuristic demonstrates that parsing, in-context learning, in-weight learning and constrained decoding can improve NL2SH accuracy by up to 32%. Our findings emphasize the importance of dataset quality, execution-based evaluation and translation method for advancing NL2SH translation. Our code is available at https://github.com/westenfelder/NL2SH

## introduction the default command-line interface (cli) for interacting with linux systems is the bourne-again shell (bash) @xcite . bash commands allow computer users to control processes, interact with the file system and manage the network. however, using bash requires knowledge of numerous utilities, each with unique parameters and complex syntax @xcite . moreover, the reference documentation for these utilities, called manual pages, can be cumbersome and confusing @xcite . this makes the cli a barrier for inexperienced users and increases the chance of errors for experienced users @xcite . language models that convert natural language to command-line instructions, a task referred to as nl2sh, nl2cmd or nl2bash translation, offer a promising solution to this problem @xcite . we use the term nl2sh model to refer to models trained specifically for the task of nl2sh, as well as the nl2sh capabilities of general-purpose large language models (llms). figure 1 shows an example of natural language to bash command translation. nl2sh models are well suited for clis because they are designed for text-based interactions. nl2sh models can simplify human-computer interactions by allowing users to interact with linux systems through natural language on the command line. this advancement enhances usability by reducing the need for syntax memorization @xcite . input: natural language list files in the /workspace directory that were accessed over an hour ago. output: bash command find /workspace -type f -amin +60 the use of nl2sh models necessitates benchmarks to measure task accuracy @xcite @xcite @xcite . a nl2sh benchmark requires test data consisting of natural language prompts and ground truth commands (referred to as instruction-command pairs). given a natural language prompt, a nl2sh model generates a bash command. a benchmark must then use a heuristic to determine if the model command is functionally equivalent to the ground truth command. determining functional equivalence of commands is difficult because there are multiple possible correct com-mands for a given task, due to a wide range of interchangeable utilities. further, command execution may not result in identical outputs, neutralizing evaluation with string comparison. current benchmarks do not accurately measure nl2sh model performance due to errors in assessment data and inaccurate heuristics for determining the functional equivalence of commands @xcite @xcite . this makes it difficult to assess model capabilities and measure methods for improving model performance. to address these challenges, we investigate the following research questions: (1) how can we validate nl2sh datasets to ensure models are evaluated using accurate assessments? (2) how can we design a functional equivalence heuristic that accurately measures the quality of model translations? (3) how can we improve the accuracy of nl2sh models as measured by a reliable benchmark? our contributions are summarized as follows: (1) we create a manually verified test dataset of 600 instruction-command pairs and a training dataset of 40,939 pairs, increasing the size of previous test and training datasets by 441% and 135%, respectively. (2) we present a novel functional equivalence heuristic that combines command execution with llm evaluation of command outputs, capable of determining the functional equivalence of two bash commands with 95% confidence, a 16% increase over previous heuristics. (3) we evaluate popular llms using our test data and heuristic and demonstrate that parsing, in-context learning, in-weight learning and constrained decoding can improve nl2sh accuracy by up to 32%. ## background nl2sh translation falls under the broader domain of machine translation, where models automatically translate text or speech from one language to another. llms are well suited for this task, enabling translation that was impossible with previous methods @xcite . evaluating nl2sh models requires determining the functional correctness of generated commands. functional correctness is defined as whether the code produces the correct output for each input, as specified, or as compared to ground truth @xcite . functional correctness does not consider the diversity of code generated, or other factors such as run time and memory consumption @xcite . ensuring the functional correctness of code is difficult because validation methods are error-prone and take an impractical amount of time for large volumes of code @xcite . there are two main validation techniques: static and dynamic analysis. static analysis checks code without execution, using parsers, lexical analysis or control flow checking @xcite . dynamic analysis evaluates code outputs and runtime behavior using an execution environment @xcite . some frameworks combine static and dynamic analysis @xcite . determining the functional correctness of bash commands translated from natural language is a sub-problem of validating code correctness. we assess the functional correctness of bash commands by comparing the generated (model) command with a ground truth bash command. we define the term "functional equivalence heuristic" (feh) to describe a heuristic that performs this comparison and determines the functional correctness of a bash command. due to varying definitions in this field, we provide notation in ## conclusion in this paper, we explore applications for llms in nl2sh translation and benchmarking. we identify issues with current benchmarks, including inaccurate datasets and unreliable functional equivalence heuristics. to address these problems, we correct and expand nl2sh datasets and create a new heuristic to determine the functional equivalence of bash commands. we assess our heuristic and find that bash command execution paired with language model evaluation of command outputs can determine the functional equivalence of commands more accurately than previous heuristics. using our dataset and heuristic, we evaluate how constrained decoding, parsing, in-context learning and in-weight learning impact the performance of llama, qwen and gpt models. we find that parsing and in-context learning reliably improve the performance of open and closed-source llms for the task of nl2sh translation. ultimately, we find that nl2sh translation is a difficult task for llms, necessitating further research in this field. in future work, we plan to investigate efficient means to verify our training dataset and conduct more fine-tuning experiments.