Explainable Natural Language to Bash Translation using Abstract Syntax Tree (2021)

Abstract: Natural language processing for program synthesis has been widely researched. In this work, we focus on generating Bash commands from natural language invocations with explanations. We propose a novel transformer based solution by utilizing Bash Abstract Syntax Trees and manual pages. Our method incorporates tree structure information in the transformer architecture and provides explanations for its predictions via alignment matrices between user invocation and manual page text. Our method performs on par with the state of the art performance on Natural Language Context to Command task and performs better than fine-tuned T5 and Seq2Seq models.

## introduction natural language processing (nlp) of programming language source code is a well-established field with a recent resurgence of using nlp techniques to assist programming. advances have been made in code generation @xcite , code summarization @xcite , bug detection @xcite , code translation @xcite etc. however, there is a lack of explainable methods for assisting software developers. bash is a unix command language with numerous utilities (like cut, tr, sed etc) for interacting with the operating system. it is difficult for a novice developer to remember the purpose and syntax of these bash utilities. usually, to understand the intended usage of a utility, one looks up relevant manual page using man, apropos and info. however, these pages have been described as "frequently hard to read", byzantine and formal @xcite and finding relevant information from them itself becomes a challenging task. hence it gets difficult for a new user to learn bash. previous works in this area @xcite @xcite have explored transformer @xcite , sequence to sequence with attention @xcite (seq2seq), grammar guided methods and retrieval based methods for translating natural language to bash commands. however, these methods are predictive and do not provide explanations or reasoning for their predictions. these methods propose a black-box solution to the translation problem relying on only parallel data. there is a lack of user trust in such predictions. motivated by these challenges, we propose an explainable neural machine translation model. to the best of our knowledge, ours is the first method to explore the use of manual page data for assisting translation. besides translating natural language invocations to bash code, our model also provides alignment matrices between user invocations and manual text, which explain the predictions by the model. such alignments could be helpful to gain developer confidence in using the system. at the same time, it can help improve the learning curve for bash while providing a more user-friendly interface with the operating system. we apply our method to nlc2cmd challenge @xcite and compare the results with baselines like t5 @xcite , seq2seq and the winning solution to nlc2cmd challenge @xcite . our method performs better than t5 and seq2seq while providing explanations for the predictions. code for our model is available on github foot_0 . we also release parsed and cleaned utility descriptions from linux manuals and tldr foot_1 (a community-contributed collection of utility descriptions) which were used in our experiments. problem statement assume that i is the set of all natural language invocations, c is the set of all bash commands without pipes, processsubstitution and command-substitution, u is the set of all bash utilities and s is the set of all sentences from manual pages of bash commands. given a dataset d := {p, m }, where for example, @xmath0. @xmath1, x, to represent row vectors, upper case bold characters, w, to represent two dimensional matrices, italic characters, s, to represent scalars and upper case bold-italic characters, x, to represent sequence of row vectors. subscripts, as in x i , represent i th coordinate of the vector x while superscript, as in x i , are used to index vectors in a sequence. the set of positive even numbers is denoted by n e and the set of positive odd numbers is denoted by n o . element-wise product of two vectors is denoted by . the inner product of two vectors is denoted by •, • and the frobenius inner product of two matrices is denoted by •, • f . ## method we model the problem as a natural language to bash abstract syntax tree (ast) translation task by transforming a bash command to an ast. our method is motivated by human cognition while programming. a developer usually looks up information from documentation while performing a programming task. similarly, our model uses a convolutional neural network to match invocation against manual page data. it then generates a bash command guided by manual page information. to incorporate the information at the correct time-step during the command generation process, we utilize abstract syntax tree (ast) for bash. ## results and analysis results are mentioned in ## ablation study we propose three components, namely tree coordinates (section 2.1), parent attention (section 2.2) and guidance module (section 2.3). in shiv and quirk (2019) design sinusoidal embeddings for incorporating tree structure. their method works only for binary trees. @xcite propose a method to jointly learn code context and structure of source code for code summarization task. @xcite propose several ways of communicating code structure to transformer for code completion task. @xcite present an analysis of convolutional neural networks for text. they show that filters in such networks capture different semantic classes of n-grams. ## future work in future work, we would experiment with different sequential representations of the ast. linux manpage is a rich source of information for this task. exploring it further might be useful. an interesting idea is to split the invocations into constituent tasks. planned prediction, proposed in @xcite , is also an exciting idea to explore. ## conclusion this work explores the use of manual page data for natural language to bash translation and proposes a cognitively inspired transformer based architecture for generating abstract syntax trees for bash. our method also explains its predictions by alignment matrices between user invocation and manual page text. additionally, we obtain insights about the distinguishing features of each utility from these alignment matrices. we test our method on nlc2cmd data.