P rogramming in N atural L anguage with fu SE : S ynthesizing M ethods from S poken U tterances U sing D eep N atural L anguage U nderstanding (2020)

Abstract: The key to effortless end-user programming is natural language. We examine how to teach intelligent systems new functions, expressed in natural language. As a first step, we collected 3168 samples of teaching efforts in plain English. Then we built fuSE, a novel system that translates English function descriptions into code. Our approach is three-tiered and each task is evaluated separately. We first classify whether an intent to teach new functionality is present in the utterance (accuracy: 97.7% using BERT). Then we analyze the linguistic structure and construct a semantic model (accuracy: 97.6% using a BiLSTM). Finally, we synthesize the signature of the method, map the intermediate steps (instructions in the method body) to API calls and inject control structures (F1: 67.0% with information retrieval and knowledge-based methods). In an end-to-end evaluation on an unseen dataset fuSE synthesized 84.6% of the method signatures and 79.2% of the API calls correctly.

## introduction intelligent systems became rather smart lately. one easily arranges appointments by talking to a virtual assistant or controls a smart home through a conversational interface. instructing a humanoid robot in this way no longer seems to be futuristic. for the time being, users can only access built-in functionality. however, they will soon expect to add new functionality themselves. for humans, the most natural way to communicate is by natural language. thus, future intelligent systems must be programmable in everyday language. today's systems that claim to offer programming in natural language enable laypersons to issue single commands or construct short scripts (e.g. @xcite ; @xcite ); usually no new functionality is learned. only a few addressed learning new functionality from natural language instructions (e.g. @xcite ; @xcite ). however, even recent approaches still either restrict the language or are (over-)fitted to a certain domain or application. we propose to apply deep natural language understanding to the task of synthesizing methods from spoken utterances. our approach combines modern machine learning techniques with information retrieval and knowledge-based methods to grasp the user's intent. as a first step, we have performed a user study to investigate how laypersons teach new functionality with nothing but natural language. in a second step, we develop fu se (function synthesis executor). fu se translates teaching efforts into code. on the basis of the gathered data we constructed a three-tiered approach. we first determine, whether an utterance comprises an explicitly stated intent to teach a new skill. then, we decompose these teaching efforts into distinct semantic parts. we synthesize methods by transferring these semantic parts into a model that represents the structure of method definitions. finally, we construct signatures, map instructions of the body to api calls, and inject control structures. ## related work the objective of programming in natural language was approached from different perspectives over the years. quite a few approaches are natural language interfaces to code editors @xcite @xcite . however, they assume that users literally dictate source code. thus, these approaches are intended for developers rather than laypersons. other approaches such as voxelurn by @xcite aim to naturalize programming languages to lower the hurdle for programming novices. approaches for end-user programming in natural language take up the challenge of bridging the semantic gap between informal spoken or written descriptions in everyday language and formal programming languages. early systems were syntaxbased @xcite @xcite @xcite . some were already capable to synthesize short scripts including control structures and comments, e.g. nlp for nlp by @xcite . others take the user in the loop and create scripts with a dialog-driven approach @xcite . in further developments intelligent assistants offer their service to assist with programming @xcite . often these assistants support multi-modal input, e.g. voice and gestures @xcite . others combine programming in natural language with other forms of end-user programming, such as programming by example @xcite or programming by demonstration @xcite . some authors such as @xcite and @xcite ) take a knowledgebased approach by integrating domain and environmental information in the form of ontologies. @xcite employ a neural network to learn a situational context model that integrates the system environment and the human-systeminteraction, i.e. the dialog. many recent approaches integrate semantic parsing in the transformation process @xcite @xcite . even though the natural language understanding capabilities are often impressive, the synthesized scripts are still (semantically) erroneous in most cases. additionally, learning of new functionality is not covered by approaches of that category so far. programming in natural language is of particular interest in the domain of humanoid robotics @xcite @xcite . people expect to teach them as they teach human co-workers. therefore, some authors, e.g. @xcite , use task descriptions that were intended to instruct humans to benchmark their approach. however, often the assumed vocabulary is rather technical @xcite . thus, the usability for laypersons is limited. ## approach the goal of our work is to provide a system for programming in (spoken) natural language. laypersons shall be enabled to create new functionality in terms of method definitions by using natural language only. we offer a general approach, i.e. we do not restrict the natural language regarding wording and length. since spontaneous language often comprises grammatical flaws, disfluencies, and alike, our work must be resilient to these issues. we decompose the task in three consecutive steps. the rationale behind this decision is as follows. on the one hand, we can implement more focused (and precise) approaches for each task, e.g. using machine learning for one and information retrieval for another. on the other hand, we are able to evaluate and optimize each approach individually. the stages of our three-tiered approach are the following (see figure 1 for an example): the first two stages are classification problems. thus, we apply various machine learning techniques. the first stage is a sequence-to-single-label task, while the second is a typical sequence-tosequence task. for the first we compare classical machine learning techniques, such as logistic regression and support vector machines, with neural network approaches including the pre-trained language model bert @xcite . for the second task we narrow down to neural networks and bert. a more detailed description of the first two stages may be found in @xcite . the implementation of the third stage is a combination of syntactic analysis, knowledge-based techniques and information retrieval. we use semantic role labeling, coreference analysis, and a context model @xcite to infer the semantic model. afterwards, we synthesize method signatures heuristically and map instructions from the body to api calls using ontology search methods and datatype analysis. additionally, we inject control structures, which we infer from keywords and syntactic structures. to cope with spontaneous (spoken) language, our approach relies on shallow nlp techniques only. ## evaluation to measure the performance of fu se on unseen data, we set up an end-to-end evaluation. we created two new scenarios. they take place in the kitchen setting again, but involve different actions and objects. in the first, subjects are supposed to teach the robot, how to start the dishwasher and in the second, how to prepare cereals. once more we used prolific to collect the data and set the number of participants to 110. however, we accepted only 101 submissions, i.e. 202 descriptions. we randomly drew 50 descriptions each. since the evaluation of the overall approach entails the same output as the third stage, we prepared the gold standard like in subsection 3.4 and used the same ontologies. in the end-to-end evaluation our approach synthesized 73 method signatures; five were missed due to an incorrect first-stage classification. out of 73 synthesized methods we assessed seven to be inappropriate. additionally, 36 parameters were mapped correctly and no false positives were created. except for the missing method signatures the results are in line with the third-stage evaluation. the results for the method body synthesis, as depicted in to assess how well fu se generalizes to truly spoken utterances we evaluated on another dataset. it is a collection of recordings from multiple recent projects. the setting (instructing a humanoid robot in a kitchen setting) is the same. however, none of the scenarios involved teaching new functionality. thus, we can only measure fu se 's ability to construct scripts. the descriptions in this dataset comprise control structures to a much larger extent. altogether the dataset comprises 234 recordings and manual transcriptions. the 108 subjects were mostly under-graduate and graduate students. on the transcripts we assess the mapping of methods and parameters individually. the results for both and entire calls are depicted in ## conclusion we have presented fu se , a system for programming in natural language. more precisely, we aim to enable laypersons to teach an intelligent system new functionality with nothing but spoken instructions. our approach is three-tiered. first, we classify whether a natural language description entails an explicitly stated intent to teach new functionality. if an intent is spotted, we use a second classifier to separate the input into semantically disjoint parts; we identify declarative and specifying parts and filter out superfluous information. finally, we synthesize method signatures from the declarative and method bodies from the specifying parts. method bodies contain instructions and control structures. instructions are mapped to api calls. we implemented the first two steps using classical machine learning and neural networks. teaching intents are identified with an accuracy of 97.7% (using bert). the classification of the semantics is correct in 97.6% of the cases (using a bilstm). we evaluated fu se on 100 descriptions obtained from a user study. the results are promising; fu se correctly synthesized 84.6% of the method signatures. the mapping of instructions in the body to api calls achieved an f 1 -score of 66.9%. in a second evaluation on a speech corpus the f 1 -score for api calls is 79.2%. we plan to evaluate fu se in other domains. it will be interesting to see, if we can reuse (or transfer) the machine learning models as well as the rest of the approach. future adoptions to fu se will include the integration of a dialog component. we may query the user in case of ambiguous statements or missing parameters. we have implemented an extensible dialog module and shown that it can be used to resolve ambiguous references, word recognition errors, and missing conditions @xcite . however, we still have to figure out, how to query users properly if an api mapping is ambiguous or parameters are missing. another improvement concerns the analysis of verb references. humans often refer to previous actions, which may cause superfluous instructions. we will also implement a sanity check that considers feasibility and meaningfulness of the sequence of actions in the method body. the latter may involve a feedback mechanism via the dialog component. giving feedback to newly learned method definitions that may be lengthy and therefore unhandy to repeat as a whole is an interesting challenge.