Transformer with Tree-order Encoding for Neural Program Generation
Anonymous ACL submission
Abstract
While a considerable amount of semantic parsing approaches have employed RNN architectures for code generation tasks, there have beenonly few attempts to investigate the applicability of Transformers for this task.
Including hierarchical information of the underlying programming language syntax has proven to be effectivefor code generation.
Since the positional encoding of the Transformer can only represent positions in a flat sequence, we have extended theencoding scheme to allow the attention mechanism to also attend over hierarchical positionsin the input.
Furthermore, we have realized adecoder based on a restrictive grammar graphmodel to improve the generation accuracy andensure the well-formedness of the generatedcode.
While we did not surpass the state ofthe art, our findings suggest that employing atree-based positional encoding in combinationwith a shared natural-language subword vocabulary improves generation performance oversequential positional encodings.
Introduction
Automatically generating source code from instructions in natural language can reduce the humanmachine language barrier.
Efforts on overcomingthis barrier have led to numerous semantic parsing approaches ranging from statistical semanticparsing with focus on inducing rules (Zelle and Mooney, 1996) or probabilistic grammars (Zettlemoyer and Collins, 2005) to neural semantic parsing approaches based on encoder-decoder architectures, which have proven to be effective in mapping natural language into a formal meaning representation such as logical forms (Jia and Liang,2016; Suhr et al., 2018) or general-purpose programming languages (GPPL) (Ling et al., 2016; Iyer et al., 2018;
Yin and Neubig, 2017, 2018; Iyeret al., 2019).
Recent syntax-driven neural semantic parsing approaches integrated compositional structures of logical forms (Dong and Lapata, 2016) or, in the case of GPPL, the underlying syntax (Rabinovich et al., 2017; Yin and Neubig, 2017;
Hayati et al., 2018; Sun et al., 2019;
Xu et al., 2020), which substantially improved generation accuracy.
However, existing grammar-constrained decoding approaches for a GPPL like Python either use a manually defined list of production rules (Yin and Neubig, 2017) or the ASDL (Wang et al., 1997) abstract grammar definition (Python Software Foundation, 2018) used by the CPython implementation of the Python language (Rabinovich et al., 2017;
Yin and Neubig, 2018).
The manual approach allows for flexibility in modeling, but requires manually checking and possibly re-modeling the rules with every new release, which is prone to human error.
Although the ASDL eliminates these drawbacks, it is not sufficiently restrictive and therefore cannot guarantee the generation of well-formed, i.e., parsable code.
In compilers and interpreters, this is not problematic because some component in the pipeline consisting of parser, static checker and dynamic checker catches each of these types of error (Aho et al., 2007).
In neural semantic parsing, however, it unnecessarily allows the generation of invalid code.
Building on this finding, we propose a syntaxdriven neural semantic parsing approach employing a more restrictive grammar model for the task of generating well-formed code from instructions or descriptions in natural language as depicted in Table 1.
Most syntax-driven neural semantic parsing approaches employ encoder-decoder architectures based on RNNs (Dong and Lapata, 2016; Rabinovich et al., 2017;
Hayati et al., 2018; Iyer et al., 2019, i.a.).
With the Transformer (Vaswani et al., 2017), a more powerful generation of neural architectures was introduced, surpassing RNNs in a variety of NLP tasks, including machine transla-
Dataset
NL Specification
Formal Meaning Representation
Atis flight from ci0 mn0 dn ( lambda $0 e ( and ( flight $0 ) ( from $0 ci0 ) ( day number $0 dn0 ) ( month $0 mn0 ) )
Geo
What is the area of the state with the smallest population density?
( area:i ( argmin \ $ ( state:t \ $0 )
( density:i \ $0 ) ) )
CoNaLa
How to load a csv file?
import csv with open(’file.csv’) as csvfile: reader = csv.
DictReader(csvfile)
Hearthstone
Treant NAME END 2 ...
Minion TYPE END
Druid PLAYER CLS END
NIL RACE END NIL
RARITY END NIL class Treant(MinionCard): def init (self): super().
init (”Treant”, 1,
CHARACTER CLASS.DRUID,
CARD RARITY.COMMON) def create minion(self, ): return Minion(2, 2) tion (Nguyen et al., 2020; Chen et al., 2018) andconstituency parsing (Wang et al., 2019; Hareret al., 2019).
Unlike RNNs, Transformers relysolely on self-attention and therefore can capturelong-range context dependencies while being easyto parallellize.
Since the Transformer is invariant to sequenceordering, it is of particular importance to explicitlyinclude position information of sequence tokensthrough learned or fixed positional encodingschemes (Vaswani et al., 2017;
Shiv and Quirk,2019) or biased attention weights (Shaw et al.
,2018; Nguyen et al., 2020).
While the inclusion ofhierarchical information has proven to be effectivefor code generation, the positional encoding by(Vaswani
et al., 2017) can only represent positionsin a flat sequence.
Taking up this thought, wepropose a position encoding scheme for the
Transformer which allows the attention mechanismto also attend over hierarchical positions in theinput.
Similar to (Shiv and Quirk, 2019), we alsorely on paths in an abstract tree representation ofinput code snippets to encode hierarchical codestructure.
However, (Shiv and Quirk, 2019)’sencoding, cannot be applied to trees of variablewidth since the depth and width of the encoding islimited.
Our contributions are the following:
1) A restrictive grammar graph model, which
can be automatically generated from any tree-
like data, not requiring an ASDL with the
only condition that ordered typed trees are
available as input.
The grammar model is
used to determine the hierarchical relations
for the tree encoding, but can also be used for
constrained decoding, ensuring the generation
of parsable code.
A tree encoding scheme based on the sinu-
soidal encoding introduced by (Vaswani et al.,
2017) to enable the Transformer model to
learn hierarchical relations in an ordered tree
of arbitrary width.
We evaluated our approach on several benchmark datasets for code generation including Hearthstone (Ling et al., 2016), CoNaLa (Yin et al., 2018), Atis (Dahl et al., 1994) and Geo (Tang and Mooney, 2001).
Experimental results show that he tree encoding performs better than the sequential encoding used by the original Transformer architecture on the Hearthstone dataset.
On the CoNaLa dataset, we achieve improvements when employing a separate subword vocabulary for the string literals extracted from the code snippets.
Our code is publicly available at https: //github.com/REMOVED_FOR_REVIEW .
Related Work
A large variety of syntax-driven neural semantic parsing approaches generate abstract syntax trees (ASTs) by predicting a sequence of grammar rules.
Yin et al.
(Yin and Neubig, 2017, 2018), Rabinovich et al.
(Rabinovich et al., 2017) and Sunet al.
(Sun et al., 2019) define a grammar modelthat captures the syntax of the target programminglanguage and generate ASTs based on a series ofpredicted actions, i.e., apply a production rule toexpand a non-terminal node or populate a terminal node.
Most of these approaches derive the underlying grammar model from the official Pythonabstract grammar definition (ASDL).
However, abstract grammar definitions are designed for compilation or interpretation (Aho et al., 2007) and notfor code generation and therefore, as in the caseof the Python grammar, allow syntax trees that donot correspond to parsable code.
We counteractthis drawback by creating a more restrictive grammar model from the parsable code samples of thetraining dataset.
Based on this grammar model, wegenerate ASTs by predicting the next AST node onthe input tree path in depth-first preorder.
To capture the compositional structure of logical forms, (Dong and Lapata, 2016) propose a treedecoder that recursively generates sub-trees, conveying hierarchical information by feeding the parent node’s hidden representation as an additionalinput.
Expanding on this, (Rabinovich et al., 2017)realize a grammar-based modular decoder with adynamically determined composition of modulesthat reflect the structure of ASTs.
(Sun et al., 2020)propose a grammar dependent Transformer architecture for code generation and use convolutionallayers to process natural language input and modelhierarchical structure.
In contrast, we convey thecompositional structure through the attention mechanism aided by a hierarchical positional encoding,not through parent feeding.
Since we do not employ a grammar dependent architecture, with constrained decoding disabled, our approach can beapplied to any other tree-structured data for whichno abstract grammar is available.
For encoding nodes in a tree, we use the samenotion of path as (Shiv and Quirk, 2019), and liketheirs, our tree encoding for a path can be described as a composition of affine transformations.
However, with (Shiv and Quirk, 2019)’s encoding,there is only one affine transformation per path segment encoding, which might limit the robustnessof the encoding.
While their tree encoding of anode results from concatenating the one-hot encodings of the node’s reversed path segments andweighting the encodings using a learned parameter,we employ the parameter-free sinusoidal encoding by (Vaswani et al., 2017) to encode each path segment.
Since the tree depth and width of (Shiv and
Quirk, 2019) encoding is fixed, it is not possible to apply it on arbitrarily wide trees, and sibling relationships can only be represented in binary trees.
These limitations do not apply to our encoding scheme, which can be applied to ordered trees of variable width.
Data Representation
3. Input Data
As input for training, we use several Python code corpora (Ling et al., 2016;
Yin et al., 2018; Oda et al., 2015) consisting of pairs of natural language instructions and corresponding code snippets.
We parse the Python code snippets into abstract syntax trees (ASTs) and generate a sequence of AST tokens in depth-first preorder, as depicted in Figure 1a.
The AST consists of object nodes (e.g. Module ) and attribute nodes visualised by small squares and circles in the figure.
An attribute node is either a singleton (e.g. value ) or an attribute list (e.g. body ) that can point to several object nodes.
We integrated a special node for marking an attribute list end ( le ) and special nodes for marking the start ( sos ) and the end ( eos ) of the code fragment.
An object node can be an inner node (e.g.
Module ) or a leaf node (e.g. ’a’ ) and can have none, one or more attribute nodes.
Each attribute node has a defined order in its object node’s list of attribute nodes.
An edge from an object node to an attribute indicates that the attribute belongs to that object node, and an edge from an attribute to an object node means that instances of that object node are allowed to appear as the value of the attribute.
We create the sequential representation of an
AST by traversing the AST in depth-first preorder and adding the names of the object nodes to the
AST sequence.
An AST sequence consists of object node tokens and special nodes tokens.
If an object node is the last or the only node in an attribute list, we add a list end token after the object node’s token in the AST sequence.
From the ASTs of the code samples corpora only used for training, we create a grammar model that captures the Python programming language syntax.
From the natural language input and the string literals extracted from the ASTs we create a common subword vocabulary using the BPE implementation of (Kudo and
Richardson, 2018).
From the tokens from the AST
Module sos
Assign
Name ’a’ id n target start body end eos value
Num le le    2          (a) AST
Node
Path sos [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Module [1, 1, 0, 0, 0, 0, 0, 0, 0, 0]
Assign [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]
Name [1, 1, 1, 1, 1, 1, 0, 0, 0, 0] ’a’ [1, 1, 1, 1, 1, 1, 1, 1, 0, 0] le [2, 1, 1, 1, 1, 1, 0, 0, 0, 0]
Num [1, 2, 1, 1, 1, 1, 0, 0, 0, 0] [1, 1, 1, 2, 1, 1, 1, 1, 0, 0] le [2, 1, 1, 1, 0, 0, 0, 0, 0, 0] eos [1, 2, 0, 0, 0, 0, 0, 0, 0, 0]
[Figure]
sequence, we generate a different word vocabularywithout string literals.
Using a BPE segmentationfor string literals has the advantage that we caneffectively adjust the vocabulary size and reducethe number of tokens required to encode the inputsequences (Kudo, 2018).
3. Grammar Model
An ASDL grammar does not necessarily specifythe exact set of valid ASTs, but only a contextfree, parsing-optimized superset of ASTs.
As aresult, the grammar is not sufficiently restrictiveand therefore allows the generation of ASTs that donot correspond to any parsable program, or ASTsthat are parsable but would not withstand simplestatic checks.
For instance, the Python ASDL allows spaces within variable identifiers or constantstring or number literals on the left side of an assignment, which is an invalid syntax.
CPython’sparser (3.7) still allows constructs such as functioncalls or index operations with constants as targetsto be parsed into ASTs, e.g. ”1[2] = 3()”, resultingin a static check error.
As an improvement, we propose to generate a restricted grammar model that allows only AST compositions (i.e. attribute-child combinations) that occur in a corpus of Python code samples, e.g. thetrain dataset of a task.
The construction of thegrammar graph essentially induces a context-freegeneralization of the ASTs from the training datasetcontaining only parsable code samples.
This way,the likelihood of generating invalid ASTs can bereduced significantly.
The grammar graph defines asub-language of the language defined by the ASDL by construction, so the knowledge contained in the ASDL is still preserved in the grammar graph and not discarded.
Indeed, the set of ASTs is restricted to what occurs in the training data, but this will more likely prevent the model from predicting invalid rather than valid AST compositions.
The advantage of the restrictive grammar graph is that the model can only learn what it has seen in the data, ensuring the generation of parsable ASTs.
In any case, the model does not allow the prediction of AST combinations that it has not seen, i.e., that did not occur in the training data.
We define the grammar model as a bipartite labeled directed graph, whose nodes are the AST object nodes and attribute nodes.
The grammar graph, is used to compute edge paths for the tree encoding and can also be used to create constraint masks for limiting possible candidates from the current prediction during inference.
Tree Encoding
Since the Transformer model is invariant to the input order, due to its attention-based architecture, required position information needs to be explicitly included in the input sequence.
Thus, (Vaswani et al., 2017) encode positions in a sequence with a parameter-free scheme using sinusoidal functions of different frequencies and add these position encodings to the embeddings of the input sequence tokens of the first encoder and decoder layer.
However, flattening ASTs into an input sequence of AST tokens does not preserve the hierarchical  order between the tree nodes.
In particular, nodeswhose tokens are next to each other in the ASTsequence can be far apart in the tree, namely if thepreceding token in the AST sequence is the lastnode of a deep subtree.
Conversely, the tokens oftwo neighboring nodes in the AST are far apart inthe sequence when the first of two direct siblings isthe root of a large subtree.
To counteract this problem, we propose a treebased positional encoding built on (Vaswani et al.
,2017)’s
sinusoidal encoding scheme, which allowsthe attention mechanism to also attend over hierarchical positions in the input AST sequence.
4. Edge Paths
To define the tree encoding, we interpret the AST asan ordered tree T
= ( V T , E T , r T ) consisting of aset of nodes V T with n object nodes and m attributenodes, a set of edges E T and root r T .
A node v ∈ V T is uniquely identified by an edgepath p v of a configurable length L if the depth of
T is at most L .
An edge path is not a path as ingraph theory, i.e., a sequence of neighboring nodes,but rather structured like a path in a file system.
For each object and attribute node in the AST, theoutgoing edges receive a consecutively numberededge index idx .
For example in Figure 1a, thefirst edge of the object node Assign to the attributenode target has index 1 and the second edge to theattribute node value has index 2.
The edge path of a node consists of the edgeindices of the edges on the path from the node tothe root node.
Edge paths shorter than L are paddedwith zeros and the root node’s edge path consistsonly of zeros.
For example, the edge path of Assignconsists of the indices of the four edges on the pathto the root node sos .
Formally, each edge path p v = ( idx j,l ) n j =1 L l =1consists of edge indices that lead from the targetnode upwards to the root node.
The l -th edge index of the edge path refers to the l -th edge of the(reverse) path from the target node up to the root.
4. Encoding Scheme
To encode the position of a node v in a tree T , weapply the sinusoidal encoding by (Vaswani et al.
to each edge index of the node’s edge path p vand compute the positional tree encoding T E j,l =(cid:0) EE idx j,l ,i ′ (cid:1)
i ′∈{ 1 ,...,d idx } with j ∈ { 1 , . . . , n } andl ∈ { 1 , . . .
, L } as follows:
[Equation]
355 EE idx j , l , 2 i +
=
cos ( ω i · idx j , l ) with ω i = 1 / 10000
i didx for i ∈ { 1 , . . .
, d idx 2 } .
The parameter d idx is the dimension of the positional encoding of an edge index idx j,l and determines the number of encodings per edge index, i.e., d idx∈ N sine and cosine pairs.
This results in an encoding of length d idx ∗ L for an edge path, which also corresponds to d m , the size of the node’s embedding.
The tree encoding T E v j of a node v j is the concatenation of the encodings of its edge path indices:
[Equation]
with j ∈ { 1 , . . . , n } .
4. Encoding Properties
The tree encoding entails the following properties, which result from the composition of the edge paths and the sine and cosine functions used to encode the edge paths:
Uniqueness property
The concatenation of the edge indices encodings of a edge path results in a unique tree encoding for a node v j .
This is because for a node v j , the combination of its edge path indices idx j,l is unique and the sine (and cosine) function is injective on the products of its edge path indices and each frequency ω
i ′ due to the transcendality of π .
Shifting property
The tree encoding of a child node v j contains the first ( L − 1) ∗ d idx dimensions of its parent node’s tree encoding, shifted to the right by d idx dimensions.
Two nodes are siblings if these shifted parent dimensions contain identical values for both nodes.
Similar to (Shiv and Quirk, 2019), the shifting property is intended to enable the attention mechanism to identify whether a node is an ancestor or a sibling of a node v j .
However, relative distances between sibling nodes cannot be represented by shifting alone.
Linear combination property
Due to the mathematical properties of the sine and cosine functions (Vaswani et al., 2017) hypothesizes that it is possible for the attention mechanism to attend to sequential order relationships, i.e., to learn relative positions.
We make use of this property to allow attending to sibling nodes by their relative positions.
From the positional encoding of an edge index idx j,l , the positional encoding of another  edge index idx j,l + k can be computed for any integer offset k , using the sum rules for trigonometricfunctions.
The positional encoding of a siblings ofa node v j with an offset k < 0 and l = 1 can beexpressed as a linear combination of the sine andcosine values of the node v j edge encoding.
Evaluation
5. Datasets
For the evaluation we use the Python code generation benchmarks CoNaLa (Yin et al., 2018) and
Hearthstone (HS) (Ling et al., 2016) and for semantic parsing we use the datasets GEO (Tang and
Mooney, 2001) and Atis (Dahl et al., 1994).
Hearthstone is a corpus for the automatic generation of code for cards in the trading cardgame HearthStone and consists of 665 Pythonclasses, each representing one card.
CoNaLa isa Python corpus containing 2,379 curated and593,891 mined NL-code pairs mined from the developer forum Stack Overflow.
CoNaLa containsas NL intent real-world questions to diverse implementation topics instead of pseudo-code annotations.
GEO is a corpus of natural language questions about US geography and consists of 600 training and 280 test examples of NL-Prolog queriespairs.
Atis is a corpus of natural language queriesfor a flights database featuring 4473 training and448 test examples of NL-lamba-calculus pairs.
5. Metrics
As evaluation metrics, we use the BLEU score asimplemented by (Yin et al., 2018), the exact matchaccuracy and the token- and sequence-level prefixprecision and recall.
The BLEU score measures the similarity between the generated code and the reference codein terms of n-grams.
We compute the token-level
BLEU score on normalized and tokenized code bydetermining the precision on n-grams, then takethe geometric mean of the precisions, and applya brevity penalty for predictions shorter than theexpected code.
Exact match accuracy measures the ratio of predictions that were predicted without a single error.
This ensures the semantic correctness of the code,but it is very conservative in that it doesn‘t allowthe slightest syntactic variation, and that no distinction is made between a prediction that is 0% correctand one that is 99% correct.
Token- and sequence-level prefix precision and recall interpolate exact match accuracies based on the idea that if some prefix was predicted correctly, the model didn‘t make a mistake up to that point, so it is alright to give the model some credit for that.
On token level, it measures the ratio of tokens in correct prefixes, while on sequence level, it measures the average percentage of the longest correct prefix, ignoring snippet length.
The prefix-based metrics also partially measure semantical correctness because the expected snippet is by definition semantically correct.
5. Experimental Setup
We performed the experiments on a cluster of IBM
AC922 nodes with dual Power9 CPUs (2.80-3.
GHz, 22 cores each), 256 GB RAM and 6 Nvidia
Volta V100 accelerators with 32 GB RAM.
And on nodes with AMD EPYC CPUs (2.3 GHz, 24 cores each), 1 TB RAM and 8 A100 GPUs with 40 GB
RAM per node.
In the following experiments, unless specified otherwise, we train each task for 500 epochs with batch size 15 , learning rate 0 . 0001 , 6 encoder and to 8 decoder layers, 16 attention heads, model dimension 512 , maximum tree height 32 and feedforward dimension 2048 .
For inference, we set the number of beams to k = 30 and use a maximum beam length of 250 tokens.
5. Evaluation Results
We performed the evaluation on two tasks, namely semantic parsing on GEO and ATIS and code generation on CoNaLa and Hearthstone.
In both cases, the goal is to generate formal meaning representations from natural language input, that is, lambdacalculus expressions or Python code.
We scored about 18% and 70% BLEU score for the CoNaLa and Hearthstone benchmarks, respectively (cf.
table 2). State of the art approaches have more sophisticated implementations that include dynamic composite neural architecture, additional embedded information, pre-training, or re-ranking. Our results on the semantic parsing datasets GEO and ATIS are listed in table 3.
On ATIS we achieved an exact match accuracy of 86,1%, which is comparable to all our baselines except for the leading approach.
The exact match accuracy on
GEO is below the baselines, which we conjecture to be caused by the small amount of training data.
To test whether the tree-encoded Transformer learns to predict the AST structure correctly, we [Table] Our [Table]
[Table]
Our system [Table] Pos. Enc.
Incl.
Str.
Lit.
EM
Acc.
Seq vs.
Tree
Seq.
Recall
Seq vs.
Tree
Token
Recall
Seq vs.
Tree
Seq.
Prec.
Seq vs.
Tree
Token
Prec.
Seq vs.
Tree
Seq yes 4,5% 26,9% 23,0% 28,4% 26,4%
Seq no 10,6%
{+Tree yes 7,6% +3,
+}
+2,0%
23,7% +0,7%
30,5% +2,1%
+1,4%
Tree [Table] looked at the exact match accuracy and token- andsequence-level precision and recall, as shown intable 4. We masked all string literals and computed thelongest common prefix between the best predictedand the expected AST sequence for each samplefrom the test dataset.
We do this on a model wetrained with tree encoding and then on a model wetrained with sequential encoding.
With the exclusion of string literals, the prefixprecision and recall jump from about 25% to about50% with both sequential and tree encoding.
Fromthe prefix analysis, we can take away that the stringliterals have a significant impact on the quality ofthe prediction and that longer sequences are moredifficult to predict.
What we also found is that treeencoding gives an improvement of up to 3.0% whenexcluding string literals over sequential encoding.
[-Our results on the semantic parsing datasets GEOand ATIS are listed in table 
-]
Conclusion
We propose a Transformer-based architecture witha tree-based positional encoding and constraineddecoding based on a grammar model derived fromtraining data.
We evaluate it on four differentdatasets.
While we do not surpass state-of-the-artmethods, we see relative improvement of applyingtree encoding over sequential encoding.
In the future, we will work on improved versions of training with constraint masks and grammar models that respect larger contexts than onlythe immediate containing attribute.
Also, we willfurther investigate attention-based mechanisms forgenerating tree structures, also in the context ofdomain-specific languages as generation target. Association for Computational Linguistics
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung and
Luke Zettlemoyer
Y. Oda, H. Fudaba, G. Neubig
H. Hata, S. Sakti,
T. Toda
Python
Foundation. 2018.
CPython 3.7 abstract grammar definition ”Parser/Python.asdl”.
Maxim Rabinovich, Mitchell Stern, and Dan Klein.
In ”Proceedings of the 2018 Conference of the North
American the Association
tional
Srinivasan
Daniel
Wang, Andrew
Appel,
Korn and
In Proceedings of the Twenty-First Conference on
Uncertainty in Artificial Intelligence , pages 658–666.
AUAI Press.