T REE - TO - TREE N EURAL N ETWORKS FOR P ROGRAM
T RANSLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Program translation is an important tool to migrate legacy code in one language into an ecosystem built in a different language.
In this work, we are the ﬁrst to consider employing deep neural networks toward tackling this problem.
We observe that program translation is a modular procedure, in which one sub-tree in the source is translated into the corresponding target sub-tree at each step.
To capture this intuition, we design a tree-to-tree neural network as an encoder-decoder architecture to translate a source tree into a target one.
Meanwhile, we develop an attention mechanism for the tree-to-tree model, so that when the decoder expands one non-terminal in the target tree, the attention mechanism locates the corresponding sub-tree in the source to guide the expansion of the decoder.
We develop two tasks to evaluate the program translation capability of our tree-to-tree model against two state-of-the-art approaches.
We observe that our approach is consistently better than the baselines on all tasks and all metrics with a margin of up to 10% .
These results demonstrate that the tree-to-tree model is a promising tool for tackling the program translation problem.
I NTRODUCTION
Programs are the main tool for building computer applications, the IT industry, and the digital world.
Various programming languages have been invented to facilitate programmers to develop programs for different applications.
At the same time, the variety of different programming languages also introduces a burden when programmers want to combine programs written in different languages together.
Therefore, there is a tremendous need to enable program translation between different programming languages.
Nowadays, to translate programs in different programming languages, typically programmers would manually investigate the correspondence between the grammars of the two languages, then develop a rule-based translator.
However, this process can be inefﬁcient and error-prone.
In this work, we make the ﬁrst attempt to examine whether we can leverage deep neural networks to build a translator automatically.
Intuitively, the program translation problem in its format is similar to a natural language translation problem.
Some previous work propose to adapt phrase-based statistical machine translation approaches for code migration (Karaivanov et al., 2014; Nguyen et al., 2015). We may also consider applying neural machine translation (NMT) methods, such as sequence-to-sequence-based models (Bahdanau et al., 2015; Cho et al., 2015; Eriguchi et al., 2016;
He et al., 2016; Vaswani et al., 2017), to handle the program translation problem.
However, a big challenge making a sequenceto-sequence model ineffective is that, unlike human languages, programming languages all have rigorous grammars and are not tolerant to typos and grammatical mistakes.
It has been demonstrated that it is very hard for an RNN-based sequence generator to generate syntactically correct programs when the lengths grow large (Karpathy et al., 2015).
In this work, we observe that the main issue of an RNN that makes it hard to produce syntactically correct programs is that it entangles two sub-tasks together: (1) learning the grammar; and (2) aligning the sequence with the grammar.
When these two tasks can be handled separately, the performance can typically boost.
For example, Dong & Lapata (2016) employs a tree-based decoder to separate the two tasks.
In particular, the decoder in (Dong & Lapata, 2016) leverages the tree structural information to (1) generate the nodes at the same depth of the parse tree using an LSTM decoder; and (2) expand a non-terminal and generate its children in the parse tree.
Such an approach has been demonstrated to achieve the state-of-the-art results on several semantic parsing tasks.
Inspired by this observation, we hypothesize that the structural information in both input and output parse trees can be leveraged to enable such a separation.
Inspired by this intuition, we propose tree-to-tree neural networks to combine both a tree encoder and a tree decoder.
In particular, we observe that in the program translation problem, both input and output programs have their parse trees.
In addition, a cross-language compiler typically follows a modular procedure to translate the individual sub-components in the source tree into their corresponding target ones, and then compose them to form the ﬁnal target tree.
Therefore, we design the workﬂow of a tree-to-tree neural network to align with this procedure: when the decoder expands a non-terminal, it locates the corresponding sub-tree in the source using an attention mechanism, and uses the information of the sub-tree to guide the non-terminal expansion.
In particular, a tree encoder is helpful in this scenario, since it can aggregate all information of a sub-tree to the embedding of its root, so that the embedding can be used to guide the non-terminal expansion of the target tree.
We follow the above intuition to design the tree-to-tree model.
To the best of our knowledge, this is the ﬁrst tree-to-tree neural network architecture proposed for translation tasks in the literature.
To test our hypothesis, we develop two program translation tasks.
The ﬁrst class of tasks is synthetic, and we use it to evaluate program translation between two different programming paradigms (one imperative and the other functional).
The second class of tasks takes advantage of a cross-compiler between two full-ﬂedged languages: CoffeeScript and JavaScript.
We use it to evaluate the translation of programs using Python-style succinct syntax into the ones using C-style verbose syntax.
For training and evaluation, we use a pCFG-based generator to generate programs satisfying the grammar of a programming language.
This is different from some previous work on code migration, which use open-source projects implemented in both the source and the target programming languages to build the dataset (Karaivanov et al., 2014; Nguyen et al., 2015).
While handling realworld programs is also an important direction, randomly generated programs can reach corner cases that may not frequently seen in practice.
Therefore, testing on randomly generated programs is an effective way to examine the correctness of the learned model for program translation.
Our evaluation demonstrates that our tree-to-tree model outperforms other state-of-the-art approaches on the program translation tasks, and yields a margin of up to 10% on the token accuracy and up to 5% on the program accuracy.
These results demonstrate that our tree-to-tree model is promising toward tackling the program translation problem.
P ROGRAM TRANSLATION PROBLEM
In this work, we consider the problem of translating a program in one language into another.
One approach is to model the problem as a machine translation problem between two languages, and thus numerous neural machine translation approaches can be applied.
For the program translation problem, however, a unique property is that each input program unambiguously corresponds to a unique parse tree.
Thus, rather than modeling the input program as a sequence of tokens, we can consider the problem as translating a source tree into a target tree.
Note that most modern programming languages are accompanied with a well-developed parser.
Thus we can assume the parse trees of both the source and the target programs can be easily computed.
The challenge of the problem in our consideration is that the cross-compiler that translates programs from one language into another typically does not exist.
Therefore, even if we assume the existence of parsers for both the source and the target languages, the translation problem itself is still nontrivial.
We formally deﬁne the problem as follows.
Deﬁnition 1 (Program translation) .
Given two programming languages L s and L t , each being a set of instances ( p k , T k ) , where p k is a program, and T k is its corresponding parse tree.
We assume that there exists a translation oracle π , which maps instances in L s to instances in L t .
Given a dataset of instance pairs ( i s , i t ) such that i s ∈
L s , i
t ∈
L t and π ( i s )
=
i t , our problem is to learn a function
F that maps
each i s ∈ L s into i t
=
π
( i s ) .
Note that this problem deﬁnition does not restrict on whether F knows only the program p k or only the parse tree T k .
Thus it can leverage both information to ﬁnish the translation task.
Also, in the
CoffeeScript Program:
x=1 if y==
JavaScript Program: if (y ==
= 0) { x = 1; }
Block
If
Op===
Value
IdentifierLiteral y
Value
NumberLiteral
Assign
Value
IdentifierLiteral x
Value
NumberLiteral
Block
Program
IfStatement
BinaryExpression
Identifier y =
==
Literal
BlockStatement
ExpressionStatement
AssignExpression
Identifier x =
Literal
Parse Tree
Parse Tree target language, since a program and its parse tree uniquely matches each other, F only needs to predict one of them to be correct.
In this deﬁnition, L s and L t are the source and target languages respectively, and we use the oracle π to model the goal of the translation. Notice that different tasks can be modeled with different choices of π . For example, besides program translation between two languages, we can also model the compiler optimization by setting L s = L t and choosing π that maps an un-optimized program into an optimized one.
In this work, we focus on the problem setting that we can get sufﬁcient aligned input-output examples for training.
When such an alignment is lacking, the program translation problem can be harder.
Several techniques for NMT have been proposed to handle this issue, such as dual learning (He et al., 2016), which have the potential to be extended for the program translation task.
We leave these more challenging problem setups as future work.
T REE - TO - TREE N EURAL N ETWORK
In this section, we present our design of the tree-to-tree neural network.
We ﬁrst motivate the design, and then present the details.
3. P ROGRAM TRANSLATION AS A TREE - TO - TREE TRANSLATION PROBLEM
Figure 1 presents an example of translation from CoffeeScript (CoffeeScript, 2009-2017) to
JavaScript.
We observe several interesting properties of the program translation problem.
First, the translation can be modular.
The ﬁgure highlights a sub-component in the source tree corresponding to x=1 and its translation in the target tree corresponding to x=1; .
This correspondence is regardless of other parts of the program.
Consider when the program grows longer and this statement may repetitively occur multiple times, it may be hard for a sequence-to-sequence model to capture the correspondence based on only token sequences without structural information.
Second, such a correspondence makes it a natural solution to locate the referenced sub-tree in the source when expanding a non-terminal in the target into a sub-tree.
Inspired by this intuition, we design the tree-to-tree neural network with attention, so that when decoding a non-terminal into a (sub-)tree, the model can employ an attention mechanism to locate the referenced source sub-tree.
The soft-attention mechanism makes the model differentiable, so that it can be trained end-to-end.
Third, although the corresponding sub-trees are analogues to each other, they differ in two aspects.
On the one hand, the non-terminal nodes are different.
For example, the non-terminal
IdentifierLiteral in CoffeeScript is called Identifier in JavaScript.
Some nonterminals, such as Value , may not even have a correspondence.
On the other hand, the topology structures of the corresponding sub-trees can be different.
For example, the Assign node in the
Block
If
Op===
Value
Identifier Literal y
Value
Number Literal
Assign
Value
IdentifierLiteral x
Value
NumberLiteral
Block
Program
IfStatement
BinaryExpression
Expanding
To Expand
Source Tree
Target Tree ℎℎℎℎℎℎℎℎℎℎℎℎℎℎℎℎℎℎ 1 ′ ℎ 2 ′ ℎ 3 ′ ℎ 4 ′ ℎ 5 ′
Attention map: 𝑤 𝑖 ∝ exp(ℎ 𝑖 𝑇 𝑊 0 ℎ 5 ′ )
Source embedding: 𝑒 𝑠 =
∑ 𝑖=117 𝑤 𝑖 ℎ 𝑖
=
ℎ 1 ;…;ℎ 17 𝑤
Combined embedding: 𝑒 𝑡 = tanh(𝑊 1 𝑒 𝑠 + 𝑊 2 ℎ 5 ′)
Predicting the node: node = argmax 𝐬𝐨𝐟𝐭𝐦𝐚𝐱(𝑊e t ) source tree has two children, while its correspondence in the target tree has three: one additional = is inserted in the middle.
These differences can become a challenge to build a program translation model. 3. T REE - TO - TREE N EURAL N ETWORK A tree-to-tree neural network follows an encoder-decoder architecture to encode the source tree into an embedding, and decode the embedding into the target tree.
To capture the intuition of the modular translation process, the decoder employs an (soft) attention model to locate the corresponding source sub-tree when expanding the non-terminal.
We illustrate the workﬂow of a tree-to-tree model in
Figure 2.
In the following, we present each different component of the model.
Converting a tree into a binary one.
Note that the input and output trees may contain multiple branches.
Although we can design tree-encoders and tree-decoders to handle trees with arbitrary number of branches, we observe that encoder and decoder for binary trees can be more effective.
Thus, the ﬁrst step is to convert both the source tree and the target tree into a binary tree.
To this aim, we employ the Left-child/Next-sibling encoding for this conversion.
Binary Tree Encoder.
The encoder part employs a Tree-LSTM (Tai et al.
to compute embeddings for both the entire tree and each sub-tree.
In particular, consider a node N that is attached with a token t s in its one-hot encoding representation, and it has two children N L and N R , which are its left child and right child respectively.
The encoder recursively computes the embedding for the N from the bottom up.
Assume that the left child and the right child maintain the LSTM state ( h L , c L ) and ( h R , c R ) respectively.
Then the LSTM state ( h, c ) of N is computed using
[Equation]
Here, A is a trainable word embedding matrix of size d × V s , where d is the embedding dimension and V s is the vocabulary size of the inputs.
All different W s
and U s are different trainable matrix of size d × d , and all different b s are different trainable d -dimensional vectors.
Note that a node may lack one or both of its children.
In this case, the encoder sets the LSTM state of the missing child to be zero.
Binary Tree Decoder.
The decoder generates the target tree from one single root node.
The decoder ﬁrst copies the LSTM state ( h, c ) of the root of the source tree, and attaches it to the root node of the target tree.
Then the decoder maintains a queue of all nodes to be expanded, and recursively expands each of them.
In each iteration, the decoder pops one node from the queue, and expands it.
In the following, we call the node being expanded the expanding node .
First, the decoder will predict the token of expanding node.
To this end, the decoder computes the embedding e t of the expanding node N , and then feeds it into a softmax regression network for prediction:
[Equation]
Here, W is a trainable matrix of size V t × d , where V t is the vocabulary size of the outputs and d is the embedding dimension.
Note that e t is computed using the attention mechanism, which we will explain later.
Each token t t is one of a non-terminal, a terminal, or a special (cid:104) EOS (cid:105) token.
If t t = (cid:104) EOS (cid:105) , then the decoder ﬁnishes expanding this node.
If t t is a terminal, the decoder does not generate a left child for it.
Otherwise, if t t is a non-terminal, then the decoder generates one new node as the left child of the expanding one.
Assuming ( h, c ) is the LSTM state of the expanding node, then the LSTM state ( h (cid:48) , c (cid:48) ) of its left child is computed as:
[Equation]
Here, B is a trainable word embedding matrix of size d × V t .
This new left child is pushed into the queue of all nodes to be expanded.
If t t (cid:54) =
(cid:104) EOS (cid:105) , then the decoder always generates a right child for the expanding node.
The LSTM state ( h (cid:48)(cid:48) , c (cid:48)(cid:48) ) of the right child is computed as:
[Equation]
Note that here the left child and right child use two different sets of parameters for LSTM L and
LSTM R respectively.
The new right child is also pushed into the queue of all nodes to be expanded.
The entire target tree generation process terminates until the queue is empty.
Attention mechanism to locate the source sub-tree.
Now we consider how to compute e t .
One straightforward approach is to compute e t as h , which is the hidden state attached to the expanding node.
However, in doing so, the embedding will soon forget the information about the source tree when generating deep nodes in the target tree, and thus the model yields very poor performance.
To better facilitate the information of the source tree, our tree-to-tree model employs the attention mechanism to locate the source sub-tree corresponding to the sub-tree rooted at the expanding node.
In fact, we compute the following probability:
[Equation]
where N t is the expanding node.
We denote this probability as P ( N s | N t ) , and we compute it as
[Equation]
where W 0 is a trainable matrix of size d × d .
To leverage the information from the source tree, the decoder can sample a source node N s following P ( N s | N t ) or simply predict the most-likely source node N s = argmax N s P ( N s | N t ) , and then get its hidden h s as e s .
This embedding can then be combined with h , the hidden state of the expanding node, to compute e t as follows:
[Equation]
where W 1 and W 2 are two trainable matrices of size d × d .
However, this approach suffers a big issue that the entire network is no longer end-to-end differentiable unless explicit supervision is given to train P ( N s | N t ) .
Unfortunately, this supervision is unavailable.
One can rely on policy gradient such as REINFORCE to mitigate the issue, but doing so is expensive and may not be effective.
We solve this problem by changing the above hard-attention scheme to a soft-attention mechanism.
Note that in Equation (6), we only need a continuous embedding vector e s rather than the discrete choice of N s .
Therefore, we can approximate e s as the expectation of the hidden state value across all N s conditioning on N t .
In fact, we have
[Equation]
Note that this ﬁnal formula (7) coincides with the standard attention formalism.
Combining all equations from (1) to (7), the entire tree-to-tree neural network is fully differentiable and can be trained end-to-end.
E VALUATION
In this section, we evaluate our tree-to-tree neural network with several baseline approaches on the program translation task.
To do so, we ﬁrst develop two benchmark datasets in Section 4.1 for evaluating different aspects.
Then we evaluate our tree-to-tree model with two baseline approaches, which are a sequence-to-sequence model (Bahdanau et al., 2015) and a sequence-to-tree model (Dong &
Lapata, 2016).
In the following, we start with presenting the details of the benchmark datasets and model details, and then present the results.
4. D ATASETS
To evaluate different approaches for the program translation problem, we ﬁrst develop a synthetic translation task, and then leverage a full-ﬂedged program translator from CoffeeScript to JavaScript to develop a more realistic task.
For the synthetic task, we design the source language to be imperative and the target one to be functional.
Such a design makes the source and target languages use different programming paradigms, so that the translation can be challenging.
Below is an example of the translation:
Source program
Target program for i=1; i < 10; i+1 do letrec f i
= if x > 1 then if i < 10 then y= let = if x > 1 then let y=1 in () else let y=2 in () else in f i+ y= else () endfor in f [Table] learning [Table]
by a factor of
[Table]
[Table] the validation loss does not decrease for 500 mini-batches [Table] This example demonstrates that a for-loop is translated into a recursive function.
We manually implement a translator, which is used to acquire the ground truth.
The formal grammars of the two languages can be found in Appendix A.1, and the implementation of the translator between the two languages can be found in Appendix B.
For the CoffeeScript tasks, the programming paradigm is identical.
CoffeeScript employs a Pythonstyle succinct syntax, while JavaScript employs a C-style verbose syntax.
To control the program length of the training and test data, we develop a pCFG-based program generator and a subset of the core CoffeeScript grammar.
We also limit the set of variables to restrict the vocabulary size.
We rely on the CoffeeScript compiler to generate the corresponding JavaScript ground truth.
The grammar used to generate the programs in our experiments can be found in Appendix A.2.
For both tasks, we are interested in the following two metrics:
• Token accuracy.
Given a source program, we run the evaluated model to translate it into a target program, and calculate the percentage of the tokens that can match the ground truth.
• Program accuracy.
Given a set of source programs, we run the evaluated model to translate them into target programs, and calculate the percentage of the predicted programs that can entirely match the ground truth.
While the program accuracy is more meaningful for the program translation task, token accuracy can provide more insight on different models’ performance.
Meanwhile, for both tasks, we randomly generate 100,000 pairs of source and
4. M ODEL DETAILS
We evaluate our tree-to-tree model against a sequence-to-sequence model (Bahdanau et al., 2015; Vinyals et al., 2015) and a sequence-to-tree model (Dong & Lapata, 2016).
Note that for a sequenceto-sequence model, there can be four variants to handle different input-output formats.
For example, given a program, we can simply tokenize them into a sequence of tokens.
We call this format as raw program , denoted as P. We can also use the parser to parse the program into a parse tree, and then serialize the parse tree as a sequence of tokens.
Our serialization of a tree follows its depth-ﬁrst traversal order, which is the same as (Vinyals et al., 2015).
We call this format as parse tree , denoted as T. For both input and output formats, we can choose either P or T. Similarly, for a sequence-to-tree model, we have two variants based on its input format being either P or T. Note that the sequenceto-tree model generates a tree as output, and thus requires its output format to be T (unserialized).
For the same reason, our tree-to-tree only has one form.
Therefore, we have 7 different models in our evaluation.
The hyper-parameters used in different models are summarized in Table 1. [Table]
4. R ESULTS ON SYNTHETIC TASKS
We evaluate our tree-to-tree models with baseline approaches on the synthetic tasks.
We create two datasets for evaluation, where one dataset contains programs of an average length 20, while in the other dataset, the average length of the programs is 50.
These lengths correspond to the number of tokens in the source programs.
The experimental results are presented in Table 2.
We can observe that our tree2tree model can outperform all baseline models on both tasks and using both metrics.
Especially, on the dataset with longer programs, the token accuracy signiﬁcantly outperforms all seq2seq models by a margin of around 30 points.
Its margin over a seq2tree model can also reach more than 6 points.
These results demonstrate that tree2tree model is more capable of learning the program translation relationship.
We also evaluate the effectiveness of attention mechanism on our tree2tree model. We observe that the program accuracy goes down to 0% , and the token accuracy also decreases dramatically.
This underlines the importance of attention mechanism for our tree2tree model.
We observe that increasing the length of the programs indeed makes the problem harder for all models: the program accuracy dramatically shrinks.
However, we observe that the decrease of the tree2tree model on the token accuracy is only around 3 points, and much smaller than other models’.
This shows that the tree2tree model has more potential to handle longer inputs.
We observe that seq2seq performs better than seq2tree on shorter programs, but worse on longer ones.
We attribute this phenomenon to the fact that for shorter programs, predicting correct structure does not bring many beneﬁts to improve program accuracy; instead, it may hurt the performance, since when generating the ASTs, more tokens are required to be predicted than generating the programs themselves.
However, on longer programs, the structure is more crucial to make sure a program is structurally correct, and thus a seq2tree model starts outperforming a seq2seq one.
In addition, we observe that different input formats affect both seq2seq and seq2tree models.
For a seq2seq model, outputting target programs makes it more effective than outputting their parse trees.
We attribute the reason to that a seq2seq decoder needs to predict more tokens when using the parse tree as its output format, and thus is more likely to make mistakes.
However, the input format does not seem to affect too much in these tasks on the token accuracy.
This observation is consistent for the seq2tree model as well: we observe that the differences on the choice of input format do not make too many differences on its prediction performance.
4. R ESULTS ON C OFFEE S CRIPT TASKS
We now move on to the CoffeeScript dataset.
Besides the program length, we want to understand whether the vocabulary size will affect model’s performance.
Therefore, we created two subset of the CoffeeScript grammar:
Coffee1 uses only two variables (i.e., x,y ) and two literals (i.e., 0,1 ), and Coffee2 can choose from a much larger set of variable names (i.e., all alphabetical characters) and literals (i.e., all 1-digit numerical literals).
We also generate two datasets for each grammar, [Table] JavaScript translation.
For our tree2tree model, “(-Attn)” means the tree2tree model without the attention mechanism.
containing programs of length 10 and 20 respectively. Note that CoffeeScript has a more succinct syntax, and thus a program of 10 or 20 has the same level of complexity of the input programs for our synthetic tasks of length 20 or 50 respectively.
The evaluation results are presented in Table 3.
Most of the observations from our synthetic tasks remain: our tree2tree model outperforms all baseline models; without the attention mechanism, the performance of our tree2tree model degrades signiﬁcantly; all models perform worse on longer inputs; a seq2seq model performs better than a seq2tree model on shorter programs, but worse on longer ones.
However, we observe some additional phenomena.
First, using a smaller vocabulary makes both accuracy metrics lower for all models.
We attribute this to the lack of variety in the program generator, which makes the generated program may share common sub-clauses or sub-trees.
The common sub-tree appears in multiple places in the source program, and they can make the attention mechanism harder to learn to locate the right one when decoding.
Since all models employ an attention mechanism, this property has hurt all models.
We observe that our tree2tree model can achieve a program accuracy of 98 .
8% for programs of the length 10, which outperforms the best baseline by 6 points.
This shows that our tree2tree model is more robust against this issue.
Second, for a seq2seq model, using the tree input format demonstrates a better performance than using raw programs as the inputs when the programs contain a larger vocabulary.
This is different than our observations on the synthetic dataset.
We attribute this to the reason that a tree input provides richer information to facilitate the decoder, especially when the attention mechanism is used to locate the attended sub-sequence.
Therefore, we argue that employing more structural information can beneﬁt the translation procedure.
For the small vocabulary, however, this beneﬁt is offset by the misleading effect discussed above.
Thus, the improvement is not signiﬁcant.
R ELATED WORK
Statistical machine translation.
Machine learning models, especially deep neural networks such as sequence-to-sequence based models, have been applied successfully to machine translation tasks between different natural languages (Bahdanau et al., 2015; Cho et al., 2015; Eriguchi et al., 2016;
He et al., 2016; Vaswani et al., 2017).
Eriguchi et al.
(2016) proposes a tree-to-sequence model, which employs a Tree-LSTM (Tai et al., 2015; Zhu et al., 2015) as the encoder, and leverages the parse tree of input sentences to facilitate the translation.
However, the parse tree information used in their work may not be accurate.
In our work, the parse tree is accurate, so that we can combine both a tree-encoder and a tree-decoder together to achieve a good performance.
Some recent work have applied statistical machine translation techniques to program translation (Allamanis et al., 2017;
Karaivanov et al., 2014; Nguyen et al., 2015; 2016).
Karaivanov et al. (2014) and Nguyen et al. (2015) propose to adapt phrase-based statistical machine translation models and leverage grammatical structures of programming languages for code migration. Nguyen et al.
(2016) proposes to use Word2Vec vector representation for APIs in libraries used in different programming languages, then learns a transformation matrix for API mapping.
On the contrary, our work is the ﬁrst to employ deep learning techniques for program translation.
Neural networks with tree structures.
In recent years, various neural networks with tree structures have been proposed to employ the structured information of the data.
(Dong & Lapata, 2016;
Rabinovich et al., 2017; Parisotto et al., 2017; Yin & Neubig, 2017; Alvarez-Melis & Jaakkola, 2017; Tai et al., 2015; Zhu et al., 2015; Socher et al., 2011a; Eriguchi et al., 2016;
Zhang et al., 2016;
Socher et al., 2011b; Kusner et al., 2017).
In these work, different tree-structured encoders are proposed for embedding the input data, and different tree-structured decoders are proposed for predicting the output trees.
In particular, Socher et al.
(2011b) and Kusner et al.
(2017) propose tree-structured autoencoders to learn vector representations of trees, and show better performance on tree reconstruction and other tasks such as sentiment analysis.
In this work, we are the ﬁrst to design the tree-to-tree neural network for translation tasks.
Neural networks for parsing. Other work study using neural networks to generate parse trees from input-output examples (Dong & Lapata, 2016; Vinyals et al., 2015; Aharoni & Goldberg, 2017;
Rabinovich et al., 2017; Yin & Neubig, 2017; Alvarez-Melis & Jaakkola, 2017; Dyer et al., 2016;
Chen et al., 2017; 2016).
Dong & Lapata (2016) proposes a seq2tree model that allows the decoder
RNN to generate the output tree recursively in a top-down fashion.
This approach achieves the state-of-the-art results on many semantic parsing tasks.
Some other work incorporate the knowledge about the grammar into the architecture designs (Yin & Neubig, 2017; Rabinovich et al., 2017) to achieve better performance on speciﬁc tasks.
However, these approaches are harder to generalize to other tasks.
Again, none of them is designed for program translation or proposes a tree-to-tree architecture.
Neural networks for code generation.
A recent line of research study using neural networks for code generation (Balog et al., 2017; Devlin et al., 2017; Parisotto et al., 2017; Ling et al., 2016;
Rabinovich et al., 2017; Yin & Neubig, 2017).
In (Ling et al., 2016; Rabinovich et al., 2017; Yin & Neubig, 2017), they study generating code in a DSL from natural language input or programs from another DSL.
However, their designs require additional manual efforts to adapt to new DSLs in consideration.
In our work, we consider the tree-to-tree model as a generic approach that can be applied to any grammar.
C ONCLUSION AND FUTURE WORK
In this work, we are the ﬁrst to consider neural network approaches for the program translation problem, and are the ﬁrst to propose a tree-to-tree neural network that combines both a tree-RNN encoder and a tree-RNN decoder.
Our extensive evaluation demonstrates that our tree-to-tree neural network outperforms several state-of-the-art baselines, including sequence-to-sequence models and sequence-to-tree models.
This renders our tree-to-tree model as a promising tool toward tackling the program translation problem.
At the same time, we observe many challenges that existing techniques are not capable to handle.
For example, all models’ performances degrade when the inputs become longer; it is also unclear how to handle an inﬁnite vocabulary set that may be employed in real applications; further, the training requires a dataset of aligned input-output pairs, which may be lacking in practice.
We consider all these problems as important future work in the research agenda toward solving the program translation problem.
A G RAMMARS FOR GENERATING THE PROGRAMS IN THE EVALUATION A. G RAMMARS FOR THE SYNTHETIC TASK
Below are the grammar speciﬁcations of the source language and the target language used in the synthetic task respectively.
[Equation]
Expr> + <Const> | <
Expr> − <Var> | <
Expr> − <Const> <Cmp> ::= <
Expr> == <
Expr> | <Expr> > < Expr> | <
Expr> < <
Expr> <Assign> ::= <
Var>
= <
Expr> <If> :
:= if <Cmp> then <statement> else <statement> endif <
For> ::= for <Var> = <
Expr> ; <Cmp> ; <Expr> do <Statement> endfor <
Single> ::= <
Assign> | <
If> | <
For> <Seq> ::= <Single> ; <
Single> | <Seq> ; <Single> <Statement> ::
= <
Seq> | < Single> <Unit> ::= () <
Expr> ::= <
Var> | <
Expr> + <Var> | < Expr> − <Var> <Cmp> ::= <
Expr> == <
Expr> | <Expr> > < Expr> | <
Expr> < <
Expr> <Term> ::= <LetTerm> | <
Expr> | <Unit> | <IfTerm> <LetTerm> ::= let <Var>
= <
Term> in <Term> | letrec <Var> <Var> = <
Term> in <Term> <IfTerm> ::= if <
Cmp> then <Term> else <
Term>
A. G RAMMAR FOR THE C OFFEE S CRIPT TASK
Below is the grammar for the CoffeeScript task.
<
Expr> ::= <
Var> | <
Const> | <
Expr> + <Var> | <
Expr> + <Const> | <
Expr> * <Var> | <
Expr> * <Const> | <
Expr> == <
Var> | <
Expr> == <
Const> <
Simple> ::
= <
Var>
= <
Expr> | <
Expr> <IfShort> ::= <
Simple> if <Expr> | <IfShort> if <Expr> <WhileShort> ::= <
Simple> while <
Expr> | <WhileShort> while <Expr> <ShortStatement> ::= <
Simple> | <IfShort> | <WhileShort> <Statement> ::= <ShortStatement> | if <Expr> <br> <indent+> <Block> <indent-> | while <
Expr> <br> <indent+> <Block> <indent-> | if <
Expr> <br> <indent+> <Block> <indent-> <br> else <br> <indent+> <Block> <indent-> | if <Expr> then <
ShortStatement> else <ShortStatement> <Block> ::
= <Statement> | <Block> <br> <Statement> above grammar
<br> denotes the newline character B P YTHON IMPLEMENTATION OF THE TRANSLATOR FOR THE SYNTHETIC TASK t1 = s e l f .
t r a n s l a t e f r o m f o r ( a s t [ 2 ] ) t2 =
s e l f .
t r a n s l a t e f r o m f o r ( a s t [ 3 ] ) r e t u r n [ ’ < IF > ’ , cmp , t1 , t2 ] e l i f a s t [0]
== ’ <
FOR > ’ :
var =
s e l f .
t r a n s l a t e f r o m f o r ( a s t [ 1 ] ) i n i t =
s e l f .
t r a n s l a t e f r o m f o r ( a s t [ 2 ] ) cmp =
s e l f .
t r a n s l a t e f r o m f o r ( a s t [ 3 ] ) inc = s e l f .
t r a n s l a t e f r o m f o r ( a s t [ 4 ] ) body =
s e l f .
t r a n s l a t e f r o m f o r ( a s t [ 5 ] ) tb =
[ ’ < LET > ’ , ’ blank ’ , body , [ ’ < APP > ’ , ’ func ’ , inc ] ] func body =
[ ’ < IF > ’ , cmp , tb , ’ < UNIT > ’ ]
t r a n s l a t e = [ ’ <
LETREC > ’ , ’ func ’ , var , func body , [ ’ < APP > ’ , ’ func ’ , inc ] ] r e t u r n t r a n s l a t e e l i f a s t [0]
== ’ < ASSIGN > ’ :
r e t u r n [ ’ < LET > ’ , a s t [1] , a s t [2] , ’ < UNIT > ’ ]
e l i f a s t [0]
== ’ <
Expr > ’ :
r e t u r n a s t e l i f a s t [0] == ’ < Op+ > ’ :
r e t u r n a s t e l i f a s t [0] == ’ <
Op − > ’ : r e t u r n a s t e l i f a s t [0]
== ’ <
CMP > ’ :
r e t u r n a s t e l s e : r e t u r n a s t