Original: Are Transformers All That Karel Needs?
Revised: Are Transformers All That Karel Needs?

Original: Abstract
Revised: Abstract

Original: Recent works have shown the promise of using neural networks for the task of program synthesis from input-output examples.
Revised: Recent works have shown the promise of using neural networks for the task of program synthesis from input-output examples.

Original: The Karel dataset has been a benchmark for evaluating program synthesis approaches.
Revised: The Karel dataset has been a benchmark for evaluating program synthesis approaches.

Original: Several techniques have been proposed to use neural guided program synthesis with Karel being used as a baseline.
Revised: Several techniques have been proposed to use neural guided program synthesis with Karel being used as a baseline.

Original: In this paper, we propose using Transformer-based architectures as a baseline for the program synthesis task on the Karel dataset. Speciﬁcally, we leverage GPT2 as our decoder modelto perform program synthesis for the
Revised: Most of these techniques use an LSTM based model for decoding and improve performance by proposing complex algorithmic additions, such as using inferred execution traces, latent execution of partial programs and debugging generated programs.

Original: Karel DSL. We show that  changing the model architecturefrom an LSTM to a transformer based architecture, we are able to signiﬁcantly improve on supervised learning approaches obtaining atop-1 generalization accuracy of 82.4%. Further, applying execution guided search on the output beams of the model increases the accuracy of our approach to 89.64%.
Revised: We observe that by changing the base architecture to a transformer based one, speciﬁcally GPT2, we are able to apply simple execution guidance on top to achieve a generalization accurary of 89.64%, which is within 2.36 percentage points of the current state-of-the-art on Karel which uses ensembling.

Original:  Introduction
Revised:  Introduction

Original: Programming by example is program generation task such that the generated program satisﬁes a speciﬁcation given in the form of input-output pairs.
Revised: Programming by example is program generation task such that the generated program satisﬁes a speciﬁcation given in the form of input-output pairs.

Original: Advances in neural networks have spurred interest in using deep learning techniques to solve problems in the programming languages space.
Revised: Advances in neural networks have spurred interest in using deep learning techniques to solve problems in the programming languages space.

Original: While domains such as performing arithmetic operations, string manipulation, list sorting, query generation etc have been studied in literature, generating programs for languages with more complex structures like looping and conditionals is a harder problem to solve.
Revised: While domains such as performing arithmetic operations, string manipulation, list sorting, query generation etc have been studied in literature, generating programs for languages with more complex structures like looping and conditionals is a harder problem to solve.

Original: The Karel programming language [28] was used in the 1980s as an introductory language by Stanford
Revised: The Karel programming language [28] was used in the 1980s as an introductory language by Stanford

Original: CS students.
Revised: CS students.

Original: Its DSL is aimed at controlling a robot in a grid space, where it must navigate to pick and place markers while avoiding the walls.
Revised: Its DSL is aimed at controlling a robot in a grid space, where it must navigate to pick and place markers while avoiding the walls.

Original: The DSL speciﬁcation (Fig. 1) has support for conditional and looping constructs, which make it an ideal test bed for evaluating program synthesis approaches.
Revised: The DSL speciﬁcation (Fig. 1) has support for conditional and looping constructs, which make it an ideal test bed for evaluating program synthesis approaches.

Original: Devlin et al.[10] introduced the Karel dataset for performing neural program induction.
Revised: Devlin et al.[10] introduced the Karel dataset for performing neural program induction.

Original: Their task was to learn to represent a program and by training on input-output examples and predicting the output of a program given a new input grid.
Revised: Their task was to learn to represent a program and by training on input-output examples and predicting the output of a program given a new input grid.

Original: Our work is focused on program synthesis, whereby we aim to generate a program given the input-output pairs obtained on executing the program.
Revised: Our work is focused on program synthesis, whereby we aim to generate a program given the input-output pairs obtained on executing the program.

Original: Bunel et al.[3] introduced a neural approach for program synthesis using the Karel dataset as a benchmark.
Revised: Bunel et al.[3] introduced a neural approach for program synthesis using the Karel dataset as a benchmark.

Original: They used 5 IO pairs as an input to the model to synthesize the program, while using the 6th pair as a held out test sample to evaluate the generalization performance.
Revised: They used 5 IO pairs as an input to the model to synthesize the program, while using the 6th pair as a held out test sample to evaluate the generalization performance.

Original: They presented both supervised and reinforcement learning based approaches and showed that RL naturally achieved better generalization ability since it maximized the probability of generating semantically correct programs.

Original: Since then a number of approaches have been proposed and studied to improve the performance in the Karel  task.
Revised: Since then a number of approaches have been proposed and studied to improve the performance in the Karel program synthesis task.

Original: Theseapproaches include inferring trace executions [32], using execution based decoding [6], learning to execute latent programs [7] and to repair programs after decoding [16].
Revised: These include inferring trace executions [32], using execution based decoding [6], learning to execute latent programs [7] and to repair programs after decoding [16].

Original: We posit that transformer based models should be able to achieve much better performance on Karel, even without employing techniques like program repair or modeling intermediate state representations. Transformer based models have been shown to perform well for generating programs from a natural 35th Conference on Neural Information Processing Systems (aiplans 2021), Sydney, Australia. language speciﬁcation as well as to perform code completion in languages like Python. Scholak,
Revised: These methods focus on making complex changes to how the input and output to the model is processed, while keeping the base architecture as an LSTM.

Original: Schucher, and Bahdanau[31] show that using vanilla transformers along with some execution guided search are able to perform well in synthesizing SQL queries.
Revised: We posit that transformer based models should be able to achieve much better base performance on Karel which can be improved further by adding relatively simpler execution guided search.

Original:  Related Work
Revised:  Related Work

Original: With the advent of neural program synthesis [27, 11], the models are able to work on different domains with little modiﬁcation and are more robust towards giving generalizable results.
Revised: With the advent of neural program synthesis [27, 11], the models are able to work on different domains with little modiﬁcation and are more robust towards giving generalizable results.

Original: Initial works in neural program synthesis utilized neural networks to prune the search space for traditional program synthesis techniques [19, 25], but then [3] introduced neural models using LSTM layers capable of generating programs.
Revised: Initial works in neural program synthesis utilized neural networks to prune the search space for traditional program synthesis techniques [19, 25], but then [3] introduced neural models using LSTM layers capable of generating programs.

Original: Subsequent works improved upon Bunel et al.
Revised: Subsequent works improved upon Bunel et al.

Original: ’s work while keeping the network architecture relatively unchanged.
Revised: ’s work while keeping the network architecture relatively unchanged.

Original: [32]further showed that using the information from traces of individual examples helps the model in generating better programs.
Revised: [32] showed that using the information from traces of individual examples helps the model in generating better programs.

Original: [7, 6, 24] devised techniques to execute the partially generated programs on the inputs to obtain the intermediate states that help the model get a better estimation of the next tokens to be generated.
Revised: [7, 6, 24] devised techniques to execute the partially generated programs on the inputs to obtain the intermediate states that help the model get a better estimation of the next tokens to be generated.

Revised: While [6] uses methods to sample executable partial programs from the output beams to perform search, [7] try to execute partial programs in the latent space, removing the need for algorithmic methods of sampling partial programs.

Original: Gupta et al.
Revised: Gupta et al.

Original: [16] improve upon execution guided search by introducing a neural debugger to repair the generated programs.
Revised: [16] improve upon execution guided search by introducing a neural debugger to repair the generated programs.

Original: Introduction of Transformers [34] in Natural Language Processing (NLP) has been revolutionary.
Revised: Introduction of Transformers [34] in Natural Language Processing (NLP) has been revolutionary.

Original: Transformers overcame pitfalls of LSTMs and GRUs like exploding and vanishing gradients, weakness in modelling long-term dependencies by using a multi-head attention mechanism that can process the input tokens in parallel rather than sequentially.
Revised: Transformers overcame pitfalls of LSTMs and GRUs like exploding and vanishing gradients, weakness in modelling long-term dependencies by using a multi-head attention mechanism that can process the input tokens in parallel rather than sequentially.

Original: With time, larger and more capable Transformer models were introduced that improved the performance of neural models in different problem statements within NLP [9, 23, 21, 22, 30] and produced comparable performance in computer vision [4, 12, 20] and speech processing as well [5, 35, 36].
Revised: With time, larger and more capable Transformer models were introduced that improved the performance of neural models in different problem statements within NLP [9, 23, 21, 22, 30] and produced comparable performance in computer vision [4, 12, 20] and speech processing as well [5, 35, 36].

Original: Recently, a lot of attention is being paid towards problems such as source code generation, code completion, function naming, bug ﬁxing, etc., in programming languages and impressive results are produced with transformer models.
Revised: Recently, a lot of attention is being paid towards problems such as source code generation, code completion, function naming, bug ﬁxing, etc., in programming languages and impressive results are produced with transformer models.

Original: [33] is a general purpose code completion tool for multiple programming languages, [2, 14, 26, 8, 17] used transformers for natural language to code generation, [30] introduced an unsupervised neural transcompiler that learns to translate programs between
Revised: [33] is a general purpose code completion tool for multiple programming languages, [2, 14, 26, 8, 17] used transformers for natural language to code generation, [30] introduced an unsupervised neural transcompiler that learns to translate programs between

Original: C++, Java and Python even in the absence of parallel training corpus, [13] utilized a BERT based architecture for natural language code search and code documentation generation.
Revised: C++, Java and Python even in the absence of parallel training corpus, [13] utilized a BERT based architecture for natural language code search and code documentation generation.

Revised: Scholak, Schucher, and Bahdanau[31] show that using vanilla transformers along with some execution guided search are able to perform well in synthesizing SQL queries.

Original: ProgRES [1], a recently introduced large scale few-shot program induction benchmark on C++ programs, uses a transformer architecture (BART) as a baseline model for synthesizing programs.
Revised: ProgRES [1], a recently introduced large scale few-shot program induction benchmark on C++ programs, uses a transformer architecture (BART) as a baseline model for synthesizing programs.

Original: This shows that transformers can capture the syntactical information of the programming languages they are trying to model.
Revised: This shows that transformers can capture the syntactical information of the programming languages they are trying to model.

Original: The results show that large language models are capable of solving programming tasks without the need of adding additional contextual information.
Revised: The results show that large language models are capable of solving programming tasks without the need of adding additional contextual information.

Original:  Approach 
Revised:  Approach 3.1 Dataset

Original: Our approach is inspired by the promise shown by transformer architectures in the NLP and program synthesis community. Our input-output encoding is handled by a convolutional neural network as described in [3]. For our decoder, we used GPT2 architecture with 6 layers as our base model.

Original: We downloaded the model available on [18] as our starting point for the transformer, but initialized both the IO encoder and GPT2 decoder layers with random weights which are then learned end-to-end.

Original: Our model architecture is shown in Figure 2.

Original: During inference, we take the top beam output of the model as the chosen program for evaluation.

Original: When adding simple execution guidance (SEG), we sample the top 50 beams of the model, and use the

Original: Karel interpreter to evaluate them on the 5 IO pairs.

Original: We take the top ranked program which satisﬁes all the 5 pairs, before using this to evaluate against the 6th held out IO sample.

Original: If no beam satisﬁes all the 5 pairs, then we return a NULL program which counts as an incorrect output during evaluation.

Original: This method of using the interpreter as guidance for beam selection is more rudimentary than methods detailed in the past [6] and is similar to the greedy search employed by [16].

Original: However, since we do not execute partial programs, we do not need to add measures to deal with issues surrounding sampling for looping and branching constructs.

Original: The Karel dataset consists of a train/validation/test split of around 1.1 million synthetically generated programs written in the Karel language for the training set, and 2.5k programs each for the validation and testing set.
Revised: The Karel dataset consists of a train/validation/test split of around 1.1 million synthetically generated programs written in the Karel language for the training set, and 2.5k programs each for the validation and testing set.

Original: Each program is accompanied by 6 input output(IO) pairs.
Revised: Each program is accompanied by 6 input output(IO) pairs.

Original: These IO pairs are image representations of the state of the Karel robot in the grid world.
Revised: These IO pairs are image representations of the state of the Karel robot in the grid world.

Original: We use the programs and IO pairs in the training set to train our model, and report results by running our inference procedure on the IO samples in the test set.
Revised: We use the programs and IO pairs in the training set to train our model, and report results by running our inference procedure on the IO samples in the test set.

Revised: 3.2 Model Architecture

Revised: Our input-output encoding is handled by a convolutional neural network as described in [3].

Revised: For our transformer decoder, we used GPT2 architecture with 6 layers as our base model.

Revised: We downloaded the model available on [18] as our starting point for the transformer, but initialized both the IO encoder and GPT2 decoder layers with random weights which are then learned end-to-end.

Revised: Our model architecture is shown in Figure 2.

Original: Following [3], we set up the task as supervised sequence generation conditioned on a set of inputoutput representations.
Revised: Following [3], we set up the task as supervised sequence generation conditioned on a set of inputoutput representations.

Original: The decoder is fed a combined representation of the IO encoding and the output of the previous timestep.
Revised: The decoder is fed a combined representation of the IO encoding and the output of the previous timestep.

Original: We used Adam optimizer, batch size 8 (due to computational constraints), with initial learning rate of 1e-4.
Revised: We used Adam optimizer, batch size 8 (due to computational constraints), with initial learning rate of 1e-4.

Original: We used a linear learning rate scheduler with warmup of 139606 steps (around 1 epoch) and gradient clipping to ensure that the transformer model doesn’t diverge during training [29] and trained for 20 epochs.
Revised: We used a linear learning rate scheduler with warmup of 139606 steps (around 1 epoch) and gradient clipping to ensure that the transformer model doesn’t diverge during training [29] and trained for 20 epochs.

Original: The model was trained on a DGX-1 machine 
Revised: The model was trained on a DGX-1 machine with an Intel Xeon E5-2698 CPU, 32GB of available system memory to the training job, and an Nvidia Tesla V100 GPU, using an allocation of 16GB VRAM.

Original: [Table]
Revised: [Table]

Original: with an Intel Xeon E5-2698 CPU, 32GB ofavailable system memory to the training job, and
Revised: During inference, we take the top ranked beam output of the model as the chosen program for evaluation.

Original: an
Revised: When adding simple execution guidance (SEG), we sample the top 50 beams of the model, and use the Karel interpreter to evaluate them on the 5 IO pairs.

Original: Nvidia Tesla V100 GPU, using an allocation of 16GB VRAM.
Revised: We take the top ranked program which satisﬁes all the 5 pairs, before using this to evaluate against the 6th held out IO sample.

Original:   Results
Revised: If no beam satisﬁes all the 5 pairs, then we return a NULL program which counts as an incorrect output during evaluation. This method of using the interpreter as guidance for beam selection is more rudimentary than methods detailed in the past [6] and is similar to the greedy search employed by [16]. However, since we do not execute partial programs, we do not need to add measures to deal with issues surrounding sampling for looping and branching constructs.  Results

Original: Table 1 shows our top-1 generalisation and exact match accuracies on the Karel test dataset.
Revised: Table 1 shows our top-1 generalisation and exact match accuracies on the Karel test dataset.

Original: Our generalisation accuracy is calculated by using the program obtained by feeding 5 input-output pairs to the model and evaluating this against the 6th held out IO sample.
Revised: Our generalisation accuracy is calculated by using the program obtained by feeding 5 input-output pairs to the model and evaluating this against the 6th held out IO sample.

Original: Baselines We compare our results against previous approaches published for program synthesis.
Revised: We compare our results against previous approaches published for program synthesis.

Original: Chen, Liu, and Song [6] report results for both a single decoder model, as well as using ensemble approaches.
Revised: Chen, Liu, and Song [6] report results for both a single decoder model, as well as using ensemble approaches.

Original: We have reported both results here, however our approach only uses a single model during inference.
Revised: We have reported both results here, however our approach only uses a single model during inference.

Original: Results For Transformer + Simple Execution Guidance, this accuracy is calculated on the program selected after searching the beams.
Revised: For SEG results, this accuracy is calculated on the program selected after searching the beams.

Original: We obtain a generalisation accuracy of 82.4% on our base transformer model which increases to 89.64% after adding the execution guidance.
Revised: We observe a generalisation accuracy of 82.4% on our base transformer model which increases to 89.64% after adding the execution guidance.

Original: We also obtain an exact match accuracy of 44.8% after performing SEG during inference.
Revised: We also observe an exact match accuracy of 44.8% after performing SEG during inference.

Original: We also ran experiments to validate the advantage of using a transformer model on the current state-of-the-art which is the SED approach by Gupta et al.
Revised: We also conducted experiments to validate the advantage of using a transformer model on the current state-of-the-art which is the SED approach by Gupta et al.

Original: We fed the output beams of our transformer model to the debugger LSTM trained without ﬁne-tuning as per [16].
Revised: We fed the output beams of our transformer model to the debugger LSTM trained without ﬁne-tuning as per [16].

Original: We used an implementation available on GitHub [15].
Revised: We used an implementation available on GitHub [15].

Original: With this approach, mentioned as Transformer + Debugger in Table 1, we achieved a top-1 generalisation accuracy of 90.44%.
Revised: With this approach, mentioned as Transformer + Debugger in Table 1, we achieved a top-1 generalisation accuracy of 90.44%.

Original: One thing to note here, we did not perform the ﬁne-tuning step which was described by Gupta et al.
Revised: One thing to note here, we did not perform the ﬁne-tuning step which was described by Gupta et al.

Original: due to memory constraints in our infra.
Revised: due to memory constraints in our system.

Original: Despite this, our results show the usefulness of using a transformer based model for the program synthesis task on Karel and can be used in conjunction with many algorithmic approaches described in literature.
Revised: 4.1 Error Analysis

Original: While analyzing the results(Fig. 3a) on the test set of 2500 programs, we found that in 86.4% (or 2160 instances), the execution guidance was selecting the top ranked beam from the model output.
Revised: While analyzing the results on the test set of 2500 programs, we found that in 86.4% (orinstances), the execution guidance was selecting the top ranked beam from the model output.

Original: In 124 cases, our inference method returned a NULL program, indicating that no beam satisﬁed allinput-output pairs.
Revised: In 124 cases, our inference method returned a NULL program, indicating that no beam satisﬁed allinput-output pairs.

Original: The remaining samples tailed off asper the ranking given by  the model.
Revised: Figure 3a shows that accuracy of the model decreases as the length of the ground truth program increases. Figure 3b shows that in the majority of cases, the programs generated by the model are shorter than the ground truth program expected. This bias could perhaps be rectiﬁed by balancing the length of the programs in the dataset. Figure 3c shows that SEG seems to choose longer programs when the top beam doesn’t satisfy all 6 IO pairs.

Original: In Figure 3b, we see the impact of increasing the number of IO samples the model is allowed to see before generating the program.
Revised: Figure 3d analyzes the programs that satisﬁed all 6 IO pairs, while not being an exact match to the ground truth. It shows the number of programs that have the same trace as the target program for N IO pairs.

Original: Figure 3c shows  the trend
Revised: This shows that the [Figure]

Revised: gap between the generalization and exact match accuracies could be explained by the fact that two programs could have the same trace for the IO pairs despite being not an exact match.

Revised: Figure 3e shows that the accuracy of the model increases as more IO pairs are provided to the model Figure 3f shows that the accuracy of the model shows a rising trend as the number for beams searched during

Revised: SEG is increased.

Original:  Conclusion
Revised:  Conclusion

Original: We report results for program synthesis on Karel by using transformers as the base architecture, replacing the LSTM-based models that have been reported earlier.
Revised: We report results for program synthesis on Karel by using transformers as the base architecture, replacing the LSTM-based models that have been reported earlier.

Original: We show that this change in architecture results in an improvement in generalisation accuracy over the base model without having to model intermediate trace representations.
Revised: We show that this change in architecture results in a signiﬁcant improvement in generalisation accuracy.

Original: Additionally, even performing simple execution guided search over the beam outputs, we are able to achieve near state-of-the-art results on the task.
Revised: Further, adding even performing simple execution guided search over the beam outputs, we are able to achieve performance at par with more complex approaches.

Original: We feel that employing ensemble based methods as described in [6] can further enhance our performance.