Original: Are Transformers All That Karel Needs?
Revised: Are Transformers All That Karel Needs?

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Recent works have shown theincredible promise of using neural networks for thetask of program synthesis from input-output examples.
Revised: Recent works have shown the promise of using neural networks for the task of program synthesis from input-output examples.

Revised: The Karel dataset has been a benchmark for evaluating program synthesis approaches.

Revised: Several techniques have been proposed to use neural guided program synthesis with Karel being used as a baseline.

Original: In this paper, we proposeusing Transformer-based architectures as a baseline for the program synthesistask on the Karel dataset.
Revised: In this paper, we propose using Transformer-based architectures as a baseline for the program synthesis task on the Karel dataset.

Original: Speciﬁcally, we leverage DistillGPT as our decodermodel to perform program synthesis for the Karel DSL.
Revised: Speciﬁcally, we leverage GPT2 as our decoder model to perform program synthesis for the Karel DSL.

Original: We show that changingthe model architecture from an LSTM to a transformer based architecture, we areable to signiﬁcantly improve on supervised learning approaches obtaining a top-1generalization accuracy of 82.4%.
Revised: We show that changing the model architecture from an LSTM to a transformer based architecture, we are able to signiﬁcantly improve on supervised learning approaches obtaining a top-1 generalization accuracy of 82.4%.

Original: Further, applying execution guided search onthe output beams of the model increases the accuracy of our approach to 89.64%
Revised: Further, applying execution guided search on the output beams of the model increases the accuracy of our approach to 89.64%.

Original:  Introduction
Revised:  Introduction

Original: Programming by example is program generation task such that the generated program satisﬁes aspeciﬁcation given in the form of input-output pairs.
Revised: Programming by example is program generation task such that the generated program satisﬁes a speciﬁcation given in the form of input-output pairs.

Original: Advances in neural networks have spurredinterest in using deep learning techniques to solve problems in the programming languages space.
Revised: Advances in neural networks have spurred interest in using deep learning techniques to solve problems in the programming languages space.

Original: While domains such as performing arithmetic operations, string manipulation, list sorting, querygeneration, etc have been studied in literature, generating programs for languages with more complexstructures like looping and conditionals is a harder problem to solve.
Revised: While domains such as performing arithmetic operations, string manipulation, list sorting, query generation etc have been studied in literature, generating programs for languages with more complex structures like looping and conditionals is a harder problem to solve.

Original: The Karel programming language [24] was used in the 1980s as an introductory language by Stanford
Revised: The Karel programming language [28] was used in the 1980s as an introductory language by Stanford

Original: CS students.
Revised: CS students.

Original: Its DSL is aimed at controlling a robot in a grid space, where it must navigate to pick andplace markers while avoiding the walls.
Revised: Its DSL is aimed at controlling a robot in a grid space, where it must navigate to pick and place markers while avoiding the walls.

Original: The DSL speciﬁcation (Fig. 1) has support for conditionaland looping constructs, which make it an ideal test bed for evaluating program synthesis approaches.
Revised: The DSL speciﬁcation (Fig. 1) has support for conditional and looping constructs, which make it an ideal test bed for evaluating program synthesis approaches.

Original: Devlin et al.[9] introduced the Karel dataset for performing neural program induction.
Revised: Devlin et al.[10] introduced the Karel dataset for performing neural program induction.

Original: Their task wasto learn to represent a program and by training on input-output examples and predicting the output ofa program given a new input grid.
Revised: Their task was to learn to represent a program and by training on input-output examples and predicting the output of a program given a new input grid.

Original: Our work is focused on program synthesis, whereby we aim togenerate a program given the input-output pairs obtained on executing the program.
Revised: Our work is focused on program synthesis, whereby we aim to generate a program given the input-output pairs obtained on executing the program.

Original: Bunel et al.[3] introducedusing a neural approach for program synthesis using the Karel dataset as abenchmark.
Revised: Bunel et al.[3] introduced a neural approach for program synthesis using the Karel dataset as a benchmark.

Original: They used 5 IO pairs as an input to the model to synthesize the program, while usingthe 6th pair as a held out test sampleon which to evaluate the generalization performance.
Revised: They used 5 IO pairs as an input to the model to synthesize the program, while using the 6th pair as a held out test sample to evaluate the generalization performance.

Original: Theypresented both supervised and reinforcement learning based approaches and showed that RL naturallyachieved better generalization ability since it maximized the probability of generating semanticallycorrect programs.
Revised: They presented both supervised and reinforcement learning based approaches and showed that RL naturally achieved better generalization ability since it maximized the probability of generating semantically correct programs.

Original: Since then a number of approaches have been proposed and studied to improve the performancein the Karel task.
Revised: Since then a number of approaches have been proposed and studied to improve the performance in the Karel task.

Original: These approaches include inferring trace executions [29], using execution baseddecoding [6], learning to execute latent programs [7] and to repair programs after decoding [14].
Revised: These approaches include inferring trace executions [32], using execution based decoding [6], learning to execute latent programs [7] and to repair programs after decoding [16].

Original: We posit that transformer based models should be able to achieve much better performance on Karel,even without employing techniques like program repair or modeling intermediate state representations.
Revised: We posit that transformer based models should be able to achieve much better performance on Karel, even without employing techniques like program repair or modeling intermediate state representations.

Original: Transformer based models have been shown to perform well for generating programs from a natural  language speciﬁcation as well as to perform code completion in languages like Python.
Revised: Transformer based models have been shown to perform well for generating programs from a natural 35th Conference on Neural Information Processing Systems (aiplans 2021), Sydney, Australia. language speciﬁcation as well as to perform code completion in languages like Python.

Original: Scholak,
Revised: Scholak,

Original: Schucher, and Bahdanau[28] show that using vanilla transformers along with some execution guidedsearch are able to perform well in synthesizing SQL queries.
Revised: Schucher, and Bahdanau[31] show that using vanilla transformers along with some execution guided search are able to perform well in synthesizing SQL queries.

Original:  Related Work
Revised:  Related Work

Original: With the advent of neural program synthesis [23, 10], the models are able to work on differentdomains with little modiﬁcation and are more robust towards giving generalizable results.
Revised: With the advent of neural program synthesis [27, 11], the models are able to work on different domains with little modiﬁcation and are more robust towards giving generalizable results.

Original: Initial works in neural program synthesis utilized neural networks to prune the search space fortraditional program synthesis techniques [16, 22], but then [3] introduced neural models using LSTMlayers capable of generating programs.
Revised: Initial works in neural program synthesis utilized neural networks to prune the search space for traditional program synthesis techniques [19, 25], but then [3] introduced neural models using LSTM layers capable of generating programs.

Original: Subsequent works improved upon Bunel et al.’s
Revised: Subsequent works improved upon Bunel et al.

Original:  work whilekeeping the network architecture relatively unchanged.
Revised: ’s work while keeping the network architecture relatively unchanged.

Original: [29] further showed that using the informationfrom traces of individual examples helps the model in generating better programs.
Revised: [32] further showed that using the information from traces of individual examples helps the model in generating better programs.

Original: [7, 6, 21] devisedtechniques to execute the partially generated programs on the inputs to obtain the intermediate statesthat help the model get a better estimation of the next tokens to be generated.
Revised: [7, 6, 24] devised techniques to execute the partially generated programs on the inputs to obtain the intermediate states that help the model get a better estimation of the next tokens to be generated.

Original: Gupta et al.
Revised: Gupta et al.

Original: [14]improve upon execution guided search by introducing a neural debugger to repair the generatedprograms.
Revised: [16] improve upon execution guided search by introducing a neural debugger to repair the generated programs.

Original: Introduction of Transformers [31] in Natural Language Processing (NLP) has been revolutionary.
Revised: Introduction of Transformers [34] in Natural Language Processing (NLP) has been revolutionary.

Original: Transformers overcame pitfalls of LSTMs and GRUs like exploding and vanishing gradients, weakness in modelling long-term dependencies by using a multi-head attention mechanism that can processthe input tokens in parallel rather than sequentially.
Revised: Transformers overcame pitfalls of LSTMs and GRUs like exploding and vanishing gradients, weakness in modelling long-term dependencies by using a multi-head attention mechanism that can process the input tokens in parallel rather than sequentially.

Original: With time, larger and more capable Transformermodels were introduced that improved the performance of neural models in different problem statements within NLP [8, 20, 18, 19, 26] and produced comparable performance in computer vision [4,11, 17] and speech processing as well [5, 32, 33].
Revised: With time, larger and more capable Transformer models were introduced that improved the performance of neural models in different problem statements within NLP [9, 23, 21, 22, 30] and produced comparable performance in computer vision [4, 12, 20] and speech processing as well [5, 35, 36].

Original: Recently, a lot of attention is being paid towards problems such as source code generation, codecompletion, function naming, bug ﬁxing, etc., in programming languages and impressive resultsare produced with transformer models.
Revised: Recently, a lot of attention is being paid towards problems such as source code generation, code completion, function naming, bug ﬁxing, etc., in programming languages and impressive results are produced with transformer models.

Original: [30] is a general purpose code completion tool for multipleprogramming languages, [2, 13] used transformers for human language to code generation, [26]introduced an unsupervised neural transcompiler that learns to translate programs between C++, Javaand Python even in the absence of parallel training corpus, [12] utilized a BERT based architecture fornatural language code search and code documentation generation.
Revised: [33] is a general purpose code completion tool for multiple programming languages, [2, 14, 26, 8, 17] used transformers for natural language to code generation, [30] introduced an unsupervised neural transcompiler that learns to translate programs between C++, Java and Python even in the absence of parallel training corpus, [13] utilized a BERT based architecture for natural language code search and code documentation generation.

Original: ProgRES [1], a recently introducedlarge scale few-shot program induction benchmark on C++ programs, uses a transformer architecture(BART) as a baseline model for synthesizing programs.
Revised: ProgRES [1], a recently introduced large scale few-shot program induction benchmark on C++ programs, uses a transformer architecture (BART) as a baseline model for synthesizing programs.

Original: This shows that transformers can capture thesyntactical information of the programming languages they are trying to model.
Revised: This shows that transformers can capture the syntactical information of the programming languages they are trying to model.

Original: The results show thatlarge language models are capable to solve programming tasks without the need of adding additionalcontextual information.
Revised: The results show that large language models are capable of solving programming tasks without the need of adding additional contextual information.

Original:  Approach
Revised:  Approach

Original: Our approach is inspired by the promise shown by transformer architectures in the NLP and programsynthesis community.
Revised: Our approach is inspired by the promise shown by transformer architectures in the NLP and program synthesis community.

Original: Our input-output encoding is handled by a convolutional neural network asdescribed in [3].
Revised: Our input-output encoding is handled by a convolutional neural network as described in [3].

Original: For our decoder, we used DistillGPT2, available on [15], as our base transformerarchitecture.
Revised: For our decoder, we used GPT2 architecture with 6 layers as our base model.

Original: It uses the technique described in [27] to distill the GPT-2 model down to get a modelwhich has 37% lesser weights compared to the original architecture.

Original: We chose this model  as itrequires a lower computational cost to train the model. Both the IO encoder and DistillGPT decoderlayers are initialized with random weights and both are  learned end-to-end.
Revised: We downloaded the model available on [18] as our starting point for the transformer, but initialized both the IO encoder and GPT2 decoder layers with random weights which are then learned end-to-end.

Original: Our model architecture isshown in Figure 2.
Revised: Our model architecture is shown in Figure 2.

Original: During inference, we take the top beam output of the model as the chosen program for evaluation.
Revised: During inference, we take the top beam output of the model as the chosen program for evaluation.

Original: When adding simple execution guidance (SEG), we sample the top 50 beams of the model, and use the
Revised: When adding simple execution guidance (SEG), we sample the top 50 beams of the model, and use the

Original: Karel interpreter to evaluate them on the 5 IO pairs.
Revised: Karel interpreter to evaluate them on the 5 IO pairs.

Original: We take the top ranked program which satisﬁesall the 5 pairs, before using this to evaluate against the 6th held out IO sample.
Revised: We take the top ranked program which satisﬁes all the 5 pairs, before using this to evaluate against the 6th held out IO sample.

Original: If no beam satisﬁes allthe 5 pairs, then we return a NULL program which counts as an incorrect output during evaluation.
Revised: If no beam satisﬁes all the 5 pairs, then we return a NULL program which counts as an incorrect output during evaluation.

Original: This method of using the interpreter as guidance for beam selection is more rudimentary than methodsdetailed in the past [6] and is similar to the greedy search employed by [14].
Revised: This method of using the interpreter as guidance for beam selection is more rudimentary than methods detailed in the past [6] and is similar to the greedy search employed by [16].

Original: However, since wedo not execute partial programs, we do not need to add measures to deal with issues surroundingsampling for looping and branching constructs.
Revised: However, since we do not execute partial programs, we do not need to add measures to deal with issues surrounding sampling for looping and branching constructs.

Original:  Experimental Setup
Revised:  Experimental Setup

Original: The Karel dataset consists of a train/validation/test split of around 1.1 million synthetically generatedprograms written in the Karel language for the training set, and 2.5k programs each for the validationand testing set.
Revised: The Karel dataset consists of a train/validation/test split of around 1.1 million synthetically generated programs written in the Karel language for the training set, and 2.5k programs each for the validation and testing set.

Original: Each program is accompanied by 6 input output(IO) pairs.
Revised: Each program is accompanied by 6 input output(IO) pairs.

Original: These IO pairs are imagerepresentations of the state of the Karel robot in the grid world.
Revised: These IO pairs are image representations of the state of the Karel robot in the grid world.

Original: We use the programs and IO pairs inthe training set to train our model, and report results by running our inference procedure on the IOsamples in the test set.
Revised: We use the programs and IO pairs in the training set to train our model, and report results by running our inference procedure on the IO samples in the test set.

Original: Following [3], we set up the task as supervised sequence generation conditioned on a set of inputoutput representations.
Revised: Following [3], we set up the task as supervised sequence generation conditioned on a set of inputoutput representations.

Original: The decoder is fed a combined representation of the IO encoding and theoutput of the previous timestep.
Revised: The decoder is fed a combined representation of the IO encoding and the output of the previous timestep.

Original: We used Adam optimizer, batch size 8 (due to computationalconstraints), with initial learning rate of 1e-4.
Revised: We used Adam optimizer, batch size 8 (due to computational constraints), with initial learning rate of 1e-4.

Original: We used a linear learning rate scheduler with warmupof 139606 steps (around 1 epoch) and gradient clipping to ensure that the transformer model doesn’tdiverge during training [25] and trained for 20 epochs.
Revised: We used a linear learning rate scheduler with warmup of 139606 steps (around 1 epoch) and gradient clipping to ensure that the transformer model doesn’t diverge during training [29] and trained for 20 epochs.

Original: The model was trained on a DGX-1 machinewith an Intel Xeon E5-2698 CPU, 32GB of available system memory to the training job, and an Nvidia Tesla V100 GPU, using an allocation of 16GB VRAM.
Revised: The model was trained on a DGX-1 machine

Original: [Table]
Revised: [Table]

Revised: with an Intel Xeon E5-2698 CPU, 32GB of available system memory to the training job, and

Original: al
Revised: an

Original: [Table] [Table] [Table] [Table]
Revised: Nvidia Tesla V100 GPU, using an allocation of 16GB VRAM.

Original:  Results
Revised:  Results

Original: [Figure] Table 1 shows our top-1 generalisation and exact match accuracies on the Karel test dataset.
Revised: Table 1 shows our top-1 generalisation and exact match accuracies on the Karel test dataset.

Original: Ourgeneralisation accuracy is calculated by using the program obtained by feeding 5 input-output pairsto the model and evaluating this against the 6th held out IO sample.
Revised: Our generalisation accuracy is calculated by using the program obtained by feeding 5 input-output pairs to the model and evaluating this against the 6th held out IO sample.

Revised: Baselines We compare our results against previous approaches published for program synthesis.

Revised: Chen, Liu, and Song [6] report results for both a single decoder model, as well as using ensemble approaches.

Revised: We have reported both results here, however our approach only uses a single model during inference.

Original:  For Transformer + Simple Execution Guidance, this accuracy is calculated on the program selected after searching the beams.
Revised: Results For Transformer + Simple Execution Guidance, this accuracy is calculated on the program selected after searching the beams.

Original: We obtain a generalisation accuracy of 82.4% on our base transformer model which increases to89.64% after adding the execution guidance.
Revised: We obtain a generalisation accuracy of 82.4% on our base transformer model which increases to 89.64% after adding the execution guidance.

Original: We also obtain an exact match accuracy of 44.8% afterperforming SEG during inference.
Revised: We also obtain an exact match accuracy of 44.8% after performing SEG during inference.

Revised: We also ran experiments to validate the advantage of using a transformer model on the current state-of-the-art which is the SED approach by Gupta et al.

Revised: We fed the output beams of our transformer model to the debugger LSTM trained without ﬁne-tuning as per [16].

Revised: We used an implementation available on GitHub [15].

Revised: With this approach, mentioned as Transformer + Debugger in Table 1, we achieved a top-1 generalisation accuracy of 90.44%.

Original:  While analyzing the results (Fig. 3) on the test set of 2500 programs, we found that in 86.4% (or2160 instances), the execution guidance was selecting the top ranked beam from the model output.
Revised: One thing to note here, we did not perform the ﬁne-tuning step which was described by Gupta et al. due to memory constraints in our infra. Despite this, our results show the usefulness of using a transformer based model for the program synthesis task on Karel and can be used in conjunction with many algorithmic approaches described in literature. While analyzing the results (Fig. 3a) on the test set of 2500 programs, we found that in 86.4% (or 2160 instances), the execution guidance was selecting the top ranked beam from the model output.

Original: In124 cases, our inference method returned a NULL program, indicating that no beam satisﬁed all 5input-output pairs.
Revised: In 124 cases, our inference method returned a NULL program, indicating that no beam satisﬁed allinput-output pairs.

Original: The remaining samples tailed off as per the ranking given by the model.
Revised: The remaining samples tailed off as per the ranking given by the model.

Revised: In Figure 3b, we see the impact of increasing the number of IO samples the model is allowed to see before generating the program.

Revised: Figure 3c shows the trend

Original:  Conclusion
Revised:  Conclusion

Original: We report results for program synthesis on Karel by using transformers as the base architecture,replacing the LSTM-based models that have been reported earlier.
Revised: We report results for program synthesis on Karel by using transformers as the base architecture, replacing the LSTM-based models that have been reported earlier.

Original: We show that this change inarchitecture results in an improvement in generalisation accuracy over the base model without havingto model intermediate trace representations.
Revised: We show that this change in architecture results in an improvement in generalisation accuracy over the base model without having to model intermediate trace representations.

Original: Additionally, even performing simple execution guidedsearch over the beam outputs, we are able to achieve near state-of-the-art results on the task.
Revised: Additionally, even performing simple execution guided search over the beam outputs, we are able to achieve near state-of-the-art results on the task.

Original: We feelthat steps such as learning trace representations or program repair are complimentary to our work andcould be used to boost performance in conjunction with transformer-based models.
Revised: We feel that employing ensemble based methods as described in [6] can further enhance our performance.