Original: Towards Neural Functional Program Evaluation
Revised: Towards Neural Functional Program Evaluation

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: This paper explores the capabilities of current transformer-based language modelsfor program evaluation of simple functional programming languages.
Revised: This paper explores the capabilities of current transformer-based language models for program evaluation of simple functional programming languages.

Original: We introducea new program generation mechanism that allows control over syntactic sugar forsemantically equivalent programs.
Revised: We introduce a new program generation mechanism that allows control over syntactic sugar for semantically equivalent programs.

Original: T5 experiments reveal that neural functionalprogram evaluation performs surprisingly well, achieving high 90% exact programmatch scores for most in-distribution and out-of-distribution tests. 
Revised: T5 experiments reveal that neural functional program evaluation performs surprisingly well, achieving high 90% exact program match scores for most in-distribution and out-of-distribution tests. Using pretrained T5 weights has signiﬁcant advantages over random initialization.

Original: We presentand evaluate on three datasets to study generalization abilities that are speciﬁc tofunctional programs based on: type, function composition, and reduction steps. 
Revised: We present and evaluate on three datasets to study generalization abilities that are speciﬁc to functional programs based on: type, function composition, and reduction steps. Code and data are publicly available at https://github.com/ElementAI/neural-interpreters.

Original:  Introduction
Revised:  Introduction

Original: Neural models originally developed for natural language processing show promising performance formodeling computer programming languages (Chen et al., 2021; Austin et al., 2021).
Revised: Neural models originally developed for natural language processing show promising performance for modeling computer programming languages (Chen et al., 2021; Austin et al., 2021).

Original: This has led toa number of interesting applications, and deep-learning models are now successfully and routinelyapplied in tools that assist developers in writing and understanding programs and code.
Revised: This has led to a number of interesting applications and deep-learning models are now successfully and routinely applied in tools that assist developers in writing and understanding programs and code.

Original: For instance,neural language models can synthesize (Gulwani et al., 2017; Ellis et al., 2020), complete (Chenet al., 2021), and summarize programs (Elnaggar et al., 2021), whether they are written in mainstreamlanguages like Python and Java, or in domain-speciﬁc languages like SQL and regex.
Revised: For instance, neural language models can synthesize (Gulwani et al., 2017; Ellis et al., 2020), complete (Chen et al., 2021) and summarize programs (Elnaggar et al., 2021), whether they are written in mainstream languages like Python and Java or in domain-speciﬁc languages like SQL and regex.

Original: A less explored application of neural models is program evaluation (Reed and De Freitas, 2015;
Revised: A less explored application of neural models is program evaluation (Reed and De Freitas, 2015;

Original: Zaremba and Sutskever, 2014).
Revised: Zaremba and Sutskever, 2014).

Original: Here, the challenge is to predict the output of a program given itsinput.
Revised: Here, the challenge is to predict the output of a program given its input.

Original: In software engineering, this problem is solved by program interpreters (Nystrom, 2021),which are implemented as rule-based, non-differentiable systems that take formal program code asinput and produce outputs, effects, and/or errors.
Revised: In software engineering, this problem is solved by program interpreters (Nystrom, 2021) which are implemented as rule-based, non-differentiable systems that take formal program code as input and produce outputs, effects and/or errors.

Original: While these interpreters must reject programs thatare syntactically incorrect or contain semantic errors , neural networks could realize differentiableinterpreters to allow for evaluation of incomplete, informally-, or even incorrectly-speciﬁed programs.
Revised: While these interpreters must reject programs that are syntactically incorrect or contain semantic errors, neural interpreters allow the evaluation of incomplete, informally-, or even incorrectly-speciﬁed programs.

Original: Developers could use them to predict output or effects of programs before they are fully written,which could aid in their very creation.
Revised: Developers could use them to predict output or effects of programs before they are fully written, which could aid in their very creation.

Original: Previous studies have focused mainy on simple imperativeprogramming languages, e.g., FORTH (Bošnjak et al., 2017) and subsets of Python (Bieber et al.,2020).
Revised: Previous studies have focused mainly on simple imperative programming languages, e.g., FORTH (Bošnjak et al., 2017) and subsets of Python (Bieber et al., 2020).

Original: For these languages, evaluation is complicated by control ﬂow, state, and mutability.
Revised: For these languages, evaluation is complicated by control ﬂow, state, and mutability.

Original: Stateof-the-art techniques solve these complications with specialized, complex model architectures, butresults are not as promising as one might hope (Bieber et al.
Revised: State-of-the-art techniques solve these complications with specialized, complex model architectures, but results are not as promising as one might hope (Bieber et al.

Original: (2020) report an accuracy of 62 .
Revised: (2020) report an accuracy of 62 .

Original: 1% ontheir task and dataset).
Revised: 1% on their task and dataset).

Original: This paper studies neural program evaluation for functional programming languages, which sharefew of the complications of imperative languages (Feser et al., 2016).
Revised: This paper studies neural program evaluation for functional programming languages, which share few of the complications of imperative languages (Feser et al., 2016).

Original: Our emphasis is on lambdacalculi, which sit at the core of modern functional programming languages like Scheme and Haskell(Pierce and Benjamin, 2002).
Revised: Our emphasis is on lambda calculi, which sit at the core of modern functional programming languages like Scheme and Haskell (Pierce and Benjamin, 2002).

Original: We show that program evaluation can be recast such that it is tractablefor a language model based on the standard transformer architecture (Vaswani et al., 2017).
Revised: We show that program evaluation can be recast such that it is tractable for a language model based on the standard transformer architecture (Vaswani et al., 2017).

Original: Ratherthan predicting a program’s output given an input, we train the model to reduce a program to a formthat cannot be reduced further.
Revised: Rather than predicting a program’s output given an input, we train the model to reduce a program to a form that cannot be reduced further.

Original: This generalizes the notion of program evaluation, as it can also beused to model partial application of functions and programs.
Revised: This generalizes the notion of program evaluation, as it can also be used to model partial application of functions and programs.

Original: Unlike works that use neural programs to strengthen generalization of models (Chen et al., 2020; Nyeet al., 2020) on another downstream task, we study the generalization of neural program evaluationexplicitly.
Revised: Unlike works that use neural programs to strengthen generalization of models (Chen et al., 2020; Nye et al., 2020) on another downstream task, we study the generalization of neural program evaluation explicitly.

Original: We propose three data splits to measure “type generalization”, “function compositionalgeneralization”, and “reduction steps generalization” of our approach.
Revised: We propose three data splits to measure “type generalization”, “function compositional generalization”, and “reduction steps generalization” of our approach.

Original: Furthermore, we investigatethe effect of syntax and abstraction on neural program evaluation.
Revised: Furthermore, we investigate the effect of syntax and abstraction on neural program evaluation.

Original: We deﬁne two lambda calculithat are equally capable of expressing semantically equivalent programs, yet are different in theirsyntax.
Revised: We deﬁne two lambda calculi that are equally capable of expressing semantically equivalent programs, yet are different in their syntax.

Original: We ﬁnd that the neural program evaluation problem is slightly less tractable for the lambdacalculus with the simpler syntax.
Revised: We ﬁnd that the neural program evaluation problem is slightly less tractable for the lambda calculus with the simpler syntax.

Original: Lastly, we compare two evaluation strategies for lambda calculus,lazy evaluation and eager evaluation, and ﬁnd that they are similarly tractable.
Revised: Lastly, we compare two evaluation strategies for lambda calculus, lazy evaluation and eager evaluation, and ﬁnd that they are similarly tractable.

Original:  Evaluation of Lambda Calculus
Revised:  Evaluation of Lambda Calculus

Original: Our experiments are based on synthetic data for two lambda calculi.
Revised: Our experiments are based on synthetic data for two lambda calculi.

Original: The ﬁrst lambda calculus (1 st
Revised: The ﬁrst lambda calculus (1 st

Original: LC) is untyped and has three syntactic categories: variable, lambda, and application expressions, i.e.,
Revised: LC) is untyped and has three syntactic constructs: variable, lambda, and application expressions, i.e.,

Original: [Equation]
Revised: [Equation]

Original: The second lambda calculus (2 nd LC) adds types and additional syntactic categories:
Revised: The second lambda calculus (2 nd LC) adds types and additional syntactic constructs:

Original: [Equation]
Revised: [Equation]

Original: Terms are generated exclusively from the 2 nd LC using types to guide the generation.
Revised: Terms are generated exclusively from the 2 nd LC using types to guide the generation.

Original: First, theprogram type is chosen at random from the set of types t .
Revised: First, the program type is chosen at random from the set of types t .

Original: Terms e are then generated according tothe type, beginning at the root of the term tree and proceeding down the branches to the leaves. 
Revised: Terms e are then generated according to the type, beginning at the root of the term tree and proceeding down the branches to the leaves. Speciﬁcally, production is recursive and top to bottom (root to leaves).

Revised: Generation starts with the outermost type constructor, and proceeds by sampling from all compatible term constructors.

Revised: Production stops when none of the resulting leaf nodes contains hole terms, e .

Original: Careis taken to ensure that the generated program does not contain any free variables, is well-typed, andthat it terminates without error.
Revised: Care is taken to ensure that the generated program does not contain any free variables, is well-typed, and that it terminates without error.

Original: Terms from the 1 st LC are obtained by transforming terms from the2 nd LC.
Revised: Terms from the 1 st LC are obtained by transforming terms from the 2 nd LC.

Original: We  use Church encoding to represent syntactic categories from the 2 nd LC that do not appearin the 1 st LC (Pierce and Benjamin, 2002).
Revised: We therefore only consider the normalisable subset of the 1 st LC. We use Church encoding to represent syntactic constructs from the 2 nd LC that do not appear in the 1 st LC (Pierce and Benjamin, 2002).

Original: For example, lists are represented by their right fold:
Revised: For example, lists are represented by their right fold:

Original: [Equation]
Revised: [Equation]

Revised: where h and τ are head and tail terms, and where f , e , and l refer to the combining function, the initial value, and the Church-encoded input list of elements, respectively.

Original: We consider two reduction strategies: lazy and eager evaluation.
Revised: We consider two reduction strategies: lazy and eager evaluation.

Original: Lazy evaluation reduces a termto weak head normal form (WHNF), which is a term that is not an application expression.
Revised: Lazy evaluation reduces a term to weak head normal form (WHNF), which is a term that is not an application expression.

Original: Forexample, the term ( λ x . x)( λ y .
Revised: For example, the term ( λ x . x)( λ y .

Original: y) is reduced to the WHNF λ y .
Revised: y) is reduced to the WHNF λ y .

Original: y , while the term λ y .
Revised: y , while the term λ y .

Original: ( λ x . x) y is alreadyin WHNF and thus not reduced further.
Revised: ( λ x . x) y is already in WHNF and thus not reduced further.

Original: Eager evaluation not only reduces a term to WHNF, but alsoreduces all subterms of the term.
Revised: Eager evaluation not only reduces a term to WHNF, but also reduces all subterms of the term.

Original: We call this the “deep” reduction and refer to the result as the deepnormal form (DNF).
Revised: We call this the “deep” reduction and refer to the result as the deep normal form (DNF).

Original: The terms ( λ x . x)( λ y .
Revised: The terms ( λ x . x)( λ y .

Original: y) and λ y .
Revised: y) and λ y .

Original: ( λ x . x) y are reduced to the same DNF λ y .
Revised: ( λ x . x) y are reduced to the same DNF λ y .

Original: y .
Revised: y .

Original: We deﬁne the number of reduction steps of a term to be the number of reductions performed on itsroot term and all its subterms.
Revised: We deﬁne the number of reduction steps of a term to be the number of reductions performed on its root term and all its subterms.

Original: Reduction to WHNF is a special case of reduction to DNF, and thenumber of reduction steps to WHNF is always smaller or equal to the number of reduction steps to
Revised: Reduction to WHNF is a special case of reduction to DNF, and the number of reduction steps to WHNF is always smaller or equal to the number of reduction steps to

Original: DNF.
Revised: DNF.

Original: We use the number of reduction steps to denote the degree of evaluation complexity of a term,with more steps translating to higher complexity.
Revised: We use the number of reduction steps to denote the degree of evaluation complexity of a term, with more steps translating to higher complexity.

Original: This can then be used to evaluate the performanceof our neural program evaluation models.
Revised: This can then be used to evaluate the performance of our neural program evaluation models.

Original:  Experiments
Revised:  Experiments

Original: In our experiments, we train a language model based on the encoder-decoder transformer architectureto reduce programs to their normal forms.
Revised: In our experiments, we train a language model based on the encoder-decoder transformer architecture to reduce programs to their normal forms.

Original: Depending on the experiment, the input to the network is aterm in either the ﬁrst or 2 nd LC, and the output is either the WHNF or the DNF of the input term. 
Revised: Depending on the experiment, the input to the network is a term in either the 1 st LC or 2 nd LC, and the output is either the WHNF or the DNF of the input term. [Table] [Table] [Table]

Original:  Terms are encoded as strings, which are then converted to sequences of tokens with ids that areused as inputs to the network.
Revised: [Table] [Figure] [Table] Terms are encoded as strings, which are then converted to sequences of tokens with ids that are used as inputs to the network.

Original: The string representation of a term is obtained by pretty-printingthe term in Haskell syntax (Jones, 2003).
Revised: The string representation of a term is obtained by pretty-printing the term in Haskell syntax (Jones, 2003).

Original: Variables are encoded as x0 , x1 , x2 , etc.
Revised: Variables are encoded as x0 , x1 , x2 , etc.

Original: and lambdaexpressions are encoded as \x0 -> e .
Revised: and lambda expressions are encoded as \x0 -> e .

Original: Application expressions are encoded as e1 e2 .
Revised: Application expressions are encoded as e1 e2 .

Original: Wherenecessary, parentheses are used to denote the precedence of application.
Revised: Where necessary, parentheses are used to denote the precedence of application.

Original: Thus, the 1 st LC term( λ x . x)( λ y .
Revised: Thus, the 1 st LC term ( λ x . x)( λ y .

Original: y) is encoded as (\x0 -> x0) (\x1 -> x1) .
Revised: y) is encoded as (\x0 -> x0) (\x1 -> x1) .

Original: In the 2 nd LC, Nil becomes [] , and
Revised: In the 2 nd LC, Nil becomes [] , and

Original: Cons e 1 e 2 becomes e1 : e2 , while lists of one or more elements are pretty-printed using Haskell’sbuilt-in list syntax: [e1, e2, ...] .
Revised: Cons e 1 e 2 becomes e1 : e2 , while lists of one or more elements are pretty-printed using Haskell’s built-in list syntax: [e1, e2, ...] .

Original: We deviate from the usual Haskell syntax for if-then-elseexpressions: if e 1 then e 2 else e 3 is encoded as ite e1 e2 e3 .
Revised: We deviate from the usual Haskell syntax for if-then-else expressions: if e 1 then e 2 else e 3 is encoded as ite e1 e2 e3 .

Original: The type of a 2 nd LC term is not [Table] [Table]
Revised: The type of a 2 nd LC term is not encoded explicitly.

Original: [Table] [Table] encoded explicitly. Hence, both lambda calculi appear to be untyped.
Revised: Hence, both lambda calculi appear to be untyped to the models except that nonnormalisable terms do not appear in the data.

Revised: Our type-driven generation process avoids generating non-normalising terms.

Original: We distinguish evaluation withvariable renaming (VR) and evaluation without variable renaming (NVR).
Revised: We distinguish evaluation with variable renaming (VR) and evaluation without variable renaming (NVR).

Original: In the former case, thevariables in the reduced program are freshly generated in the order of their appearance, while in thelatter case, the variable names are preserved during reduction, which we expect to be more tractable.
Revised: In the former case, the variables in the reduced program are freshly generated in the order of their appearance, while in the latter case, the variable names are preserved during reduction, which we expect to be more tractable.

Original: We use a pretrained T5 model (Raffel et al., 2020) as our base model.
Revised: We use a pretrained T5 model (Raffel et al., 2020) as our base model.

Original:  We ﬁne-tune the model usingthe Adafactor optimizer (Shazeer and Stern, 2018) with a maximum learning rate of 10 − 4 in a lineardecay schedule and a batch size of 2048 .
Revised: The number of parameters of the T5-Small and T5-Large models can be found in the appendix section A. We ﬁne-tune the model using the Adafactor optimizer (Shazeer and Stern, 2018) with a maximum learning rate of 10 − 4 in a linear decay schedule and a batch size of 2048 .

Original: Predictions are made using greedy decoding.
Revised: Predictions are made using greedy decoding.

Original: A dataset of 1 million unique examples is generated by sampling from the distribution of terms in the2 nd LC, which are subsequently converted to the 1 st LC.
Revised: A dataset of 1 million unique examples is generated by sampling from the distribution of terms in the 2 nd LC, which are subsequently converted to the 1 st LC.

Original: We thus use the same dataset in experimentswith the 1 st LC and the 2 nd LC.
Revised: We thus use the same dataset in experiments with the 1 st LC and the 2 nd LC.

Original: We chose to generate from 2 nd LC so that we could exploit its typesfor type-driven generation of terms.
Revised: We chose to generate from 2 nd LC so that we could exploit its types for type-driven generation of terms.

Original: Terms that are too long to ﬁt within the model’s maximum input( 512 tokens) and output lengths ( 256 tokens) are discarded.
Revised: Terms that are too long to ﬁt within the model’s maximum input ( 512 tokens) and output lengths ( 256 tokens) are discarded.

Original: Performance is evaluated using the average exact string match metric between the normal forms ofthe predicted terms and the normal forms of the terms in the dataset.
Revised: Performance is evaluated using the average exact string match metric between the normal forms of the predicted terms and the normal forms of the ground-truth terms.

Original: A value of 1 indicates that allpredicted terms are correct.
Revised: For a given example, if all predicted terms match 100% with ground-truth terms, an exact match is found.

Revised: The average is taken by dividing the number of exact matches over the total number of examples.

Original: Random Split Our ﬁndings on uniform random splits are summarized in Table 1.
Revised: Random Split Our ﬁndings on uniform random splits are summarized in Table 1.

Original: We report theaverage exact-match results on a held-out dataset of 500 examples of the best performing models.
Revised: We report the average exact-match results on a held-out dataset of 500 examples of the best performing models.

Original: Those models were trained on 90000 examples for up to 100 epochs.
Revised: Those models were trained on 90000 examples for up to 100 epochs.

Original: We ﬁnd that the results for
Revised: We ﬁnd that the results for

Original: T5-Large are all close to the maximum, and that the results for T5-Small are worse and exhibit morevariability: For T5-Small, the average exact-match is around 0 .
Revised: T5-Large are all close to the maximum, and that the results for T5-Small are worse and exhibit more variability: For T5-Small, the average exact-match is around 0 .

Original: 92 for all 2 nd LC reduction tasks,while the numbers for the 1 st LC are much lower.
Revised: 92 for all 2 nd LC reduction tasks, while the numbers for the 1 st LC are much lower.

Original: The 1 st LC results for WHNF-NVR and DNF-VRare particularly weak, below 0 .
Revised: The 1 st LC results for WHNF-NVR and DNF-VR are particularly weak, below 0 .

Original: 6 and 0 . 7 , respectively.
Revised: 6 and 0 . 7 , respectively.

Original: This pattern is also seen in the results for
Revised: This pattern is also seen in the results for

Original: T5-Large to a lesser degree.
Revised: T5-Large to a lesser degree.

Revised: Interestingly, if we do not use pretrained weights but instead use Xavier

Revised: Normal initialization (Glorot and Bengio, 2010), performance drops by close to 40 percentage pointsacross the board.

Revised: This result shows that pretraining has signiﬁcant upside beneﬁts for evaluating neural functional programs.

Revised: We intuit that Natural Language Understanding skills accumulated during pretraining translates well into neural program execution tasks.

Revised: We will leave it for future work to verify this intuition more thoroughly.

Original: Split by Type In this experiment, reported in Table 2, we split the dataset into two parts based onthe types of 2 nd LC terms. 
Revised: Split by Type In this experiment, reported in Table 2, we split the dataset into two parts based on the types of 2 nd LC terms. The examples are ordered by frequency of type occurrence.

Original: Theexamples in the training set contain the80% most common types,with the test set containing the remaining 20% .
Revised: The training set contains the most common types representing 80% of the dataset, with the test set containing the remaining 20% comprised of different and less common types.

Original:  We subsample the training set to 80000 examplesand the test set to 500 examples.
Revised: 3 We subsample the training set to 80000 examples and the test set to 500 examples.

Original: Despite the changes, the results are similar to the uniform randomsplit in Table 1.
Revised: Despite the changes, the results are similar to the uniform random split in Table 1.

Original: Performance decrease is strongest for the T5-Small model, while the T5-Large model performs almost at the level of the uniform random split, except for a 6% performance drop on the DNF tasks.
Revised: For pretrained models, performance decrease is strongest for the T5-Small model, while the T5-Large model performs almost at the level of the uniform random split, except for a 6 percentage-point performance drop on the DNF tasks.

Original: We attribute the absence of this drop for the WHNF tasks to the fact that reduction to WHNF tends to be concentrated around the root term, which is well covered by the training set.
Revised: We attribute the absence of this drop for the WHNF tasks to the fact that reduction to WHNF tends to be concentrated around the root term, which is well covered by the training set.

Revised: Similarly to previous experiments, we see that pretraining has a larger impact on performance than model size.

Revised: The performance drops by around 40 percentage points with training from scratch.

Revised: In all experiments, we observe that 2 nd LC performs slightly worse ( − 9 percentage points on average) for large models.

Original: Split by Function Composition For this experiment, we create a new evaluation dataset from thetraining examples used in the ﬁrst experiment on random splits.
Revised: Split by Function Composition For this experiment, we create a new evaluation dataset from the training examples used in the ﬁrst experiment on random splits.

Original: We produce unique examples bycomposing terms e 1 , e 2 from the training set using application, e 1 e 2 .
Revised: We produce unique examples by composing terms e 1 , e 2 from the training set using application, e 1 e 2 .

Original: The ﬁnal dataset contains 500examples of each lambda calculus and evaluation strategy.
Revised: The ﬁnal dataset containsexamples of each lambda calculus and evaluation strategy.

Original: We use the best performing models on therandom splits to evaluate the performance of the new dataset.
Revised: We use the best performing models on the random splits to evaluate the performance of the new dataset.

Original: Our ﬁndings, reported in Table 3, showthat the performance on the compositional evaluation dataset is lower compared to the random split.
Revised: Our ﬁndings, reported in Table 3, show that the performance on the compositional evaluation dataset is lower compared to the random split.

Original: The 2 nd LC results are slightly better than the 1 st LC results, and among the 1 st LC results, those onthe WHNF-VR task are particularly poor.
Revised: The 2 nd LC results are slightly better than the 1 st LC results, and among the 1 st LC results, those on the WHNF-VR task are particularly poor.

Original: We also analyzed the results on the new dataset by token length.
Revised: We also analyzed the results on the new dataset by token length.

Original: 1 st LC input and target programs areboth about 2 .
Revised: 1 st LC input and target programs are both about 2 .

Original: 5 times as long as their 2 nd LC counterparts.
Revised: 5 times as long as their 2 nd LC counterparts.

Original: When compensated for this difference, thedependence of the exact-match performance on the target length is about the same for both languages:it stays high and approximately constant for targets below 100 tokens for 1 st LC and 50 tokens for2 nd LC, and then falls off linearly with length.
Revised: When compensated for this difference, the dependence of the exact-match performance on the target length is about the same for both languages: it stays high and approximately constant for targets below 100 tokens for 1 st LC and 50 tokens for 2 nd LC, and then falls off linearly with length.

Original: See appendix A for details.
Revised: See section 3 for details.

Original: Split by Number of Reduction Steps This last experiment splits the data such that the trainingexamples contain the fewest reduction steps and the test set the most reduction steps.
Revised: Split by Number of Reduction Steps This last experiment splits the data such that the training examples contain the fewest reduction steps and the test set the most reduction steps.

Original: Since largerreduction step counts imply greater complexity, we expect a performance drop compared to therandom split.
Revised: Since larger reduction step counts imply greater complexity, we expect a performance drop compared to the random split.

Original: Our results are presented in Table 4.
Revised: Our results are presented in Table 4.

Original: Consider the WHNF reduction task.
Revised: Consider the WHNF reduction task.

Original: The 1 st LC and 2 nd LC have median reduction step counts of 4and 3 , respectively.
Revised: The 1 st LC and 2 nd LC have median reduction step counts ofand 3 , respectively.

Original: In order to cover the bulk of the distribution of reduction steps, training exampleshave therefore up to 6 reduction steps.
Revised: In order to cover the bulk of the distribution of reduction steps, training examples have therefore up to 6 reduction steps.

Original: Larger counts up to 12 steps are in the test set.
Revised: Larger counts up to 12 steps are in the test set.

Original: We subsampledthe training and test sets such that the proportion of examples per reduction step count is the samebetween the two lambda calculi.
Revised: We subsampled the training and test sets such that the proportion of examples per reduction step count is the same between the two lambda calculi.

Original: We ﬁnd that the 2 nd LC model performs much better than the 1 st
Revised: We ﬁnd that the 2 nd LC model performs much better than the 1 st

Original: LC model on the test set: For 1 st LC, the performance drops by one third compared to the randomsplit, while for 2 nd LC the performance drops by about 7% .
Revised: LC model on the test set: For 1 st LC, the performance drops by one third compared to the random split, while for 2 nd LC the performance drops by about 7 percentage points.

Original: For the DNF reduction task, the 1 st LCand 2 nd LC have median reduction step counts of 6 and 4 , respectively.
Revised: For the DNF reduction task, the 1 st LC and 2 nd LC have median reduction step counts of 6 and 4 , respectively.

Original: The training examples haveup to 8 reductions steps, and examples with between 9 and 32 reduction steps are test examples.
Revised: The training examples have up to 8 reductions steps, and examples with between 9 and 32 reduction steps are of the model. We do not observe numbers above 0 . 35 .

Revised: The six most common types are: Bool , Unit , List Bool , List Unit , Unit → Bool , and Bool → Unit .

Revised: These types are among those found in the training set, while the types in the test set are much more nested, e.g., (Unit → List List Bool → List Unit)

Revised: → List List List List Unit .

Original:  Subsampling equivalent to that of the WHNF task was applied.
Revised: test examples. Subsampling equivalent to that of the WHNF task was applied.

Original: The results for DNF reduction aresimilar to those obtained for WHNF.
Revised: The results for DNF reduction are similar to those obtained for WHNF.

Original: Again, the drop in performance is bigger for the 1 st LC modelthan for the 2 nd LC model. 
Revised: Again, the drop in performance is bigger for the 1 st LC model than for the 2 nd LC model. Input and Output Program Length

Revised: To verify if program input and output lengths (number of tokens) may explain differences between 1 st LC and 2 nd LC, we show exact match performance for increasing program lengths in Figures 1 (a) and 1 (b).

Revised: We grouped the data in 10 different program length ranges with equal amounts of examples.

Revised: The graphs plot the average program length of each range (x-axis) against the average exact match performance (y-axis).

Original: Concluding Remarks
Revised: 2 nd LC program inputs are generally smaller. However, for WHNF and NVR, when 2 nd LC and 1 st LC length intersect, 1 st LC exact match is close to 5 percentage points lower. Interestingly, for 1 st LC, performance tends to increase with input length as shown in Figure 1 (a). Output program lengths are typically 2.5 times smaller for 2 nd LC.

Revised: Curves in Figure 1 (b) display wave patterns with clear downward trends as program length increases.

Revised: For the same length, the 1 st LC results are not consistently at or below the 2 nd LC results.

Revised: Instead, the 1 st LC results appear shifted and scaled up along the length-axis.

Revised: Overall, we notice that exact-match performance on the output length is about the same for both languages: it stays high and approximately constant for targets below 50 tokens for 1 st LC and 100 tokens for 2 nd

Revised: LC and then falls off linearly with length.

Original:  We compare neural execution of programs in two lambda calculi, 1 st LC and 2 nd LC.
Revised: [Figure]  Concluding Remarks We compare neural execution of programs in two lambda calculi, 1 st LC and 2 nd LC.

Original: Both expresssemantically equivalent programs, but 1 st LC uses a simpler syntax at the cost of longer programlength.
Revised: Both express semantically equivalent programs, but 1 st LC uses a simpler syntax at the cost of longer program length.

Original: Our experiments with T5 have shown near tractable performance for synthetic in-distribution1 st LC and 2 nd LC data, but stark performance differences are observed for out-of-distribution data.
Revised: Our experiments with T5 have shown near tractable performance for synthetic in-distribution 1 st LC and 2 nd LC data, but stark performance differences are observed for out-of-distribution data.

Original: We analyze our models’ generalization capability on four splits of the data: uniformly random, byprogram type, by function composition, and by number of reduction steps.
Revised: We analyze our models’ generalization capability on four splits of the data: uniformly random, by program type, by function composition, and by number of reduction steps.

Original: On average,  1 st LC modelsgeneralize less then  2 nd LC models.
Revised: On average, models trained on 1 st LC generalize less then models trained on 2 nd LC.

Revised: We observe signiﬁcant gains from using T5 with pretrained weights compared to randomly initialized models on in-distribution and unseen-type experiments.

Original: As discussed in appendix A, input/output program lengths cannotcompletely account for differences in languages.
Revised: Input/output program lengths cannot completely account for differences in languages.

Original: Our experiments indicate that neural interpretationof functional programs is more tractable when expressed in more abstract syntax, but further evidenceis needed to  support this claim.
Revised: Our experiments indicate that neural interpretation of functional programs is more tractable when expressed in more “sugared” syntax, but further evidence is needed to completely support this claim.

Original:  T5-Large is consistently better than T5-Small.
Revised: Further, we show that T5-Large is consistently better than T5-Small.

Original: Pretaining on the T5corpus and tasks yields better results than training from scratch.
Revised: Pretraining on the T5 corpus and tasks yields better results than training from scratch.

Original: Our analysis of different evaluationstrategies show that lazy reduction is slightly easier to learn than eager reduction, but may generalizeless on nontrivial splits of the data.
Revised: Our analysis of different evaluation strategies show that lazy reduction is slightly easier to learn than eager reduction, but may generalize less on nontrivial splits of the data.

Original: Results with and without variable renaming do not show a cleartrend.
Revised: Results with and without variable renaming do not show a clear trend.

Original: This is unexpected since variable renaming is an additional syntactic operation that complicatesthe task and is not strictly necessary.
Revised: This is unexpected since variable renaming is an additional syntactic operation that complicates the task and is not strictly necessary.

Original: Future work on more difﬁcult tasks and data splits will allow us to better understand the generalizationproperties of our models as well as the impact of syntactic complexity on performance. 
Revised: Future work on more difﬁcult tasks and data splits will allow us to better understand the generalization properties of our models as well as the impact of syntactic complexity on performance. A Model sizes Model sizes are listed in Table 5.

Original: A Additional Experimental Results
Revised: [Table] B Example Data

Original: To verify if program input and output lengths (number of tokens) may explain differences between1 st LC and 2 nd LC, we show exact match performance for increasing program lengths in ﬁgures
Revised: The Table 6 lists parallel examples of 1 st LC and 2 nd LC terms and their deep normal forms from our dataset. 1 st LC (\x0 -> \x1 -> \x2 -> x1) ((\x3 -> \x4 -> (\x5 -> \x6 -> x6) (\x7 -> (\x8 -> \x9 -> x7) x4) ((\x10 -> \x11 -> \x12 -> \x13 -> x12 x (x11 x12 x13))

Revised: (\x14 -> x14) (\x15 -> \x16 -> x16)))

Original: 1and 2.
Revised: (\x17 -> \x18 -> x17)) 1 st LC DNF \x0 -> \x1 -> x 2 nd LC (\ [Table] >

Original: We grouped the data in 10 different program length ranges with equal amounts of examples.
Revised: [Table] [Table] x [Table]

Revised: >

Revised: [Table]

Revised: > foldr

Revised: [Table]

Original: The graphs plot the average program length of each range (x-axis) against the average exact matchperformance (y-axis).
Revised: > [Table] > [Table]

Original: 2 nd LC program inputs are generally smaller.
Revised: 2 nd LC DNF True 1 st LC (\ [Table] > \

Revised: [Table]

Revised: >

Revised: [Table]

Revised: [Table]

Revised: >

Revised: [Table]

Revised: >

Revised: [Table]

Revised: >

Revised: [Table]

Revised: >

Revised: [Table]

Revised: >

Revised: [Table]

Revised: >

Revised: [Table]

Revised: >

Revised: [Table]

Revised: >

Revised: [Table]

Revised: >

Revised: [Table]

Original: However, for WHNF and NVR,when 2 nd LC and 1 st LC length intersect, 1 st LC exact match is close to 5% lower.
Revised: > [Table] > [Table]

Revised: >

Original: Interestinglyfor 1 st LC, performance seems to increase with input length as shown in ﬁgure 1.
Revised: [Table] 1 st LC DNF \x0 -> [Table] >

Revised: [Table]

Revised: >

Revised: [Table]

Revised: >

Revised: [Table]

Revised: >

Revised: [Table]

Original: Output programlengths are typically 2.5 times smaller for 2 nd LC.
Revised: 2 nd LC ite False (\ [Table] > [Table]

Revised: > \

Revised: [Table]

Revised: > []) (\x

Revised: [Table]

Revised: > \

Original: Curves in ﬁgure 2 display wave patterns withoutclear downward trends as program length increases.
Revised: [Table] -> [Table] [Table]

Revised: -> \

Revised: [Table]

Revised: -> \

Original: For the same length, the 1 st LC results are notconsistently at or below the 2 nd LC results.
Revised: [Table] -> []) x [Table] 2 nd LC DNF \x0 -> \x1 ->

Revised: \x

Revised: [Table]

Original: Instead, the 1 st LC results appear shifted and scaled upalong the length-axis.
Revised: > [] 1 st LC (\x0 -> \x1 -> [Table] ) (\x2 -> \x3 -> x3) (\x4 -> x4) ((\x5 -> \x6 -> \ [Table]

Revised: -> x4) (\x5 -> x5))

Revised: ((\x6 -> \x7 -> \x8 -> x8) ((\x9 -> (\x10 -> \x11 -> \x12 -> \x13 -> x12 x10 (x11 x12 x13))

Revised: ((\x14 -> \x15 -> x15) (\x16 -> \x17 -> x17) (\x18 -> \x19 -> x18))

Revised: ((\x20 -> \x21 -> \x22 -> x22) x9) (\x23 -> (\x24 -> \x25 -> x24) (\x26 -> x23) (\x27 -> x23))

Revised: (\x28 -> \x29 -> x28))

Revised: (\x30 -> x30)))

Original: Overall, we notice that exact-match performance on the output length is aboutthe same for both languages: it stays high and approximately constant for targets below 50 tokens for1 st LC and 100 tokens for 2 nd LC and then falls off linearly with length.
Revised: 1 st LC DNF \x0 -> \x1 -> x0 (\x2 -> x2) x 2 nd LC (:) ((\x0 -> x0) ()) ((\x1 -> []) ((\x2 -> foldr (\x3 -> ite True (\x4 -> x3) (\x5 -> x3)) True ((:) (ite False False True) ((\x6 -> []) x2))) ())) 2 nd LC DNF [()] 1 st LC (\x0 -> \x1 -> x1) (\x2 -> (\x3 -> \x4 -> x4) (\x5 -> (\x6 -> (\x7 -> (\x8 -> \x9 -> x9) x7) ((\x10 -> \x11 -> \x12 -> x11) x2))

Revised: x2) (\x13 -> x13))

Revised: (\x14 -> (\x15 -> \x16 -> x16) (\x17 -> (\x18 -> \x19 -> \x20 ->

Revised: [Table]