Original: Repository-Level Prompt Generation for Large Language Models of Code
Revised: Repository-Level Prompt Generation for Large Language Models of Code

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: With the success of large language models (LLMs) of code and their use as code assistants (e.g. Codex (Chen et al., 2021) used in GitHub
Revised: With the success of large language models (LLMs) of code and their use as code assistants (e.g. Codex (Chen et al., 2021) used in GitHub

Original: Copilot 1 ), development of techniques where we can have the capability to introduce domainspecific knowledge in the prompt design process becomes important.
Revised: Copilot 1 ), development of techniques where we can have the capability to introduce domainspecific knowledge in the prompt design process becomes important.

Original: In this work, we propose a framework called Repo-Level Prompt Generator that learns to generate example-specific prompts using a set of rules.
Revised: In this work, we propose a framework called Repo-Level Prompt Generator that learns to generate example-specific prompts using a set of rules.

Original: These rules allow us to take context from the entire repository, thereby incorporating both the structure of the repository and the context from other relevant files (e.g. imports, parent class files).
Revised: These rules allow us to take context from the entire repository, thereby incorporating both the structure of the repository and the context from other relevant files (e.g. imports, parent class files).

Original: Our technique doesn’t require any access to the weights of the LLM, making it applicable in cases where we only have a blackbox access to the LLM.
Revised: Our technique doesn’t require any access to the weights of the LLM, making it applicable in cases where we only have a blackbox access to the LLM.

Original: We conduct experiments on the task of single-line code-autocompletion using code repositories taken from Google Code archives.
Revised: We conduct experiments on the task of single-line code-autocompletion using code repositories taken from Google Code archives.

Original: We demonstrate that an oracle constructed from our proposed rules gives up to 36% relative improvement over Codex, showing the quality of our proposed rules.
Revised: We demonstrate that an oracle constructed from our proposed rules gives up to 36% relative improvement over Codex, showing the quality of our proposed rules.

Original: Further, we show that when we train a model to select the best rule, we can achieve significant performance gains over
Revised: Further, we show that when we train a model to select the best rule, we can achieve significant performance gains over

Original: Codex.
Revised: Codex.

Original: 1. Introduction
Revised: 1. Introduction

Original: Large Language Models (LLMs) have shown promising results in the area of natural language processing, where models such as GPT-3 (Brown et al., 2020) and PaLM (Chowdhery et al., 2022) demonstrated remarkable performance.
Revised: Large Language Models (LLMs) have shown promising results in the area of natural language processing, where models such as GPT-3 (Brown et al., 2020) and PaLM (Chowdhery et al., 2022) demonstrated remarkable performance.

Original: Recently, LLMs have been used for modeling source code as well (Austin et al., 
Revised: Recently, LLMs have been used for modeling source code as well (Austin et al., 2021;

Original:  2022a;
Revised: Fried et al., 2022; Xu et al., 2022a;

Original: Feng et al., 2020).
Revised: Feng et al., 2020).

Original: Codex (Chen et al., 2021) is one of the earliest LLM that contains about 12 billion parameters and was trained on publicly available code from GitHub.
Revised: Codex (Chen et al., 2021) is one of the earliest LLM that contains about 12 billion parameters and was trained on publicly available code from GitHub.

Original: In particular, Codex has been deployed as part of GitHub
Revised: In particular, Codex has been deployed as part of GitHub

Original: Copilot 1 , which is an in-IDE code assistant.
Revised: Copilot 1 , which is an in-IDE code assistant.

Original: Despite the success of LLM for code, as illustrated in Figure 1 of Xu et al.
Revised: Despite the success of LLM for code, as illustrated in Figure 1 of Xu et al.

Original: (2022a)
Revised: (2022a)

Original: and mentioned in their paper "the strongest models are not publicly available".
Revised: and mentioned in their paper "the strongest models are not publicly available".

Original: For instance,
Revised: For instance,

Original: OpenAI provides access only to the generated output from
Revised: OpenAI provides access only to the generated output from

Original: Codex via an API 2 .
Revised: Codex via an API 2 .

Original: The model weights and data are unavailable.
Revised: The model weights and data are unavailable.

Original: Under this setting, most of the existing prompt generation techniques (Li & Liang, 2021; Shin et al., 2020;
Revised: Under this setting, most of the existing prompt generation techniques (Li & Liang, 2021; Shin et al., 2020;

Original: Lester et al., 2021) can’t be used because they require access to some or all of the weights of the LLM.
Revised: Lester et al., 2021) can’t be used because they require access to some or all of the weights of the LLM.

Original: Automatically generating an example-specific prompt is challenging and remains an active area of research (Liu et al., 2021a; Reynolds & McDonell, 2021).
Revised: Automatically generating an example-specific prompt is challenging and remains an active area of research (Liu et al., 2021a; Reynolds & McDonell, 2021).

Original: With increasing use of LLMs as assistive agents to humans, it is important to generate prompts that are easier for humans to understand and intuitive enough for them to change if desired.
Revised: With increasing use of LLMs as assistive agents to humans, it is important to generate prompts that are easier for humans to understand and intuitive enough for them to change if desired.

Original: In this work, we propose a framework called Repo-Level Prompt
Revised: In this work, we propose a framework called Repo-Level Prompt

Original: Generator (RLPG) that enables the capability to incorporate domain-specific knowledge in the prompt-designing process via a set of rules .
Revised: Generator (RLPG) that enables the capability to incorporate domain-specific knowledge in the prompt-designing process via a set of rules .

Original: One of the distinguishing features of
Revised: One of the distinguishing features of

Original: RLPG is that it considers the code context not just from the current file, but from the entire repository, thereby allowing us to incorporate both the structure of the repository as well as the context in the other files in the repository.
Revised: RLPG is that it considers the code context not just from the current file, but from the entire repository, thereby allowing us to incorporate both the structure of the repository as well as the context in the other files in the repository.

Original: This makes particular sense for source code where context coming from outside the current file, such as imports and parent classes, can be very useful for predicting the hole.
Revised: This makes particular sense for source code where context coming from outside the current file, such as imports and parent classes, can be very useful for predicting the hole.

Original: The choice of where from and what to take from the repository is specified by each rule.
Revised: The choice of where from and what to take from the repository is specified by each rule.

Original: For example, we could have a rule that returns all the identifiers used in the first import file.
Revised: For example, we could have a rule that returns all the identifiers used in the first import file.

Original: The prompts generated by our rules are discrete, similar to suggested in some previous works in NLP (Shin et al., 2020; Schick &
Revised: The prompts generated by our rules are discrete, similar to suggested in some previous works in NLP (Shin et al., 2020; Schick &

Original: Schütze, 2021).
Revised: Schütze, 2021).

Original: However, rather than fixing one particular rule for each example, we propose a Rule Classifier neural module that learns to come up with the best rule conditioned on that example, such that the resulting prompt is likely to produce the desired output.
Revised: However, rather than fixing one particular rule for each example, we propose a Rule Classifier neural module that learns to come up with the best rule conditioned on that example, such that the resulting prompt is likely to produce the desired output.

Original: In this work, we focus on the task of single-line codeautocompletion in a line-level maintenance setting (Shrivastava et al., 2020) in an IDE, where the objective is to predict the blanked-out portion (or target hole ) starting from the position of an imagined cursor to the end of line.
Revised: In this work, we focus on the task of single-line codeautocompletion in a line-level maintenance setting (Shrivastava et al., 2020) in an IDE, where the objective is to predict the blanked-out portion (or target hole ) starting from the position of an imagined cursor to the end of line.

Original: There might also be code that follows below this line.
Revised: There might also be code that follows below this line.

Original: This reflects the setting where a user is editing an existing file.
Revised: This reflects the setting where a user is editing an existing file.

Original: Figureprovides an illustration of our system in progress.
Revised: Figureprovides an illustration of our system in progress.

Original: The rule classifier takes in the hole position (position of the cursor) in the current file, the repository to which the current file is a part of and a list of rules as input, and chooses a particular rule to be used for that hole prediction.
Revised: The rule classifier takes in the hole position (position of the cursor) in the current file, the repository to which the current file is a part of and a list of rules as input, and chooses a particular rule to be used for that hole prediction.

Original: In the figure, the chosen rule corresponds to taking the method names and bodies from the import file (i.e., rule context ).
Revised: In the figure, the chosen rule corresponds to taking the method names and bodies from the import file (i.e., rule context ).

Original: The Prompt
Revised: The Prompt

Original: Generator uses this rule context and concatenates it with 50% of the default Codex context to get the prompt.
Revised: Generator uses this rule context and concatenates it with 50% of the default Codex context to get the prompt.

Original: The default Codex context consists of code prior to the position of the hole in the current file.
Revised: The default Codex context consists of code prior to the position of the hole in the current file.

Original: The generated prompt can then be given as input to Codex so it can make a prediction for the hole (shown in the brown box on the top) as output.
Revised: The generated prompt can then be given as input to Codex so it can make a prediction for the hole (shown in the brown box on the top) as output.

Original: Our key contributions can be listed as follows:
Revised: Our key contributions can be listed as follows:

Original: • We propose a framework called the Repo-Level Prompt Generator (RLPG) that learns to generate prompts conditioned on the example, without requiring access to the weights of the LLM.
Revised: • We propose a framework called the Repo-Level Prompt Generator (RLPG) that learns to generate prompts conditioned on the example, without requiring access to the weights of the LLM.

Original: • To incorporate domain-knowledge in the prompt design process, RLPG uses a set of repository-level rules.
Revised: • To incorporate domain-knowledge in the prompt design process, RLPG uses a set of repository-level rules.

Original: These rules are designed to incorporate both the structure of the repository as well as the relevant context from other files in the repository.
Revised: These rules are designed to incorporate both the structure of the repository as well as the relevant context from other files in the repository.

Original: • On the task of single-line code-autocompletion using code repositories in Java, we show that an oracle constructed from our proposed rules gives up to 36% relative improvement over Codex.
Revised: • On the task of single-line code-autocompletion using code repositories in Java, we show that an oracle constructed from our proposed rules gives up to 36% relative improvement over Codex.

Original: Further, we show that when we use our rule classifier to select the best rule, we can achieve up to 17% relative improvement over Codex.
Revised: Further, we show that when we use our rule classifier to select the best rule, we can achieve up to 17% relative improvement over Codex.

Original: 2. Repo-Level Prompt Generator (RLPG)
Revised: 2. Repo-Level Prompt Generator (RLPG)

Original: In this section, we provide details of our framework.
Revised: In this section, we provide details of our framework.

Original: We start by describing our proposed rules and then follow with a discussion about our rule classifier.
Revised: We start by describing our proposed rules and then follow with a discussion about our rule classifier.

Original: Finally, we briefly describe the Prompt Generator module.
Revised: Finally, we briefly describe the Prompt Generator module.

Original: 2.1. Repo-Level Rules
Revised: 2.1. Repo-Level Rules

Original: The core idea of RLPG consists of substituting part of the default Codex context with context coming from somewhere else in the repository.
Revised: The core idea of RLPG consists of substituting part of the default Codex context with context coming from somewhere else in the repository.

Original: The decision of what to take and from where in the repository to take from is governed by a set of rules.
Revised: The decision of what to take and from where in the repository to take from is governed by a set of rules.

Original: These rules were decided based on manual inspection of our training data and intend to capture common coding patterns.
Revised: These rules were decided based on manual inspection of our training data and intend to capture common coding patterns.

Original: A rule can be thought of as a function that takes in a target hole’s position and the repository that the hole is a part of as input, and returns the rule context (string).
Revised: A rule can be thought of as a function that takes in a target hole’s position and the repository that the hole is a part of as input, and returns the rule context (string).

Original: A rule is specified by three quantities: (a) rule context location, (b) rule context type and (c) rule context fraction.
Revised: A rule is specified by three quantities: (a) rule context location, (b) rule context type and (c) rule context fraction.

Original: We mention each of these along with the motivation for them below.
Revised: We mention each of these along with the motivation for them below.

Original: Rule Context Location: For a target hole position, a rule’s context location determines from where should we take code that will be part of the rule context.
Revised: Rule Context Location: For a target hole position, a rule’s context location determines from where should we take code that will be part of the rule context.

Original: We propose ten different rule context locations: (a) Current, (b) Parent Class, (c) Import, (d) Sibling, (e) Similar Name, (f) Child Class, (g) Import of Parent Class, (h) Import of Sibling, (i) Import of Similar Name, and (j) Import of Child Class.
Revised: We propose ten different rule context locations: (a) Current, (b) Parent Class, (c) Import, (d) Sibling, (e) Similar Name, (f) Child Class, (g) Import of Parent Class, (h) Import of Sibling, (i) Import of Similar Name, and (j) Import of Child Class.

Original: For each rule context location, we use a heuristic to get either a single file or a ranked list of files.
Revised: For each rule context location, we use a heuristic to get either a single file or a ranked list of files.

Original: In the latter case, we will take context from these files until we exhaust the maximum context length allocated to the rule.
Revised: In the latter case, we will take context from these files until we exhaust the maximum context length allocated to the rule.

Original: See Appendix A.1 for details on the rule context locations and Appendix A.4 for details of the ranking of files based on their rule context location.
Revised: See Appendix A.1 for details on the rule context locations and Appendix A.4 for details of the ranking of files based on their rule context location.

Original: Rule Context Type: The rule context type determines what code to take from the rule context location.
Revised: Rule Context Type: The rule context type determines what code to take from the rule context location.

Original: We propose seven different rule context types: (a) Post Lines (PL), (b)
Revised: We propose seven different rule context types: (a) Post Lines (PL), (b)

Original: Identifiers (I), (c) Type Identifiers (TI), (d) Field Declarations (FD), (e) String Literals (SL), (f) Method Names (MN), and (g) Method Names and Bodies (MNB).
Revised: Identifiers (I), (c) Type Identifiers (TI), (d) Field Declarations (FD), (e) String Literals (SL), (f) Method Names (MN), and (g) Method Names and Bodies (MNB).

Original: See Appendix A.2 for more details.
Revised: See Appendix A.2 for more details.

Original: Rule Context Fraction: The rule context fraction determines the fraction of the final prompt that will be made of the code coming from the rule context.
Revised: Rule Context Fraction: The rule context fraction determines the fraction of the final prompt that will be made of the code coming from the rule context.

Original: We use rule context fraction values of 0.25, 0.5 and 0.75.
Revised: We use rule context fraction values of 0.25, 0.5 and 0.75.

Original: Appendix A.3 for a list of our proposed rules.
Revised: Appendix A.3 for a list of our proposed rules.

Original: Note that depending on the target hole, not all rules will be applicable (e.g. if there are no parent classes in the current file, rules 8-13 and 50-55 won’t be applicable.
Revised: Note that depending on the target hole, not all rules will be applicable (e.g. if there are no parent classes in the current file, rules 8-13 and 50-55 won’t be applicable.

Original: 2.2. Rule Classifier
Revised: 2.2. Rule Classifier

Original: Given a hole position, the purpose of the rule classifier is to select the best rule r that is likely to produce a prompt that will lead to success, where success happens when the target hole h exactly matches the predicted hole ˆ h .
Revised: Given a hole position, the purpose of the rule classifier is to select the best rule r that is likely to produce a prompt that will lead to success, where success happens when the target hole h exactly matches the predicted hole ˆ h .

Original: This task is formulated as a multi-label binary classification problem because for a given target hole, more than one rule can lead to success.
Revised: This task is formulated as a multi-label binary classification problem because for a given target hole, more than one rule can lead to success.

Original: In this formulation, we treat using the default
Revised: In this formulation, we treat using the default

Original: Codex context as complete prompt, as one of the rules.
Revised: Codex context as complete prompt, as one of the rules.

Original: Next, we describes the training procedure for the rule classifier.
Revised: Next, we describes the training procedure for the rule classifier.

Original: Training: For each target hole h , we generate a groundtruth vector Y h
Revised: Training: For each target hole h , we generate a groundtruth vector Y h

Original: = { y hr } Mr =1 which is a multi-hot vector of size M , where M is the total number of rules.
Revised: = { y hr } Mr =1 which is a multi-hot vector of size M , where M is the total number of rules.

Original: This vector is obtained by feeding the prompt generated by using the context from rule r into Codex and then seeing whether ˆ h
Revised: This vector is obtained by feeding the prompt generated by using the context from rule r into Codex and then seeing whether ˆ h

Original: = h .
Revised: = h .

Original: If there is a match, we say that the rule r is successful.
Revised: If there is a match, we say that the rule r is successful.

Original: For hole h if a rule r is applicable and leads to success, y hr = 1 or will be zero otherwise.
Revised: For hole h if a rule r is applicable and leads to success, y hr = 1 or will be zero otherwise.

Original: For each hole h , we obtain a mask T h where T hr = 1 when r is applicable or
Revised: For each hole h , we obtain a mask T h where T hr = 1 when r is applicable or

Original: T hr = 0 otherwise.
Revised: T hr = 0 otherwise.

Original: The overall training loss, L can be expressed as follows:
Revised: The overall training loss, L can be expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: In the above equation, N is the total number of holes encountered while training, M h denotes the total number of applicable rules for h and BCE corresponds to the binary cross entropy loss.
Revised: In the above equation, N is the total number of holes encountered while training, M h denotes the total number of applicable rules for h and BCE corresponds to the binary cross entropy loss.

Original: Masking ensures that we consider only the rules that are applicable.
Revised: Masking ensures that we consider only the rules that are applicable.

Original: Next, we describe two variants of our rule classifier that can be used to obtain the prediction ˆ y hr .
Revised: Next, we describe two variants of our rule classifier that can be used to obtain the prediction ˆ y hr .

Original: RLPG-H: Let H h be the hole window that includes code present around the hole h excluding the hole itself.
Revised: RLPG-H: Let H h be the hole window that includes code present around the hole h excluding the hole itself.

Original: In our work, we take two lines before the hole position, the code up to the hole position and two lines after the hole position.
Revised: In our work, we take two lines before the hole position, the code up to the hole position and two lines after the hole position.

Original: We use a pretrained model F ϕ to obtain a context representation vector of size Z where Z is the dimension of the hidden state of the model F ϕ .
Revised: We use a pretrained model F ϕ to obtain a context representation vector of size Z where Z is the dimension of the hidden state of the model F ϕ .

Original: Specifically, we take the hidden state at the first position, i.e. the representation of the [CLS] token.
Revised: Specifically, we take the hidden state at the first position, i.e. the representation of the [CLS] token.

Original: To make training of the rule classifier computationally efficient, the parameters ϕ are frozen during training.
Revised: To make training of the rule classifier computationally efficient, the parameters ϕ are frozen during training.

Original: The
Revised: The

Original: RLPG-H model takes the context representation of the hole window and projects it to the rule space of size M via two dense layers with a non-linearity in between.
Revised: RLPG-H model takes the context representation of the hole window and projects it to the rule space of size M via two dense layers with a non-linearity in between.

Original: Taking sigmoid of this output gives the prediction of the rule.
Revised: Taking sigmoid of this output gives the prediction of the rule.

Original: See
Revised: See

Original: Appendix D.1 for more details.
Revised: Appendix D.1 for more details.

Original: RLPG-R: The motivation behind this variant is to use the similarity in the hole window and the rule context to determine which rule can be useful.
Revised: RLPG-R: The motivation behind this variant is to use the similarity in the hole window and the rule context to determine which rule can be useful.

Original: Given a particular hole h , let C hr denote the rule context from rule r .
Revised: Given a particular hole h , let C h r denote the rule context from rule r .

Original: Intuitively, if the hole window contains variables, say identifiers that are similar to the variables in the rule context, then there are chances that h might occur somewhere in C hr .
Revised: Intuitively, if the hole window contains variables, say identifiers that are similar to the variables in the rule context, then there are chances that h might occur somewhere in C hr .

Original: The similarity is modeled using an attention mechanism, by treating the projected hole window representation Q h as a query and the projected rule context representation, K hr (Equation 3) as a key.
Revised: The similarity is modeled using an attention mechanism, by treating the projected hole window representation Q h as a query and the projected rule context representation, K hr (Equation 3) as a key.

Original: The value V hr is same as the key.
Revised: The value V hr is same as the key.

Original: See Appendix D. for more details.
Revised: See Appendix D. for more details.

Original: 2.3. Prompt Generator
Revised: 2.3. Prompt Generator

Original: The Prompt Generator takes the rule r chosen by the rule classifier, the repository and the hole position as input and generates the prompt as output.
Revised: The Prompt Generator takes the rule r chosen by the rule classifier, the repository and the hole position as input and generates the prompt as output.

Original: The rule context is always added before the default Codex context. 
Revised: The rule context is always added before the default Codex context. [Table]

Original: 3. Experiments and Results
Revised: 3. Experiments and Results

Original: Dataset Creation: We scraped Google Code 3 for repositories in Java.
Revised: Dataset Creation: We scraped Google Code 3 for repositories in Java.

Original: We divided the repositories into train, validation and test splits where each repository in its entirety is part of a split.
Revised: We divided the repositories into train, validation and test splits where each repository in its entirety is part of a split.

Original: In each file within a repository, we remove lines that are blanks and comments, and set the hole position to be the middle character in the line.
Revised: In each file within a repository, we remove lines that are blanks and comments, and set the hole position to be the middle character in the line.

Original: See Appendix C for complete details.
Revised: See Appendix C for complete details.

Original: Methods: The details of the methods used in our experiments is given below.
Revised: Methods: The details of the methods used in our experiments is given below.

Original: Foe each method, to measure success, we used exact match between the predicted hole string gener-
Revised: Foe each method, to measure success, we used exact match between the predicted hole string generated by Codex and the target hole string.

Original: DataSplit [Figure]
Revised: In our experiments, we report this fraction, called success rate (SR).

Original: Codex: Using the default Codex context as prompt.
Revised: Codex: Using the default Codex context as prompt.

Original: Oracle: Using the ground-truth vector Y h (mentioned in Section 2.
Revised: Oracle: Using the ground-truth vector Y h (mentioned in Section 2.2).

Original: The prompt generated corresponds to using any of the successful rules with y hr = 1 .
Revised: The prompt generated corresponds to using any of the successful rules with y hr = 1 .

Original: Since this information is not available at inference, the oracle performance represents an upper bound
Revised: Since this information is not available at inference, the oracle performance represents an upper bound.

Original:  k ): Using k prompts generated with top- k rules.
Revised: Fixed Rule ( k ): Using k prompts generated with top- k rules.

Original: The k rules are fixed for all target holes.
Revised: The k rules are fixed for all target holes.

Original: The top- k rules were decided based on decreasing order of success rate of the individual rules on the validation dataset.
Revised: The top- k rules were decided based on decreasing order of success rate of the individual rules on the validation dataset.

Original: When k = 1 , this corresponds to generating a single prompt using a single rule.
Revised: When k = 1 , this corresponds to generating a single prompt using a single rule.

Original: In our case, it was r = 7 , which corresponds to taking 0.75 fraction of post lines from the current file.
Revised: In our case, it was r = 7 , which corresponds to taking 0.75 fraction of post lines from the current file.

Original: RLPG-H ( k ) and RLPG-R ( k ) :
Revised: RLPG-H ( k ) and RLPG-R ( k ) :

Original: Using k prompts generated by top- k rules.
Revised: Using k prompts generated by top- k rules.

Original: The top- k rules correspond to the top- k rule predictions given by the RLPG-H and RLPG-R variants of our rule classifier.
Revised: The top- k rules correspond to the top- k rule predictions given by the RLPG-H and RLPG-R variants of our rule classifier

Original: 3.1. Results
Revised: 3.1. Results

Original: Performance of Proposed Rules: The left part of Tableshows the performance of an oracle constructed from our proposed rules.
Revised: Performance of Proposed Rules: The left part of Tableshows the performance of an oracle constructed from our proposed rules.

Original: We see that across all data splits, the proposed rules contribute to large relative improvements when compared to Codex.
Revised: We see that across all data splits, the proposed rules contribute to large relative improvements when compared to Codex.

Original: This result was pleasantly surprising to us as Codex has not been trained on prompts that consist of context other than the default Codex context.
Revised: This result was pleasantly surprising to us as Codex has not been trained on prompts that consist of context other than the default Codex context.

Original: These results display that incorporating repo-level knowledge in the form of rules can be quite effective strategy for generating useful prompts.
Revised: These results display that incorporating repo-level knowledge in the form of rules can be quite effective strategy for generating useful prompts.

Original: Performance of the Rule Classifier: Having seen promise in our proposed rules, next we present the results of RLPG, which for each target hole, predicts a single best rule that can lead to success.
Revised: Performance of the Rule Classifier: Having seen promise in our proposed rules, next we present the results of RLPG, which for each target hole, predicts a single best rule that can lead to success.

Original: The right part of Table 2 presents the success rates along with the percentage of relative improvements for the test data.
Revised: The right part of Table 2 presents the success rates along with the percentage of relative improvements for the test data.

Original: The second and third columns correspond to the averages across all holes in the test data.
Revised: The second and third columns correspond to the averages across all holes in the test data.

Original: The last two columns correspond to the average success rate of individual repositories.
Revised: The last two columns correspond to the average success rate of individual repositories.

Original: The latter metric doesn’t account for the size of the repository.
Revised: The latter metric doesn’t account for the size of the repository.

Original: As can be seen from the table, both the RLPG variants as well as fixed rule improve the performance significantly over Codex, again highlighting the value of the proposed rules.
Revised: As can be seen from the table, both the RLPG variants as well as fixed rule improve the performance significantly over Codex, again highlighting the value of the proposed rules.

Original: In addition, both the RLPG variants show improved performance when compared to fixed rules highlighting the importance of learning a rule classifier.
Revised: In addition, both the RLPG variants show improved performance when compared to fixed rules highlighting the importance of learning a rule classifier.

Original: See Appendix B.3 for performance of all methods on individual repositories and Appendix B.1, Appendix B. and Appendix B.4 for experiments with scaling behaviour, rule-wise performance and analysis of sample cases.
Revised: See Appendix B.3 for performance of all methods on individual repositories and Appendix B.1, Appendix B. and Appendix B.4 for experiments with scaling behaviour, rule-wise performance and analysis of sample cases.

Original: 4. Related Work
Revised: 4. Related Work

Original: There are two broad categories of prompt generation techniques.
Revised: There are two broad categories of prompt generation techniques.

Original: The first category corresponds to producing continuous/soft prompts where the prompt is described in the latent space of a language model (Li & Liang, 2021;
Revised: The first category corresponds to producing continuous/soft prompts where the prompt is described in the latent space of a language model (Li & Liang, 2021;

Original: Qin & Eisner, 2021; Bragg et al., 2021; Lester et al., 2021;
Revised: Qin & Eisner, 2021; Bragg et al., 2021; Lester et al., 2021;

Original: Liu et al., 2021b).
Revised: Liu et al., 2021b).

Original: The second category produces discrete prompts where the prompt is a text string that can be interpreted by a human (Shin et al., 2020; Gao et al., 2021; Schick &
Revised: The second category produces discrete prompts where the prompt is a text string that can be interpreted by a human (Shin et al., 2020; Gao et al., 2021; Schick &

Original: Schütze, 2021).
Revised: Schütze, 2021).

Original: Our work falls in the category of discrete prompt generation techniques as we produce a prompt consisting of code tokens that can be easily interpreted by a human.
Revised: Our work falls in the category of discrete prompt generation techniques as we produce a prompt consisting of code tokens that can be easily interpreted by a human.

Original: However, in contrast to prior works that use a set of fixed templates for all examples, we learn to generate prompts conditioned on each example.
Revised: However, in contrast to prior works that use a set of fixed templates for all examples, we learn to generate prompts conditioned on each example.

Original: Another important distinction is that we do not require access to the weights of the LM.
Revised: Another important distinction is that we do not require access to the weights of the LM.

Original: To the best of our knowledge, our work is the first to explore automatic prompt generation in the domain of source code.
Revised: To the best of our knowledge, our work is the first to explore automatic prompt generation in the domain of source code.

Original: We provide additional related works in
Revised: We provide additional related works in

Original: Appendix G.
Revised: Appendix G.

Original: 5. Conclusions and Future Directions
Revised: 5. Conclusions and Future Directions

Original: We present RLPG, a framework that learns to automatically generate prompts conditioned on the example without requiring access to the weights of the LLM.
Revised: We present RLPG, a framework that learns to automatically generate prompts conditioned on the example without requiring access to the weights of the LLM.

Original: RLPG utilizes the structure of the repository as well as the context from other files in the repository using a set of easy to understand rules.
Revised: RLPG utilizes the structure of the repository as well as the context from other files in the repository using a set of easy to understand rules.

Original: For future work, we want to explore the composition of contexts coming from two or more rules to generate a prompt and incorporating user’s feedback in RLPG. 
Revised: For future work, we want to explore the composition of contexts coming from two or more rules to generate a prompt and incorporating user’s feedback in RLPG. Acknowledgement

Revised: Hugo Larochelle would like to acknowledge the support of

Revised: Canada Research Chairs and CIFAR for research funding.

Revised: The authors would like to thank Google Cloud for providing compute resources required for this project.

Original:  A. Repo-Level Rules A.1. Rule Context Location
Revised: We would also like to extend our thanks to Breandan Considine for help in crawling the Google Code data archives; Justine Gehring, Avinash Bhat and Breandan Considine for helping with resources for running experiments; and David Bieber for feedback and comments on the draft that helped us improve writing. Finally, we would like to acknowledge OpenAI for providing access to the Codex API. A. Repo-Level Rules A.1. Rule Context Location

Original: Current: take code from the current file.
Revised: Current: take code from the current file.

Original: The current file is the file that contains the target hole.
Revised: The current file is the file that contains the target hole.

Original: The code in the current file (e.g. the lines after the hole position) can be very useful in predicting the target hole.
Revised: The code in the current file (e.g. the lines after the hole position) can be very useful in predicting the target hole.

Original: Parent Class: take code from the file that contains the parent class of the target hole.
Revised: Parent Class: take code from the file that contains the parent class of the target hole.

Original: The intuition behind this is to account for cases where a method present in the parent class is invoked in the current file (i
Revised: The intuition behind this is to account for cases where a method present in the parent class is invoked in the current file (i

Original: e. the child class).
Revised: e. the child class).

Original: Import: take code from the import files used in the current file.
Revised: Import: take code from the import files used in the current file.

Original: The intuition here is same as parent class.
Revised: The intuition here is same as parent class.

Original: Sibling: take code from the files that are in the same directory as the current file.
Revised: Sibling: take code from the files that are in the same directory as the current file.

Original: Files in the same directory tend to share code variables (e.g. identifiers).
Revised: Files in the same directory tend to share code variables (e.g. identifiers).

Original: Similar Name: take code from files that have a similar name as the current file.
Revised: Similar Name take code from files that have a similar name as the current file.

Original: Similar names are determined by doing a splitting of the file name based on underscore or camel-case formatting and then matching parts of the filename.
Revised: Similar names are determined by doing a splitting of the file name based on underscore or camel-case formatting and then matching parts of the filename.

Original: If one or more parts matches, two files are considered to have similar names.
Revised: If one or more parts matches, two files are considered to have similar names.

Original: The intuition behind this is that developers tend to name files based on the functionality of the code written in that file.
Revised: The intuition behind this is that developers tend to name files based on the functionality of the code written in that file.

Original: Therefore, a similar name file might contain some portion of the code that is common with the current file and hence might be useful for predicting the target hole.
Revised: Therefore, a similar name file might contain some portion of the code that is common with the current file and hence might be useful for predicting the target hole

Original: Child Class: take code from files that have the current file as their parent class file.
Revised: Child Class: take code from files that have the current file as their parent class file.

Original: Import of Parent Class: take code from the import files used in the parent class files.
Revised: Import of Parent Class: take code from the import files used in the parent class files.

Original: Import of Sibling: take code from the import files used in the sibling files.
Revised: Import of Sibling: take code from the import files used in the sibling files.

Original: Import of Similar Name: take code from the import files used in the similar name files.
Revised: Import of Similar Name: take code from the import files used in the similar name files.

Original: Import of Child Class: take code from the import files used in the child class files.
Revised: Import of Child Class: take code from the import files used in the child class files.

Original: The last four rule context locations are useful when the target hole occurs at the very beginning of the current file where there might be less rule context coming from the other rule context locations.
Revised: The last four rule context locations are useful when the target hole occurs at the very beginning of the current file where there might be less rule context coming from the other rule context locations.

Original: A.2. Rule Context Type
Revised: A.2. Rule Context Type

Original: Post Lines (PL): Take all the lines after the target hole line.
Revised: Post Lines (PL): Take all the lines after the target hole line.

Original: This context type is applicable only when the rule context location is the current file.
Revised: This context type is applicable only when the rule context location is the current file.

Original: For the example shown in Figure 1 of the main paper, post lines will take all the lines after the line mg.InitializeToAssignment(CurrentAssignments()) till we reach the end of the file ( AffinityPropagation.java ).
Revised: For the example shown in Figure 1 of the main paper, post lines will take all the lines after the line mg.InitializeToAssignment(CurrentAssignments()) till we reach the end of the file ( AffinityPropagation.java ).

Original: Identifiers (I): Take all the identifiers used in the rule context location.
Revised: Identifiers (I): Take all the identifiers used in the rule context location.

Original: Identifiers are the names of variables used in the code.
Revised: Identifiers are the names of variables used in the code.

Original: For example, for the rule context taken from the imported file shown in Figure 1 in the main paper (highlighted in violet), identifiers are InitializeToAssignment (line 1), a (line 1), currentAssignment_ (line 2), a ( line 2), clone (line 2), alreadyInitialized_ (line 3), justOneRound_ (line 4).
Revised: For example, for the rule context taken from the imported file shown in Figure 1 in the main paper (highlighted in violet), identifiers are InitializeToAssignment (line 1), a (line 1), currentAssignment_ (line 2), a ( line 2), clone (line 2), alreadyInitialized_ (line 3), justOneRound_ (line 4).

Original: Type Identifiers (TI): Take all the type identifiers used in the rule context location.
Revised: Type Identifiers (TI): Take all the type identifiers used in the rule context location.

Original: Type Identifiers define the type of an identifier.
Revised: Type Identifiers define the type of an identifier.

Original: For example, in the code snippet class DPAffinityPropagation extends AffinityPropagation , [ AffinityPropagation is labeled as a type identifier.
Revised: For example, in the code snippet class DPAffinityPropagation extends AffinityPropagation , [ AffinityPropagation is labeled as a type identifier.

Original: Similarly in the snippet DPAPParameters parameters_; , DPAPParameters is a type identifier.
Revised: Similarly in the snippet DPAPParameters parameters_; , DPAPParameters is a type identifier.

Original: Field Declarations (FD): Take all the field declarations used in the rule context location.
Revised: Field Declarations (FD): Take all the field declarations used in the rule context location.

Original: The variables of a class type are introduced by field declarations.
Revised: The variables of a class type are introduced by field declarations.

Original: For example, double[][] mHijMujT_; and MessageValuePair[][] sortedMHijMujTs_; are examples of field declarations.
Revised: For example, double[][] mHijMujT_; and MessageValuePair[][] sortedMHijMujTs_; are examples of field declarations.

Original: String Literals (SL): Take all the string literals used in the rule context location.
Revised: String Literals (SL): Take all the string literals used in the rule context location.

Original: A string literal is the sequence of characters enclosed in double-quotes.
Revised: A string literal is the sequence of characters enclosed in double-quotes.

Original: For example, in the code snippet, System.err.println("DPAP load Warning: unknown parameter " + entries[0] + ", value = " + entries[1]); , we have two string literals: (a) "DPAP load Warning: unknown parameter " ; (b) ", value = " .
Revised: For example, in the code snippet, System.err.println("DPAP load Warning: unknown parameter " + entries[0] + ", value = " + entries[1]); , we have two string literals: (a) "DPAP load Warning: unknown parameter " ; (b) ", value = " .

Original: Method Names (MN): Take all the method names along with their signature used in the rule context location.
Revised: Method Names (MN): Take all the method names along with their signature used in the rule context location.

Original: For the example shown in Figure 1 of the main paper, public void InitializeToAssignment(int[] a) is the method name rule context type.
Revised: For the example shown in Figure 1 of the main paper, public void InitializeToAssignment(int[] a) is the method name rule context type.

Original: Method Names and Bodies (MNB): Take all the method names along with their signatures and corresponding bodies used in the rule context location.
Revised: Method Names and Bodies (MNB): Take all the method names along with their signatures and corresponding bodies used in the rule context location.

Original: For the example shown in Figure 1 of the main paper, the part highlighted in violet represents the method names and bodies.[Table]
Revised: For the example shown in Figure 1 of the main paper, the part highlighted in violet represents the method names and bodies.

Original: A.3. List of Repo-Level Rules
Revised: A.3. List of Repo-Level Rules

Original: [Table]
Revised: [Table]

Original: .4. Ranking of files based on rule context location
Revised: A.4. Ranking of files based on rule context location

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we provide details of the heuristics we used for selecting files for a given rule context location.
Revised: we provide details of the heuristics we used for selecting files for a given rule context location.

Original: Depending on the rule, we get either a single file or a list of files ranked based on some criteria.
Revised: Depending on the rule [Table] we get either a single file or a list of files ranked based on some criteria.

Original: For example, if the rule context location is Import, we take all the import statements used in the current file and identify the location in the current file where the corresponding imports have been used.
Revised: For example, if the rule context location is Import, we take all the import statements used in the current file and identify the location in the current file where the corresponding imports have been used.

Original: According to our heuristic, the closer is the import usage to the hole position, the more likely it is for the rule context coming from the corresponding import file to be more relevant (to predict the target hole).
Revised: According to our heuristic, the closer is the import usage to the hole position, the more likely it is for the rule context coming from the corresponding import file to be more relevant (to predict the target hole).

Original: We get a ranked list of import files sorted based on increasing order of distance (i
Revised: We get a ranked list of import files sorted based on increasing order of distance (i

Original: [Table]
Revised: [Table]

Original: e., number of lines ) between the import usage and the hole position.
Revised: e., number of lines ) between the import usage and the hole position.

Original: We start by taking all of the rule context from the first file in the ranked list and then keep iterating the ranked list until either the total context length allocated to the rule gets exhausted or we reach the end of the ranked list.
Revised: We start by taking all of the rule context from the first file in the ranked list and then keep iterating the ranked list until either the total context length allocated to the rule gets exhausted or we reach the end of the ranked list.

Original: [Table]
Revised: [Table]

Original: A.5. Truncation Strategies for Rule Context
Revised: A.5. Truncation Strategies for Rule Context

Original: If the rule context is greater than the context length allocated to it, i.e., rule context fraction multiplied by total context length, then we need to truncate the rule context.
Revised: If the rule context is greater than the context length allocated to it, i.e., rule context fraction multiplied by total context length, then we need to truncate the rule context.

Original: We followed the below two schemes for truncating context:
Revised: We followed the below two schemes for truncating context:

Original: • front: We truncate the context from the front.
Revised: • front: We truncate the context from the front.

Original: This is used for all rule context locations except Parent Class and when we take PL from Current.
Revised: This is used for all rule context locations except Parent Class and when we take PL from Current.

Original: • back: We truncate the context from the back.
Revised: • back: We truncate the context from the back.

Original: This is used when the rule context location is Parent Class and when we take rule context types other than PL from Current.
Revised: This is used when the rule context location is Parent Class and when we take rule context types other than PL from Current.

Original: The truncation strategies for each case were selected based on results on a small validation set.
Revised: The truncation strategies for each case were selected based on results on a small validation set.

Original: For Current context location, except when the rule context type is PL, we always start by taking code of rule context type from after the hole position.
Revised: For Current context location, except when the rule context type is PL, we always start by taking code of rule context type from after the hole position.

Original: This makes sense as the default Codex context will anyways contain code before the hole.
Revised: This makes sense as the default Codex context will anyways contain code before the hole.

Original: Only if this turns out to be blank, we will use the code of context type from before 
Revised: Only if this turns out to be blank, we will use the code of context type from before the hole.

Original: the hole.

Original: A.6. Other Rule Variations
Revised: A.6. Other Rule Variations

Original: We experimented with other variations that include: (a) appending class names at the beginning of the rule context, (b) using newline or space to join the rule context and the default Codex context, (c) taking all or top-k of the rule context types, (d) ordering of top-k. See Appendix for more details on these.
Revised: We experimented with other variations that include: (a) appending class names at the beginning of the rule context, (b) using newline or space to join the rule context and the default Codex context, (c) taking all or top-k of the rule context types, (d) ordering of top-k. See Appendix for more details on these.

Original: Table 1 lists our proposed rules.
Revised: Table 1 lists our proposed rules.

Original: In Figure 1, rule r = 14 is predicted and corresponds to taking rule context location of Import, Rule Context Type of MNB and Rule Context Fraction of 0.5.
Revised: In Figure 1, rule r = 14 is predicted and corresponds to taking rule context location of Import, Rule Context Type of MNB and Rule Context Fraction of 0.5.

Original: • Context Separator: This defines how we join the rule context string to the default Codex context string.
Revised: • Context Separator: This defines how we join the rule context string to the default Codex context string.

Original: We experimented with space and newline 
Revised: We experimented with space and newline as context separators.

Original: as context separators • Rule Context Formatting: We can format the rule context before giving it to the prompt generator.
Revised: • Rule Context Formatting: We can format the rule context before giving it to the prompt generator.

Original: We experimented with 
Revised: We experimented with the

Original: the following options
Revised: following options:

Original: class_name: append [class name of the file] at the beginning of the rule context taken from each file that is part of the rule context location.
Revised: class_name: append [class name of the file] at the beginning of the rule context taken from each file that is part of the rule context location.

Original: For example, if we are taking rule context from two import files f 1 and f 2 , the rule context will be formatted as: [class name of f 1 ] rule context from f 1 + space + [class name of f 2 ] rule context from f 2 .
Revised: For example, if we are taking rule context from two import files f 1 and f 2 , the rule context will be formatted as: [class name of f 1 ] rule context from f 1 + space + [class name of f 2 ] rule context from f 2 .

Original: We use this when the rule context types are MN, I, TI, FD and SL. 2.
Revised: We use this when the rule context types are MN, I, TI, FD and SL. 2.

Original: class_method_name: we apply this only when the rule context type is MNB.
Revised: class_method_name: we apply this only when the rule context type is MNB.

Original: We append method names at the beginning of each of the corresponding method bodies.
Revised: We append method names at the beginning of each of the corresponding method bodies.

Original: We also append the rule context from a file with the name of the class as described in the previous item.
Revised: We also append the rule context from a file with the name of the class as described in the previous item.

Original: 3. comment: Adding in the rule context as a comment, i.e., formatting it as: /** rule context */.
Revised: 3. comment: Adding in the rule context as a comment, i e., formatting it as: /** rule context */.

Original: This wasn’t found to be much useful.
Revised: This wasn’t found to be much useful.

Original: none: passing the rule context as it is.
Revised: none: passing the rule context as it is.

Original: We use this when the rule context type is PL.
Revised: We use this when the rule context type is PL.

Original: • Top-k Type:
Revised: • Top-k Type:

Original: For each of the rule context types, except PL, we experimented with taking the (a) first (b) last and (c) all of the rule context types, i.e., we can take first-10 identifiers.
Revised: For each of the rule context types, except PL, we experimented with taking the (a) first (b) last and (c) all of the rule context types, i e., we can take first-10 identifiers.

Original: We found ’all’ to be the best among all.
Revised: We found ’all’ to be the best among all.

Original: • Top-k: We experiment with k values of (a) 10 (b) 20 and (c) all.
Revised: • Top-k: We experiment with k values of (a) 10 (b) 20 and (c) all.

Original: We found ’all’ to work best for all rule context types.
Revised: We found ’all’ to work best for all rule context types.

Original: B. Additional Results B.1. Variation with k
Revised: B. Additional Results B.1. Variation with k

Original: Figure 3 shows the variation of the success rate over the validation data with the value of k .
Revised: Figure 3 shows the variation of the success rate over the validation data with the value of k .

Original: The purpose of this study is to see how does the performance scale with the number of prompts (or rules).
Revised: The purpose of this study is to see how does the performance scale with the number of prompts (or rules).

Original: One of the potential real-world usage scenario of this study can be when there is a human-in-the-loop confirming that the hole-prediction is good or not, with the assumption that then human is allowed to use multiple prompts for getting prediction for the same target hole.
Revised: One of the potential real-world usage scenario of this study can be when there is a human-in-the-loop confirming that the hole-prediction is good or not, with the assumption that then human is allowed to use multiple prompts for getting prediction for the same target hole.

Original: For these experiments, if any of the k selected rules lead to success, success ratio will be set to one.
Revised: For these experiments, if any of the k selected rules lead to success, success ratio will be set to one.

Original: From the figure, we notice that as we increase the value of k , the performance increases gradually at first and then saturates towards the oracle performance (79.05% for val data).
Revised: From the figure, we notice that as we increase the value of k , the performance increases gradually at first and then saturates towards the oracle performance (79.05% for val data).

Original: This behaviour is observed for both Fixed Rule as well as RLPG.
Revised: This behaviour is observed for both Fixed Rule as well as RLPG.

Original: However, we see that for the same value of k , the success rate for RLPG is higher both when taking the repo-wise as well as the hole-wise averages.
Revised: However, we see that for the same value of k , the success rate for RLPG is higher both when taking the repo-wise as well as the hole-wise averages.

Original: This indicates that sampling k rules using RLPG is better than taking top- k most frequent rules, emphasizing the contribution of our rule classifier.
Revised: This indicates that sampling k rules using RLPG is better than taking top- k most frequent rules, emphasizing the contribution of our rule classifier.

Original: Moreover, this gap in performance scales with increasing the value of k .
Revised: Moreover, this gap in performance scales with increasing the value of k .

Original: This offers an interesting analogy to some recent works (Li et al., 2022; Chen et al., 2021) that have shown such scaling behaviour but for samples taken from the output of the LLM as opposed to more samples (i.e. prompts) at the input of the LLM.
Revised: This offers an interesting analogy to some recent works (Li et al., 2022; Chen et al., 2021) that have shown such scaling behaviour but for samples taken from the output of the LLM as opposed to more samples (i.e. prompts) at the input of the LLM.

Original: 9 10 11 12 13 14 15 16 17 18 19k
Revised: 9 10 11 12 13 14 15 16 17 18 19k

Original: V a l
Revised: V a l

Original: S u cc e ss
Revised: S u cc e ss

Original: R a t e ( % )
Revised: R a t e ( % )

Original: Fixed Rule (repo-wise)
Revised: Fixed Rule (repo-wise)

Original: RLPG-R (repo-wise)
Revised: RLPG-R (repo-wise)

Original: Fixed Rule (hole-wise)
Revised: Fixed Rule (hole-wise)

Original: RLPG-R (hole-wise)
Revised: RLPG-R (hole-wise)

Revised: B.2. Rule-Wise Performance

Original: Figure 4 shows the plot of the top-20 successful rules.
Revised: Figure 4 shows the plot of the top-20 successful rules.

Original: The orange bar indicates the fraction of times (in percentage) when a particular rule was applicable and the blue bar indicates the fraction of times when that rule was successful.
Revised: The orange bar indicates the fraction of times (in percentage) when a particular rule was applicable and the blue bar indicates the fraction of times when that rule was successful.

Original: We can see that the first four rules (i.e. 5, 7, 6, 1) are using context coming from the same file.
Revised: We can see that the first four rules (i.e. 5, 7, 6, 1) are using context coming from the same file.

Original: In particular, using PL can be quite useful.
Revised: In particular, using PL can be quite useful.

Original: The next line of successful rules correspond to the sibling files.
Revised: The next line of successful rules correspond to the sibling files.

Original: Note that even though these rules have a overall low success rate as compared to the current file rules, it is more likely for them to be successful whenever they are applicable.
Revised: Note that even though these rules have a overall low success rate as compared to the current file rules, it is more likely for them to be successful whenever they are applicable.

Original: This is in contrast to r = 1 and r = 62 (Codex) where the rule being applicable doesn’t always lead to success.
Revised: This is in contrast to r = 1 and r = 62 (Codex) where the rule being applicable doesn’t always lead to success.

Original: Note that it is possible to extend our framework by adding more rules and then retraining the rule classifier.
Revised: Note that it is possible to extend our framework by adding more rules and then retraining the rule classifier.

Original: 1 20 220 25 3 23 24 28 26 4 21 62 31 27 29     successful applicable B.3. Performance on individual repositories Repo name #Total Holes
Revised: [Table]

Original: Oracle Codex Fixed Rule
Revised: successful applicable

Original: [Table]
Revised: [Table]

Original: RLPG
Revised: openprocesslogger

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: RLPG
Revised: tapestry

Original: [Table]
Revised: [Table]

Original: largemail 75.
Revised: sesame

Original: ftpserverremoteadmin 86. myt5lib 91. seamlets 92. gloodb 91.
Revised: [Table]

Original: jjskit 80.
Revised: [Table]

Original: mobileexpensetracker 75. gfsfa 80. swe574-group  76.
Revised: [Table]

Original: strudem-sicsa 77. soap-dtc 81. openprocesslogger 81. tapestry-sesame 72.
Revised: [Table]

Original: exogdx 84.
Revised: exogdx 84.

Original: designpatternjavapedro 78.
Revised: designpatternjavapedro 78.

Original: quidsee 81.
Revised: quidsee 81.

Original: healpix-rangeset 63.
Revised: healpix-rangeset 63.

Original: sol-agent-platform 73.
Revised: sol-agent-platform 73.

Original: rsbotownversion 75.
Revised: rsbotownversion 75.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: R (k=1) dovetaildb 76. project-pt-diaoc 82. realtimegc 77. fswuniceubtemplates 77.

Original: [Table] [Table] [Table]
Revised: qwikioffice-java 76. glperaudsimon 78.

Original: [Table]
Revised: xiaonei-java-api 73. ircrpgbot 83. robotsimulator2009w 75. gwt-plugindetect 84.

Original: [Table] [Table]
Revised: apiitfriends 85. wicketbits 83.

Original: [Table]
Revised: hucourses 84.

Original: [Table]
Revised: xfuze 84.

Original: Table 3, Table 4 and Table 5 present the success rates of different methods over individual repositories in the training, validation and test splits, respectively.
Revised: Table 3, Table 4 and Table 5 present the success rates of different methods over individual repositories in the training, validation and test splits, respectively.

Original: The repo-wise averages in Table 2 in the main paper were calculated by taking the average of numbers corresponding to each column.
Revised: The repo-wise averages in Table 2 in the main paper were calculated by taking the average of numbers corresponding to each column.

Original: The hole-wise averages correspond to multiplying the repo-wise numbers of each method by the total holes in the repo to get the total number of successful holes by that method for that repo.
Revised: The hole-wise averages correspond to multiplying the repo-wise numbers of each method by the total holes in the repo to get the total number of successful holes by that method for that repo.

Original: We then add the total number of successful holes across repos and divide it by the total number of holes in the entire data split to get the hole-wise averages.
Revised: We then add the total number of successful holes across repos and divide it by the total number of holes in the entire data split to get the hole-wise averages.

Original: B.4. Analysis of Sample Cases B.4.1. P OSITIVE C ASES
Revised: B.4. Analysis of Sample Cases B.4.1. P OSITIVE C ASES

Original: In Figure 1, RLPG predicts the rule that corresponds to taking method names and bodies from the imported file (i.e.
Revised: In Figure 1, RLPG predicts the rule that corresponds to taking method names and bodies from the imported file (i.e.

Original: MaximizingGibbsSampler.java ).
Revised: MaximizingGibbsSampler.java ).

Original: Note that mg.
Revised: Note that mg.

Original: before the hole position indicates that a method used in the imported file is likely to be invoked.
Revised: before the hole position indicates that a method used in the imported file is likely to be invoked.

Original: In this case, the rule context (highlighted in violet) contains the method name
Revised: In this case, the rule context (highlighted in violet) contains the method name

Original: InitializeToAssignment (part of target hole).
Revised: InitializeToAssignment (part of target hole).

Original: This in conjunction with the default Codex context which contains the method CurrentAssignments() (part of target hole) leads to generation of a successful prompt.
Revised: This in conjunction with the default Codex context which contains the method CurrentAssignments() (part of target hole) leads to generation of a successful prompt.

Original: On the other hand, the prompt created from the default Codex context fails to predict the target hole in this case.
Revised: On the other hand, the prompt created from the default Codex context fails to predict the target hole in this case.

Original: We provide some examples of cases where RLPG led to the correct prediction and Codex failed.
Revised: We provide some examples of cases where RLPG led to the correct prediction and Codex failed.

Original: Cases where part of the target hole is found exactly in the rule context.
Revised: Cases where part of the target hole is found exactly in the rule context.

Original: • RLPG = Propagation(int numVars) vs Codex = Propagation() • RLPG 
Revised: • RLPG = Propagation(int numVars) vs Codex = Propagation() • RLPG =

Original: = tersFromFile(String filename) { vs Codex = ters(String filename) { • RLPG = als("dampingFactor"))
Revised: tersFromFile(String filename) { vs Codex = ters(String filename) { • RLPG = als("dampingFactor"))

Original: { vs Codex = als("numVars")) { • RLPG = ] + ", value = " + entries[1]); vs Codex = ]); • RLPG = stem.exit(1); vs Codex = stem.err.println("DPAP load error: " + ex.get 2.
Revised: { vs Codex = als("numVars")) { • RLPG = ] + ", value = " + entries[1]); vs Codex = ]); • RLPG = stem.exit(1); vs Codex = stem.err.println("DPAP load error: " + ex.get 2.

Original: Cases where Codex takes strong hint from the preceding natural language comment, e.g. naming the method based on the comment.
Revised: Cases where Codex takes strong hint from the preceding natural language comment, e g. naming the method based on the comment.

Original: thereby producing incorrect predictions.
Revised: thereby producing incorrect predictions.

Original: • RLPG =
Revised: • RLPG =

Original: d PassMessages() vs Codex =
Revised: d PassMessages() vs Codex =

Original: d DoOneRoundOfMessagePassing() • RLPG = teger> CurrentExemplars() { vs Codex = teger> ChooseExemplars() { • RLPG = ring FileName() { vs Codex = ring GetAlgorithmFilename() {
Revised: d DoOneRoundOfMessagePassing() • RLPG = teger> CurrentExemplars() { vs Codex = teger> ChooseExemplars() { • RLPG = ring FileName() { vs Codex = ring GetAlgorithmFilename() {

Original: B.4.2. N EGATIVE C ASES
Revised: B.4.2. N EGATIVE C ASES

Original: In certain cases, extra information from rule-context might lead to confusion and produce incorrect predictions.
Revised: In certain cases, extra information from rule-context might lead to confusion and produce incorrect predictions.

Original: • RLPG = an hasConverged_; vs Codex = an converged_; • RLPG =
Revised: • RLPG = an hasConverged_; vs Codex = an converged_; • RLPG =

Original: _[i][j] = -Double.
Revised: _[i][j] = -Double.

Original: MAX_VALUE; vs Codex = _[i][j] = 0;
Revised: MAX_VALUE; vs Codex = _[i][j] = 0;

Original: C. Details of Dataset Creation C.1. Creation of Hole Completion Data
Revised: C. Details of Dataset Creation C.1. Creation of Hole Completion Data

Original: To mitigate the effects caused by potential memorization of the code present in the dataset used for training Codex, we tried to use repositories that Codex might not have been trained on and avoided code repositories from GitHub (Chen et al., 2021)).
Revised: To mitigate the effects caused by potential memorization of the code present in the dataset used for training Codex, we tried to use repositories that Codex might not have been trained on and avoided code repositories from GitHub (Chen et al., 2021)).

Original: Instead, we scraped Google Code 4 for repositories tagged with the language “Java”.
Revised: Instead, we scraped Google Code 4 for repositories tagged with the language “Java”.

Original: Then we deduplicated repositories by searching for a matching repository with the same name on GitHub.
Revised: Then we deduplicated repositories by searching for a matching repository with the same name on GitHub.

Original: For those repositories with zero matching names on GitHub, we downloaded the archive and extracted the source code (preserving the directory structure).
Revised: For those repositories with zero matching names on GitHub, we downloaded the archive and extracted the source code (preserving the directory structure).

Original: Next, we tried to determine the licenses of all repositories by either looking for a LICENSE file or matching with keywords "license", "copyright", "mit", etc.
Revised: Next, we tried to determine the licenses of all repositories by either looking for a LICENSE file or matching with keywords "license", "copyright", "mit", etc.

Original: For repos for which our process was able to come up with a known license, we selected the ones having a permissive license, i.e., MIT, ApacheV2 and BSD.
Revised: For repos for which our process was able to come up with a known license, we selected the ones having a permissive license, i.e., MIT, ApacheV2 and BSD.

Original: Since code duplication has been shown to have adverse effects (Allamanis, 2018), we removed files that are exact duplicates of each other within a repo.
Revised: Since code duplication has been shown to have adverse effects (Allamanis, 2018), we removed files that are exact duplicates of each other within a repo.

Original: One of the reasons we found this inter-repository duplication may be because sometimes developers adopt lousy practises where instead of declaring a package and importing functions, they simply copy-paste the desired file in the current folder.
Revised: One of the reasons we found this inter-repository duplication may be because sometimes developers adopt lousy practises where instead of declaring a package and importing functions, they simply copy-paste the desired file in the current folder.

Original: The target holes coming from any of the duplicate files do not form part of the hole completion dataset.
Revised: The target holes coming from any of the duplicate files do not form part of the hole completion dataset.

Original: However, these files might be used to contribute to rule context for completing a target hole in a non-duplicate file (e.g. in a sibling file).
Revised: However, these files might be used to contribute to rule context for completing a target hole in a non-duplicate file (e.g. in a sibling file).

Original: For the remaining files, we took each line that is not a blanked line or a comment, and chose the middle character as the hole position, i.e., all the characters from the middle of the line to the end of the line form target hole.
Revised: For the remaining files, we took each line that is not a blanked line or a comment, and chose the middle character as the hole position, i.e., all the characters from the middle of the line to the end of the line form target hole.

Original: To avoid large repos having strong bias on our rule classifier, we capped the contribution from each repo to be a maximum of 10000 holes.
Revised: To avoid large repos having strong bias on our rule classifier, we capped the contribution from each repo to be a maximum of 10000 holes.

Original: If the number of holes in the repo exceeds 10000, we randomly select 10000 holes.
Revised: If the number of holes in the repo exceeds 10000, we randomly select 10000 holes.

Original: Please see Table 6 for statistics of our dataset.
Revised: Please see Table 6 for statistics of our dataset.

Original: The #Holes represent the holes after deduplication and capping.
Revised: The #Holes represent the holes after deduplication and capping.

Original: C.2. Creation of Data for Repo-Level Rules
Revised: C.2. Creation of Data for Repo-Level Rules

Original: We used the tree-sitter API for Java 5 to get the parse-tree of an individual file in a repo.
Revised: We used the tree-sitter API for Java 5 to get the parse-tree of an individual file in a repo.

Original: To get information at a repo-level, for each file in the repo, we stored the following information:
Revised: To get information at a repo-level, for each file in the repo, we stored the following information:

Original: Feature

Original: [Table]
Revised: [Table]

Original: 1. list of all class names in the file.
Revised: 1. list of all class names in the file.

Original: This helped us to get the parent or child class file corresponding to a given parent or child class.
Revised: This helped us to get the parent or child class file corresponding to a given parent or child class.

Original: the file corresponding to each import statement.
Revised: the file corresponding to each import statement.

Original: for each import statement in the file, the position in the file where the import is used.
Revised: for each import statement in the file, the position in the file where the import is used.

Original: This is used for ranking the files based on the heuristics mentioned in Table 2. 4.
Revised: This is used for ranking the files based on the heuristics mentioned in Table 2. 4.

Original: list of sibling files 5.
Revised: list of sibling files 5.

Original: list of similar name files.
Revised: list of similar name files.

Original: This was done by splitting the filenames based on either camel-case or underscore.
Revised: This was done by splitting the filenames based on either camel-case or underscore.

Original: If the sub-parts of two files match, then they are said to have similar name.
Revised: If the sub-parts of two files match, then they are said to have similar name.

Original: The above meta-data was calculated only once for each repo.
Revised: The above meta-data was calculated only once for each repo.

Original: The subsequent hole completions can use the same cached information.
Revised: The subsequent hole completions can use the same cached information.

Original: In practise, we can use a hash to store and retrieve this info efficiently.
Revised: In practise, we can use a hash to store and retrieve this info efficiently.

Original: For a rule, given the rule context location, we first obtain a single file or ranked list of files (see Table 2) using the info in the parse tree in conjugation with the above repo-level meta-data.
Revised: For a rule, given the rule context location, we first obtain a single file or ranked list of files (see Table 2) using the info in the parse tree in conjugation with the above repo-level meta-data.

Original: All the rule context type information (MN, MNB, SL, I, TI, FD) can then be obtained by querying the parse tree of the selected file.
Revised: All the rule context type information (MN, MNB, SL, I, TI, FD) can then be obtained by querying the parse tree of the selected file.

Original: D. Details of the Rule Classifier D.1. RLPG-H
Revised: D. Details of the Rule Classifier D.1. RLPG-H

Original: Equation 2 shows the prediction of the rule label conditioned on the hole window representation.
Revised: Equation 2 shows the prediction of the rule label conditioned on the hole window representation.

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: We used Adam (Kingma & Ba, 2015) optimizer with a learning rate of 3e-4 and batch size of 64.
Revised: We used Adam (Kingma & Ba, 2015) optimizer with a learning rate of 3e-4 and batch size of 64.

Original: We used CodeBERT (Feng et al., 2020) as our pretrained model F ϕ to obtain the representation of hole window.
Revised: We used CodeBERT (Feng et al., 2020) as our pretrained model F ϕ to obtain the representation of hole window.

Original: The size of the representation (corresponding to the hidden dimension of the [CLS] token) is 768.
Revised: The size of the representation (corresponding to the hidden dimension of the [CLS] token) is 768.

Original: W 1 ∈ R 512 × 768 , b 1 = 512 , W 2 ∈ R 63 × 512 , b 2 = 63 .
Revised: W 1 ∈ R 512 × 768 , b 1 = 512 , W 2 ∈ R 63 × 512 , b 2 = 63 .

Original: D.2. RLPG-R
Revised: D.2. RLPG-R

Original: We have a multiheaded attention mechanism between our keys, values and query as described in Equation 5.
Revised: We have a multiheaded attention mechanism between our keys, values and query as described in Equation 5.

Original: For each head, we perform a scaled dot-product attention (Vaswani et al.
Revised: For each head, we perform a scaled dot-product attention (Vaswani et al.

Original: , 2017)(Equation 4).
Revised: , 2017)(Equation 4).

Original: [Equation]
Revised: [Equation]

Original: MultiHead( Q h
Revised: MultiHead

Original: [Equation]
Revised: [Equation]

Original: K hr
Revised: concat ( head i , head 2 ,

Original: [Equation]
Revised: [Equation]

Original: V hr ) =

Original: W O concat ( head i , head 2 , . . .

Original: head τ )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where head i =
Revised: where head i =

Original: Att ( W Qi Q h , W

Original: [Equation]

Original: K hr

Original: [Equation] WVi [Equation] hr )
Revised: Att ( W

Original: [Equation]
Revised: [Equation]

Original: In the equations above, d k is the dimension of the key, W Qi , W Ki , W Vi are the query, key and value projection matrices, τ is the number of heads and W O is the linear projection that combines the heads.
Revised: In the equations above, d k is the dimension of the key, W Qi , W Ki , W Vi are the query, key and value projection matrices, τ is the number of heads and W O is the linear projection that combines the heads.

Original: The output from Equation 5 is fed to a positionwise fully-connected feedforward network (Vaswani et al., 2017).
Revised: The output from Equation 5 is fed to a positionwise fully-connected feedforward network (Vaswani et al., 2017).

Original: We employ a residual connection (He et al., 2016) followed by layer normalization (Ba et al., 2016) before and after the feedforward network.
Revised: We employ a residual connection (He et al., 2016) followed by layer normalization (Ba et al., 2016) before and after the feedforward network.

Original: In the above equation these transformations are represented by the module G .
Revised: In the above equation these transformations are represented by the module G .

Original: The resulting output is then linearly projected and sigmoid is applied to get the prediction of the rule.
Revised: The resulting output is then linearly projected and sigmoid is applied to get the prediction of the rule.

Original: We used Adam (Kingma & Ba, 2015) optimizer with a learning rate of 3e-4 and batch size of 64.
Revised: We used Adam (Kingma & Ba, 2015) optimizer with a learning rate of 3e-4 and batch size of 64.

Original: We used CodeBERT (Feng et al., 2020) as our pretrained model F ϕ to obtain the representation of hole window and rule context.
Revised: We used CodeBERT (Feng et al., 2020) as our pretrained model F ϕ to obtain the representation of hole window and rule context.

Original: The size of the representation (corresponding to the hidden dimension of the [CLS] token) is 768.
Revised: The size of the representation (corresponding to the hidden dimension of the [CLS] token) is 768.

Original: In equations 1, 2 and 3 in Section 3.2, the projection matrices W Qi ∈ R d q × d model , W Ki ∈ R d k × d model , W Vi ∈ R d v × d model , W O ∈ R d model × τd v .
Revised: In equations 1, 2 and 3 in Section 3.2, the projection matrices W Qi ∈ R d q × d model , W Ki ∈ R d k × d model , W Vi ∈ R d v × d model , W O ∈ R d model × τd v .

Original: For the multihead attention, we used d k = d q = d v = 32 , τ = 4 and d model =
Revised: For the multihead attention, we used d k = d q = d v = 32 , τ = 4 and d model =

Original: 768 , W r ∈ R 63 × 768 and b r = 63 .
Revised: 768 , W r ∈ R 63 × 768 and b r = 63 .

Original: G module consists of a dropout layer, a residual connection, a layernorm, followed by a positionwise-feedforward network, i.e., A sequence of (a) dense layer of weights= 2048 × 768 , bias= 768 , (b) relu activation, (c) dense layer of weights= 768 × 2048 , bias= 2048 , (d) dropout layer, (e) residual connection, (f) layernorm.
Revised: G module consists of a dropout layer, a residual connection, a layernorm, followed by a positionwise-feedforward network, i.e., A sequence of (a) dense layer of weights= 2048 × 768 , bias= 768 , (b) relu activation, (c) dense layer of weights= 768 × 2048 , bias= 2048 , (d) dropout layer, (e) residual connection, (f) layernorm.

Original: A dropout value of 0.25 was used while training.
Revised: A dropout value of 0.25 was used while training.

Original: E. Experimental Details on Prompt Generation
Revised: E. Experimental Details on Prompt Generation

Original: We used the OpenAI Codex Completions API for generating the predicted hole from the Codex model.
Revised: We used the OpenAI Codex Completions API for generating the predicted hole from the Codex model.

Original: In particular, we used the code-davinci-001 engine with temperature set to 1.0 and stop criteria as newline.
Revised: In particular, we used the code-davinci-001 engine with temperature set to 1.0 and stop criteria as newline.

Original: The completion length was kept to be 24 and the maximum prompt length was 4072.
Revised: The completion length was kept to be 24 and the maximum prompt length was 4072.

Original: Tokenization was done using the suggested tokenizer 6 .
Revised: Tokenization was done using the suggested tokenizer 6 .

Original: To allow for fast computation, we used simple models like CodeBERT (Feng et al., 2020) and GraphCodeBERT (Guo et al., 2020) as our pretrained models.
Revised: To allow for fast computation, we used simple models like CodeBERT (Feng et al., 2020) and GraphCodeBERT (Guo et al., 2020) as our pretrained models.

Original: One of the limitations of these pretrained models is that the maximum context length that can be taken as input by these models is much smaller than the maximum context length allowed by Codex.
Revised: One of the limitations of these pretrained models is that the maximum context length that can be taken as input by these models is much smaller than the maximum context length allowed by Codex.

Original: Therefore, we need to truncate the rule context which means that some important parts of the rule context might be omitted while getting the rule context representation.
Revised: Therefore, we need to truncate the rule context which means that some important parts of the rule context might be omitted while getting the rule context representation.

Original: Using pretrained models that allow larger context length or models that augment the context (Wu et al., 2022) offer avenues for future work.
Revised: Using pretrained models that allow larger context length or models that augment the context (Wu et al., 2022) offer avenues for future work.

Original: F. Computational Complexity of RLPG
Revised: F. Computational Complexity of RLPG

Original: To collect the ground-truth data for training our rule classifier we queried the Codex API for each applicable rule per hole (maximum rate limit of 400 holes per minute).
Revised: To collect the ground-truth data for training our rule classifier we queried the Codex API for each applicable rule per hole (maximum rate limit of 400 holes per minute).

Original: The computational complexity of training our larger RLPG-R variant (parameters=3.6M, data=141269 holes and compute=9.19 min per epoch on a single Tesla V100 GPU) is much smaller than finetuning all or some part of Codex, thereby ensuring low latency when deployed in an IDE.
Revised: The computational complexity of training our larger RLPG-R variant (parameters=3.6M, data=141269 holes and compute=9.19 min per epoch on a single Tesla V100 GPU) is much smaller than finetuning all or some part of Codex, thereby ensuring low latency when deployed in an IDE.

Original: During inference, we need to calculate the repo-level rules statistics just once and all the subsequent hole completions in the repo can utilize this cached information.
Revised: During inference, we need to calculate the repo-level rules statistics just once and all the subsequent hole completions in the repo can utilize this cached information.

Original: So, there is no additional computational complexity in creating the prompt during inference.
Revised: So, there is no additional computational complexity in creating the prompt during inference.

Original: G. Related Work
Revised: G. Related Work

Original: LLMs for Code: Recently, there has been a lot of work around large language models of code.
Revised: LLMs for Code: Recently, there has been a lot of work around large language models of code.

Original: One class of models are the decoder-only models that correspond to generating code from left-to-right.
Revised: One class of models are the decoder-only models that correspond to generating code from left-to-right.

Original: Codex (Chen et al., 2021) is an example of a decoder-only model.
Revised: Codex (Chen et al., 2021) is an example of a decoder-only model.

Original: Other models include Google’s model (Austin et al., 2021), GPT-J-6B (Wang & Komatsuzaki, 2021),
Revised: Other models include Google’s model (Austin et al., 2021), GPT-J-6B (Wang & Komatsuzaki, 2021),

Original: GPT-Neo (Black et al., 2021b), GPT-Neo-X
Revised: GPT-Neo (Black et al., 2021b), GPT-Neo-X

Original: (Black et al., 2021a), CodeParrot (Tunstall et al., 2022) and PolyCoder (Xu et al., 2022a).
Revised: (Black et al., 2021a), CodeParrot (Tunstall et al., 2022) and PolyCoder (Xu et al., 2022a).

Original: We also have some encoder-only models that use a masked language modelling objective.
Revised: We also have some encoder-only models that use a masked language modelling objective.

Original: CodeBERT (Feng et al., 2020), GraphcodeBERT (Guo et al., 2020) and CuBERT (Kanade et al., 2020) are example of such models.
Revised: CodeBERT (Feng et al., 2020), GraphcodeBERT (Guo et al., 2020) and CuBERT (Kanade et al., 2020) are example of such models.

Original: Lastly, we have the class of encoder-decoder models that generally use a bidirectional encoding of a context to decode a series of masked tokens.
Revised: Lastly, we have the class of encoder-decoder models that generally use a bidirectional encoding of a context to decode a series of masked tokens.

Original: Code-T5 (Yue Wang, 2021), AlphaCode (Li et al., 2022) and recently released InCoder (Fried et al., 2022) are examples of such models.
Revised: Code-T5 (Yue Wang, 2021), AlphaCode (Li et al., 2022) and recently released InCoder (Fried et al., 2022) are examples of such models.

Original: We refer the reader to Xu et al.
Revised: We refer the reader to Xu et al.

Original: (2022a) for details about the size of each model and the data that was used for training these models.
Revised: (2022a) for details about the size of each model and the data that was used for training these models.

Original: Repo-Level Info: Few works use information from outside the current file.
Revised: Repo-Level Info: Few works use information from outside the current file.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2021) uses the parent class to generate the comments for the child class.
Revised: (2021) uses the parent class to generate the comments for the child class.

Original: Xu et al.
Revised: Xu et al.

Original: (2022b) incorporate three types of structural locality features while training the kNN-LM (Khandelwal et al., 2020).
Revised: (2022b) incorporate three types of structural locality features while training the kNN-LM (Khandelwal et al., 2020).

Original: These features are binary variables that correspond to the presence or absence of similar hierarchy.
Revised: These features are binary variables that correspond to the presence or absence of similar hierarchy.

Original: The three levels of hierarchy are (a) sibling file, (b) file in the same repo (c) no hierarchy.
Revised: The three levels of hierarchy are (a) sibling file, (b) file in the same repo (c) no hierarchy.

Original: In contrast we have a much richer set of rules incorporating the semantics and structure of the repository.
Revised: In contrast we have a much richer set of rules incorporating the semantics and structure of the repository.

Original: Also, we assume black-box access to the actual LM and restrict ourselves to generating prompt(input) for the LLM.
Revised: Also, we assume black-box access to the actual LM and restrict ourselves to generating prompt(input) for the LLM.

Original: Prompt Generation:
Revised: Prompt Generation:

Original: There have been promising works around prompt generation techniques in the area of natural language processing.
Revised: There have been promising works around prompt generation techniques in the area of natural language processing.

Original: Broadly, there are two categories of automatic prompt generation techniques.
Revised: Broadly, there are two categories of automatic prompt generation techniques.

Original: The first category corresponds to producing continuous/soft prompts where the prompt is described in the latent space of a language model (Li & Liang,
Revised: The first category corresponds to producing continuous/soft prompts where the prompt is described in the latent space of a language model (Li & Liang,

Original: ; Qin & Eisner, 2021; Bragg et al., 2021; Lester et al., 2021; Liu et al., 2021b).
Revised: ; Qin & Eisner, 2021; Bragg et al., 2021; Lester et al., 2021; Liu et al., 2021b).

Original: For example, Prefix-Tuning (Li &
Revised: For example, Prefix-Tuning (Li &

Original: Liang, 2021) adds a prefix to the LM that can be learned by finetuning on examples from the downstream task.
Revised: Liang, 2021) adds a prefix to the LM that can be learned by finetuning on examples from the downstream task.

Original: The second category produces discrete prompts where the prompt is a text string that can be interpreted by a human (Shin et al., 2020;
Revised: The second category produces discrete prompts where the prompt is a text string that can be interpreted by a human (Shin et al., 2020;

Original: Gao et al., 2021; Schick & Schütze, 2021).
Revised: Gao et al., 2021; Schick & Schütze, 2021).

Original: For example, Autoprompt (Shin et al., 2020) generates prompt using a fixed template consisting of trigger tokens.
Revised: For example, Autoprompt (Shin et al., 2020) generates prompt using a fixed template consisting of trigger tokens.

Original: The trigger tokens are shared across all inputs and determined by a gradient-guided search involving the LM.
Revised: The trigger tokens are shared across all inputs and determined by a gradient-guided search involving the LM.

Original: Our work falls in the category of discrete prompt generation techniques as we produce a prompt consisting of code tokens that can be easily interpreted by a human.
Revised: Our work falls in the category of discrete prompt generation techniques as we produce a prompt consisting of code tokens that can be easily interpreted by a human.

Original: However, in contrast to prior works that use a set of fixed templates for all examples, we learn to generate prompts conditioned on each example.
Revised: However, in contrast to prior works that use a set of fixed templates for all examples, we learn to generate prompts conditioned on each example.

Original: Another important distinction is that we do not require access to the weights of the LM.
Revised: Another important distinction is that we do not require access to the weights of the LM.

Original: To the best of our knowledge, our work is the first to explore automatic prompt generation in the domain of source code.
Revised: To the best of our knowledge, our work is the first to explore automatic prompt generation in the domain of source code.