Paper: Multilingual Code Retrieval Without Paired Data: New Datasets and Benchmarks (ICLR-2024)

Reviewer comment (Experiments):
The purpose of adding TATOEBA and CodeXGlue is not clearly articulated. They also make the claim of learning through anchor languages weaker because another factor is added to the training mix.

Author response:
**Response:** We used a relatively small number of source-source and target-target instances in order to make our training mix more diverse, and then help our models cope with shifts at evaluation (cf. [1] for a discussion on how diverse training data sources enable generalization to new, shifting distributions). CodeXGlue for instance gives us some code-code pairs. Even though the underlying languages differ from those we evaluate on, it’s a more similar task to those we expect at testing than the ones defined by other datasets.

Reviewer comment (Experiments):
It's not sure if the pre-training data of various models already have multi-lingual exposure. The near-0 auMMR number could also be task alignment issues.

Author response:
**Response:** We completely agree with the reviewer. While base models were trained on multiple languages, they were never trained to yield chunk/sentence level embeddings that semantically align. Fine-tuning is thus required to align with test tasks of interest.

Reviewer comment (Writing):
Another weakness is in writing. It's common to find long or poorly written sentences that's hard to read, or not understandable at all to the Reviewer with moderate effort. See questions for some examples.

Author response:
**Response:** We thank the reviewer for kindly remarking points of improvements in terms of clarity and presentation of our contributions. We addressed each related comment in what follows.

Reviewer comment (Writing):
What's the purpose of the AST parsing phase?

Author response:
**Response:** We used AST parsing to be able to collect natural language from code snippets (docstrings and comments). We also followed CodeSearchNet and split files into functions/methods.

Reviewer comment (Experiments):
Table 2: "high scores are observed across all language combinations". It's better to have a baseline to help understand a relative assessment of "high".

Author response:
**Response:** Thank you for pointing this out. We modified that sentence to make precise and clear what we mean by high similarity in that case.

We clarify that all of those scores are bounded in $\[0,1\]$, so that sentence suggests that the scores are much closer to the upper limit than to the lower one. We further remark that a random embedding or random data would score very close to 0, and would not yield a very informative baseline. In other words, our goal in that sentence is to highlight that the scores suggest a reasonable signal-to-noise ratio in the data. This is further confirmed in our experiments in Section 4.3.1 and in table 9 in the appendix: the data can be predicted to high accuracy so the semantic similarity must be high.

Reviewer comment (Writing):
Section 3.1 last paragraph: "Note that any data realization x observed..." It's not clear why we need to discuss it here. Better to clarify the purpose.

Author response:
**Response:** In that paragraph, we are highlighting that when we use $x$ to refer to a data instance, we are referring to a sequence of tokens. This paragraph is intended to introduce some notation to make the manuscript accessible to readers maybe not so familiar with sequence modeling settings.

Reviewer comment (Writing):
Section 4.1 second paragraph: ".., models able to generate train ..." Reviewer couldn't understand what does this mean with moderate effort.

Author response:
**Response:** *”Models able to generate”* refer to models that can be used to generate text. In more detail, our evaluation covers encoder-only, encoder-decoder, and decoder-only architectures. As such, encoder-decoder and decoder-only both can be used for text generation, in addition to yielding chunk/sentence level embeddings.

Reviewer comment (Writing):
Section 4.1 right after Equation 3: "... and define translation auxiliary tasks." What does this sentences mean?

Author response:
**Response:** This sentence explains that the models that are able to generate text, *i.e.*, encoder-decoder and decoder-only models, are trained with an additional “generative” objective compared to contrastive-only encoders.

To make the presentation clear, we will rephrase it and add an explicit reference to the generative objective discussed in Section 4.1 - Eq. (3).

Reviewer comment (Writing):
Section 4.3.2 last sentence: "Finally, the decoder-only model doesn't reach..." it's better to call out the model name (is it CodeGen?) to save the readers some back-n-forth work.

Author response:
**Response:** Yes, we agree with the reviewer and replaced it in the text.

Reviewer comment (Reproducibility):
Would you open source the data and evaluation code?

Author response:
**Response:** Yes! The datasets are already (anonymously) available:

- https://huggingface.co/datasets/blindsubmissions/M2CRB
- https://huggingface.co/datasets/blindsubmissions/GH_text2code

Links are also in the manuscript in the captions of tables 1 and 3 for training and testing data, respectively.

Code for data processing, fine-tuning, and evaluation will be released upon publication.

Reviewer comment (Experiments):
There are no baselines presented to understand how well the proposed technique actually works. While baselines might be difficult to get for non-English code search (Table 4), I would assume for Python-Java code search (Table 5) and English code search (Table 6) available model embeddings should work well. For instance, CodeBERT reports an average MRR of 0.7603. It is not immediately clear what the auMRRc would be for this model, and it would have been helpful to get these numbers. Similarly for the Python-Java code search, it would have been nice to get baselines from available multi-lingual pre-trained models.

Author response:
We kindly highlight that we did evaluate other models not trained by us such as CodeBERT and CodeT5. We observed however that those models have no predictive power in the settings we consider and perform as well as a random embedding. Note that CodeBERT MRR results openly available typically correspond to the performance after fine-tuning in a particular language combination. **We went beyond that and evaluated a single fine-tuned model, trained in the multilingual scheme we proposed.**

We further refer the reviewer to table 9 for results with state-of-the-art embeddings (OpenAI’s and Sentence Transformers’s) from models not trained by us, both open and closed source. Note that those results were kept in the appendix because we have no control over the training of the evaluated models, and in particular have no control over the data used for training. Still, those results attest for the predictability of the M$^2$CRB data and offer a performance upper bound that follow-up methods should strive to achieve.

Reviewer comment (Experiments):
This work requires paired NL data (such as between Spanish and English), and it incorporates this paired data in the loss function. However, another way to utilize this data could be to learn a translation model from English to Spanish using the paired data and use this translation model to translate English queries in the CodeSearchNet data to create a much larger (Spanish, Code snippet) dataset, albeit with some noise. This has the advantage of creating larger synthetic training data that can be directly used to train the code search model, instead of the objective proposed in the paper. Do the authors have some assessment on why the proposed technique is a better approach than this one?

Author response:
Thank you for commenting on the use of translation to generate training data. While we agree with the reviewer and, starting from an English-to-code dataset, one can indeed create synthetic variations of it by translating docstrings to different languages, it must be noted that translation step would add significant cost to pre-processing, and its quality could not be good enough considering we would be translating docstrings rather than natural language, and that we could be targeting low-resource languages.

Cost and quality concerns aside, the proposal is indeed valid and orthogonal/complementary to what we show rather than a competing approach: if one does what the reviewer suggested and, for instance, translates docstrings in CodeSearchNet to Spanish and French, we showed that training a model in such a dataset would enable mapping directly between any combination of natural or programming languages. Either approach used to build a set of paired datasets would be equally good.

In other words, combining existing training datasets or synthesizing via translation are both valid and complementary approaches to build a dataset with the properties our training requires. We highlight once more: our goal is to show that generalization is possible to indirectly observed language combinations.

We added a sentence in Section 3.2 remarking that other approaches are possible to create the style of training data we require.

Reviewer comment (Experiments):
The evaluation focuses on models within the 60M-360M parameter range, and exploring larger-scale models could provide insights into the effect of model size on generalization capabilities in this domain.

Author response:
**Response:** We completely agree with the reviewer and highlight that the decision on the model scale we focused on was in part influenced by the amount of compute we had available. We also note that recent literature (*e.g.*, [1]) has shown that encoders do not observe scaling gains as pronounced as decoders. For instance, the extremely popular `sentence-transformers/all-MiniLM-L12-v2` encoder has only 33M parameters. Another recently released encoder, `bigcode/starencoder`, has 125M parameters. **We thus believe the range we chose to be very well aligned with common practice in this setting**.

Reviewer comment (Experiments):
Can the training approach proposed in this paper be adapted for generative models, and if so, how would it affect their performance on text-to-code generation tasks?

Author response:
**Response:** As discussed above and in the manuscript, our proposed training strategy would be useful as-is to enable multi-linguality in encoder-decoder architectures. Even more than that, our results show that this can be done while also carrying out contrastive training on top of chunk-level embeddings obtained from the encoder. So to answer your question, **yes**, conditional generative models can benefit from our proposal, and **no**, we do not anticipate it affecting the performance of the base model it builds upon.

Reviewer comment (Experiments):
Are there any potential biases in the dataset, such as the influence of specific programming language styles or the quality of non-English docstrings, that may affect the model's generalization capability?

Author response:
**Response:** Thank you for this insightful question. We measured language-specific text-code alignment (via `BERTScore` and our proposed `GPTScore`) and did observe variation across languages. Indeed, the textual component of the data we make available consists of naturally occurring comments on GitHub and, as such, relatively high variance is expected given how large a community of developers GitHub represents, and subcommunities will have different practices. On the other hand, natural data is reflective of what a real-world codebase would look like, and thus defines a useful "in-the-wild" testbed.

Reviewer comment (Experiments):
How do the models perform when fine-tuned on different programming languages (e.g., C++, Rust, etc.) and less common natural languages? Would the performance be consistent with the results presented in the paper?

Author response:
**Response:** Our results suggest strong generalization ability to new language combinations for various model classes and training settings. We do not anticipate any issue specific to the languages highlighted by the reviewer, provided that they are included in the training mix (not necessarily paired as we’ve shown).

Reviewer comment (Experiments):
How would the results change when using larger-scale models, such as GPT-3 or the recent Megatron-LM? Would the generalization capabilities improve with increased model capacity?

Author response:
**Response:** We kindly refer the reviewer to item 3 above with related discussion. While this would need to be tested in our specific setting, recent literature suggests gains on encoders are not as pronounced with scaling as they are for decoders. This is also confirmed by the fact that our best performing models were not the biggest ones. We also highlight that our decision in terms of which models to fine-tune was in part motivated by the amount of compute we had available for this project.

Reviewer comment (Experiments):
The training and evaluations are not motivated well.

Author response:
**Response:** We agree with your other comment, *i.e.*, “*we know that multilingual LMs learn to map text or code in a shared embedding space that enables them to perform on unseen language pairs*”. However, the embedding spaces learned by LMs do not immediately lead to embeddings which can be used to query code for retrieval applications. One of the motivations of this work is to show that we can adapt the representations learned by LMs and make them much more effective for cross-lingual search. 

We further clarify that evaluations follow the standard information retrieval setting where, given a query, one seeks $k$ relevant entries from a database. That’s our target application as discussed in the text. We discuss motivations further in item 5 below. As per training, we used contrastive learning approaches to yield a semantic embedding that is language agnostic so as to enable multi-linguality.

While we believe the summary above is written down somewhat clearly in the text, we would be happy to modify any part of the content that the reviewer believes is confusing or misleading.

Reviewer comment (Novelty):
The main questions addressed in the paper are somewhat known.

Author response:
**Response:** The main question we seek to address is as follows: **Can a contrastively trained embedding enable retrieval for language combinations only indirectly seen during training?** As we argued in the text, if yes, then training of this kind of model can be carried out without paired data from all languages in the set of languages of interest. **To our knowledge, ours is the first work showing evidence of that being the case.**

Reviewer comment (Novelty):
I couldn't perceive the value and need of the dataset.

Author response:
**Response:** That’s actually the main motivating point of the line of work we carry out in this paper: **translation steps add overhead and should be removed**. This translation overhead (be it machine or human translation) mainly affects non-English-speaking communities of developers or developers required to write documentation in local language. We thus make available both data and efficient training strategies that help bypass undesirable intermediate translation steps.

We also highlight that existing machine translators are trained in natural language, and don’t necessarily work well on docstrings and code comments which embed jargon, variable names, and other code specific information. Machine translation might also be unreliable in low-resource languages.

Reviewer comment (Experiments):
If we need multilingual docstring, isn't it possible to use NMT models to translate the docstring available in English? Some experiments in the Appendix use Google translation API which should be part of the main body and discussion. This way the value of the dataset could be better demonstrated.

Author response:
**Response:** Please refer to Table 10 in the appendix where we compare direct search with models that use an external translator of docstrings. We see that multilingual models perform better than *translate+search* using an English-only model. We also see that multilingual models can even further benefit from translating docstrings into English, so even if translation is an option, multilingual training is beneficial.

Reviewer comment (Novelty):
"Moreover, the search setting is less compute-intensive relative to common language models, rendering experimentation more accessible" - Is it a good reason to study code search tasks?

Author response:
**Response:** The main motivation we considered for doing code search as opposed to generation is to address situations where **users wish to obtain pieces of code from a known and tested codebase**. There are many situations where the user desires retrieval results and in the case of retrieval augmented generation (RAG) models, our approach can serve as the underlying representation used to perform the retrieval step as well as a generator.  One can also consider the situation of proprietary codebases that are not in the training set of large code LMs. In those cases, generative models wouldn’t be able to generate code that makes sense in the context of an API or library it never had access to and knows nothing about.

We additionally highlight that code search is an **extremely active research field** with several papers coming out every week, especially so given the recent trend in combining retrievers with decoders to perform **retrieval augmented generation (RAG)**.

Finally, as noted by the reviewer, we also highlighted in the paper that retrieval applications are typically less expensive in terms of compute resources requirements, and might be more accessible as a research topic to groups with compute constraints.

Reviewer comment (Evaluation):
"Contrary to what is usually observed in more standard evaluation conditions, in this multilingual setting, scaling up model size will not necessarily improve performance." - Why?

Author response:
**Response:** As discussed in the text, we believe this to be due to the distribution shift we observe in our evaluations. Most existing work on the definition of scaling laws observed monotonic improvements with capacity specifically for in-distribution evaluation. Our evaluation on the other hand, induces a **severe distribution shift** since we evaluate models on language combinations **completely absent from the training data**. Our results thus suggest an in-distribution vs. out-distribution tradeoff controlled via model capacity.

Reviewer comment (Novelty):
In 2-3 lines, please outline what new findings this paper presents.

Author response:
**Response:** While the two datasets we open-source are central contributions, our paper reports several findings the research community can benefit from, as outlined below and discussed in the manuscript:

1- We show evidence that multilingual contrastively trained encoders generalize to language combinations only indirectly observed during training.

2- We show that encoder-decoder and decoder-only LM architectures can effectively be trained to perform both search and generation, and no performance tradeoffs are observed in this case.

3- We show that multi-linguality brings in specific challenges and design choices must be made with this setting in mind. For instance, dual-encoder architectures, state-of-the-art for text retrieval, are outperformed by single encoders in this situation.

4- We show that multilingual LLMs can be used for evaluating semantic match between docstrings and code snippets. Interestingly, the judgments of GPT-3.5-turbo correlate with `BERTScore`, and both are predictive of search performance.