Paper: A Benchmark for Learning to Translate a New Language from One Grammar Book (ICLR-2024)

Reviewer comment (Experiments):
The experiments would be improved by comparing other types of models besides recent LLMs. Sequence-to-sequence PLMs like mT5 have been shown to perform well on low-resource MT (https://aclanthology.org/2022.naacl-main.223.pdf). Furthermore, the nature of MTOB could be leveraged by more specialised neural architectures, such as neural MT models that incorporate bilingual dictionaries (https://aclanthology.org/2021.acl-long.382/, https://aclanthology.org/2020.acl-main.143.pdf). An analysis spanning different models would reveal more about the true difficulty of the task. There is a growing literature evaluating LLMs for low-resource MT (e.g. https://arxiv.org/pdf/2309.07423v1.pdf) and so far it seems that they fall short of massively multilingual NMT models, which could just be because they are trained/tuned for very different types of tasks.

Author response:
* We agree that we should have included a traditional machine translation method as a baseline. We have run baselines finetuned on the ~1100 available parallel sentences and included full details along with a table of results in the general response to all reviewers. We also agree that future work should explore other kinds of architectures/LLMs finetuned specifically for machine translation; we think an interesting research question is how to most effectively combine traditional massively multilingual machine translation data with instruction tuning/in-context learning capabilities. As you point out with the Robinson et al. paper, even though LLMs are trained on both kinds of data, the MT capabilities don’t always make it through.

Reviewer comment (Evaluation):
While some qualitative examples are provided in the appendix, the paper would be strengthened by more such analysis. Could error types across models / in-context settings be quantified to some extent? More generally, since MTOB is framed as a unique benchmark, some interesting findings from a more nuanced evaluation framework (along the lines of page 7, paragraph 3 “In contrast, the grammar book…”) would be expected. Such analysis could help motivate why MTOB is unique (e.g. if it is found that LLMs make certain types of errors on MTOB that they do not make on other tasks).

Author response:
* We agree that more interpretable evaluation is always better. Unfortunately analyzing the results in this way, especially across different kinds of retrieved context, is very time-consuming and error-prone since there is a lot of context retrieved for each example (as in Figures 7 and 8, let alone the 50K and 100K chunks). It also seems safe to say that we can’t automate this analysis using current LLMs, since they are the ones generating the outputs with errors. We think that the qualitative examples and analysis in the paper give a good sampling of the kinds of errors that the models and retrieval methods make, and the chrF scores match our subjective impressions of output quality well. The fact that the errors are qualitatively similar to hallucinations in LLMs more generally and that there are fewer errors with better LLMs—despite the exclusion from pretraining data—is evidence that in-context learning in LLMs is really improving and underscores the value of MTOB.

Reviewer comment (Experiments):
For some extremely low-resource languages, LLMs have basically no translation capabilities (see some of the zero-shot experiments here https://arxiv.org/pdf/2309.07423v1.pdf), even though these languages are included in publicly available test sets. The paper would be improved through some comparison of the MTOB task with extremely low-resource MT, e.g. qualitative differences in model performance or proof of data contamination even for extremely low-resource languages.

Author response:
1. Yes, we mention in the expanded Limitations section in the Appendix (page 22 under “Potential Contamination”) that the train/test parallel sentences are in a CSV file on GitHub as of February 2021, as part of the source code for Dictionaria. (And we mentioned in the introduction the steps we are taking to prevent additional contamination due to the release of this paper.) It is possible that LLMs were trained on it, or maybe it was filtered out because the CSV file is very messy. Regardless, we saw no signs that the model baseline outputs were memorized; they changed very reasonably with different kinds of provided context.

Reviewer comment (Experiments):
A minor wish would have been to see some additional baselines involving more standard MT approaches. If we're saying that this work has the potential of helping with the translation of under-resourced languages, it might be worth trying, amongst others: standard neural MT, trained on the little parallel data that’s available + parallel data from related languages; traditional rule-base MT; neural MT trained on synthetic data generated from templates. I recognise that these approaches would likely be more involved than simply feeding data to an LLM, but they are all reasonable approaches that a researcher interested in building MT for Kalamang might try.

Author response:
Thank you for your close reading of our paper and for your helpful feedback! 

With regard to the weakness you mentioned, we agree that we should have included a traditional machine translation method as a baseline. We have run this baseline and included full details along with a table of results in the general response to all reviewers. 

We also agree that it would also be interesting to evaluate more classical methods such as rule-based machine translation. We did not have time to design such a system in this short response period, but we will add a discussion of these methods to the paper. We think it would be especially interesting in future work to try to design a system combining rule-based translation and in-context learning from grammar books; we hope that our dataset can provide a robust means of evaluation in such cases.

Reviewer comment (Experiments):
How do you expect this approach to stack up against traditional MT (e.g. neural sequence-to-sequence MT with e.g. a transformer encoder/decoder architecture), when using techniques such as cross-lingual transfer from related languages, backtranslation, data mining, synthetic data augmentation?

Author response:
Thank you for your close reading of our paper and for your helpful feedback! 

With regard to your question, we expect that the benefit of data augmentation techniques scales with the quality of the model that generates the synthetic data. So if the number of available parallel sentences is small, the quality of synthetic data generated with a traditional finetuned NMT model will not be great, even with multilingual transfer etc. By contrast, if a long context LLM can learn to translate in context using a grammar book (and we know that humans can perform this task pretty well from the same data), then this could be used to generate much higher quality synthetic data to either self-improve or distill into a smaller model.