Zero-Shot vs. Translation-Based Cross-Lingual Transfer: The Case of Lexical Gaps (2024)

Abstract: Cross-lingual transfer can be achieved through two main approaches: zero-shot transfer or machine translation (MT). While the former has been the dominant approach, both have been shown to be competitive. In this work, we compare the current performance and long-term viability of these methods. We leverage lexical gaps to create a multilingual question answering dataset, which provides a difficult domain for evaluation. Both approaches struggle in this setting, though zero-shot transfer performs better, as current MT outputs are not specific enough for the task. Using oracle translation offers the best performance, showing that this approach can perform well long-term, however current MT quality is a bottleneck. We also conduct an exploratory study to see if humans produce translations sufficient for the task with only general instructions. We find this to be true for the majority of translators, but not all. This indicates that while translation has the potential to outperform zero-shot approaches, creating MT models that generate accurate task-specific translations may not be straightforward.

## introduction cross-lingual transfer has helped to develop natural language processing (nlp) systems for a broader set of languages through two main approaches: zero-shot transfer, where a single multilingual model is finetuned on a source language and directly applied to a target language, and translationbased transfer, where data is translated via a machine translation (mt) system before being passed to a downstream model. while zero-shot transfer is more widely used, this has been called into question both for high-resource @xcite and low-resource @xcite languages. in this work, we ask the overall question: which approach has the greater potential in creating nlp systems which perform at a high level for all languages? to do so, we create a focused question answering (qa) dataset leveraging a phenomena we expect to be challenging for both approaches: lexical gaps, or concepts which are explicitly denoted in one language that can only be expressed as a combination of words in another @xcite . lexical gaps exist in a variety of domains, such as colors or foods @xcite , however due to its prevalence across many languages, in this work we focus specifically on kinship terminology: farsi, e.g., marks the difference between amoo (english: paternal uncle), and daei (english: maternal uncle), while, in english, both relatives are generally just referred to as uncle. examples in the dataset, created using templates, are simply structured and easy to solve for a human, though accurate translation or recognition of the relevant lexical gaps are required to identify the correct answer. while focusing solely on lexical gaps through this approach reduces example diversity, there is a trade-off as we gain more control in our experiments and the ability for finer-grained analysis. evaluating on a diverse set of 5 languages -catalan, german, farsi, hindi, and vietnamese -we first investigate existing models and find that both translation-based and zero-shot approaches struggle in this setting. however, the zero-shot approach is stronger, and we find that current mt systems do not preserve the required amount of detail for the task, instead falling back to the general translation (e.g., amoo being translated to uncle). replacing mt outputs with an oracle translation, however, offers the best performance across all methods, showing that with sufficiently accurate translation, this approach can overcome the challenge of lexical gaps. to see if collecting additional data is a feasible approach for closing the gap between current mt outputs and oracle translations, we conduct italics represent the slots which were filled by kinship terms or value surface forms (e.g., "sena", "ali", "ghormeh sabzi"). the answerable column denotes if the original example is answerable or unanswerable. the t column marks if the translation is specific (✓) or general (✗), and how using the general translation affects the perceived correct answer. lgi denotes the lexical gap id, which depends on the relation between the two kinship entities found in the contexts, and whether they can conflict or not when translated to english. a case study with farsi to see if humans produce translations appropriate for the task without any task-specific instructions. our results are mixed: most, but not all, human translators produce translations close to the oracle which preserve the lexical gap information. thus, while translation may be a viable long-term approach for cross-lingual transfer, building an appropriate mt system represents a bottleneck, as relying purely on any generally collected parallel data may not be sufficient for a specific downstream task. ## dataset construction we construct a qa dataset around lexical gaps, which consists of simple, factual sentences made via templates. this approach has several benefits: it allows for a fine-grained analysis of lexical gaps, is easily extensible to multiple languages, has a small chance of overlap with the pretraining data, and creates a challenging evaluation for llms while remaining trivial for humans. a complete example from the dataset can be found in the question and context templates are then translated to the target language. to prevent an english bias, annotators are asked to create slot values which are natural to the language, and are allowed to modify the english templates to create the most natural target language translation. this creates a parallel set of templates, one in the target language and one in english. while we start with english in the dataset creation, for our experiments we refer to the english as the oracle translation from the target language, as the entity slot is filled with the specific translation and the meaning is completely preserved. by starting from the same set of english templates for each languages, the dataset remains roughly semantically parallel across the different target languages. additional details on the translation process can be found in appendix a.1. qa examples are created by pairing two context templates together and selecting one of the corresponding question templates. depending on how the slots are filled, examples can be either answerable, i.e., the answer is found in one of the contexts, or unanswerable, created by switching the entity in the question template. questions can be unanswerable due to either missing or conflicting information. we categorize examples into three categories, denoted by the lexical gap id (lgi), which depends on the relationship between the two entities found in the context template pair. lgi 0 denotes examples where the entities do not conflict with respect to a lexical gap, i.e., the general form of the two slots is different. examples with lgi 1 and 2 are those where the general value is the same, which causes conflict after poor translation. lgi 1 denotes examples which are answerable in the target language which can become unanswerable after translation, while lgi 2 denotes examples which are unanswerable in the target language and become erroneously answerable if the translation does not use the specific forms. therefore, while lgi 0 example can be either answerable or unanswerable, they will remain so regardless of the quality of translation; for lgi 1 and 2, poor translation will corrupt the context, and the answerability of the example can change. the task is framed as a 3-way classification problem, with each value representing an answer choice, along with one option for 'not answerable' @xcite . context and answer choice order are shuffled for each example. we collect translations for farsi, hindi, vietnamese, catalan, and german, with the first three representing languages with different kinship terminologies than english, and the latter two having a terminology which is practically one-to-one with english. ## human translations of lexical gaps in the previous section, we find that current mt models do not translate lexical gaps with the accuracy necessary to solve the qa task. a common approach to improve model performance is continued pretraining @xcite , and we can consider a situation where additional translations are collected for this adaptation. with this study, we are interested in learning how humans translate contexts containing lexical gaps, particularly if there are two entities which, when found independently, are typically translated to the same word. specifically, we aim to find if (1) for our examples, will humans produce translations close to our oracle translation if they have no knowledge of the downstream task? and (2) if not, how do humans translate sentences which contain conflicting lexical gaps? namely, do humans follow the 'one sense per discourse' hypothesis @xcite in our setting, and disambiguate conflicting lexical gaps which arise in the same context pair? ## conclusion in this work, we compare zero-shot and translationbased approaches to cross-lingual transfer with a dataset created around lexical gaps. using current models, zero-shot transfer offers better performance, and mt models are not capable of sufficiently accurate translations. using oracle translation reveals the long-term viability of the latter approach, however we find that generally collected translations are not always suitable for the task. as such, collecting task-specific data, or using approaches such as neuro-symbolic models, which can incorporate rules, may be necessary for strong translation-based transfer.