Interpreting Indirect Answers to Yes-No Questions in Multiple Languages (2023)

Abstract: Yes-no questions expect a yes or no for an answer, but people often skip polar keywords. Instead, they answer with long explanations that must be interpreted. In this paper, we focus on this challenging problem and release new benchmarks in eight languages. We present a distant supervision approach to collect training data, and demonstrate that direct answers (i.e., with polar keywords) are useful to train models to interpret indirect answers (i.e., without polar keywords). We show that monolingual fine-tuning is beneficial if training data can be obtained via distant supervision for the language of interest (5 languages). Additionally, we show that cross-lingual fine-tuning is always beneficial (8 languages).

## introduction multilingual question-answering has recently received substantial attention @xcite . state-of-the-art models such as xlm-e @xcite , however, achieve only 68% to 76% f1-score on multilingual question-answering benchmarks such as mlqa @xcite and xquad @xcite . large language models (llms) such as instruct-gpt @xcite obtain promising results with several english question-answering benchmarks @xcite @xcite . closed-source, proprietary llms for which the training data is unknown raise issues regarding replicability and data leakage @xcite . open-source llms such as llama are pretrained on latin or cyrillic scripts @xcite thus have limitations with other scripts. yes-no questions are questions that expect a yes or no for an answer. humans, however, often answer these kinds of questions without using a yes or no keyword. rather, they provide indirect answers that must be interpreted to reveal the underlying meaning (see examples in many nlp problems were initially investigated in english @xcite . even though yes-no questions and indirect answers have been studied for decades @xcite , previous efforts to date have predominantly focused on english (section 2). in this paper, we tackle this challenging problem in eight additional languages: hindi (hi), korean (ko), chinese (zh), bangla (bn), turkish (tr), spanish (es), nepali (ne), and persian (fa). this paper focuses on multilingual interpretation of indirect answers to yes-no questions. doing so opens the door to several applications. for example, dialogue systems could avoid inconsistencies and contradictions @xcite . consider the examples in the main contributions are as follows: foot_0 1. a distant supervision approach to collect yesno questions and direct answers along with their interpretations. we use this approach to collect training data in five languages. 2. evaluation benchmarks in eight languages in which no resources exist for interpreting indirect answers to yes-no questions. 3. experimental results showing that training with the direct answers obtained via distant supervision is beneficial to interpret indirect answers in the same language. 4. experimental results expanding on (3) and showing that multilingual training is beneficial, even when no additional training data for the language of interest is available. yes-no questions have also been studied for decades @xcite . recent work includes large corpora such as boolq @xcite (16,000 yes-no questions submitted to a search engine) and extensions including unanswerable questions @xcite . yes-no questions have also been studied within the dialogue domain @xcite . these dialogues, however, are synthetic and constrained to a handful of topics. circa @xcite , friends-qia @xcite , and swda-ia @xcite , also explore yes-no questions in dialogues (crowdsourced, modified tv scripts, and phone conversations). we are inspired by their interpretations and use their corpora in our experiments (section 5). all computational works on yes-no questions to date are in english. we are the first to target eight additional languages. crucially, we do by exploring distant supervision and cross-lingual learning; our approach does not require tedious manual annotations. ## obtaining training data for training purposes, we work with six languages. specifically, we work with existing english corpora (section 3.1) and corpora obtained in five additional languages via distant supervision (section 3.2). as we shall see, multilingual transfer learning is successful even when no examples in the language of interest are available (section 5.3). ## benchmarks in new languages we are the first to work on interpreting answers to yes-no questions in languages other than english (hindi, korean, chinese, bangla, turkish, spanish, nepali, and persian). we set to work with questions and answers written in the languages of interest to avoid translationese @xcite , so we create new benchmarks with 300 question-answer pairs per dataset for each language (chinese and turkish: 600 samples; other languages: 300 samples). for the five languages ## experiments we develop multilingual models to reveal the underlying interpretations of indirect answers to yesno questions. we follow three settings: (a) crosslingual learning to the new language (after training with english, section 5.1); (b) monolingual finetuning via distant supervision with the language of interest (section 5.2); and (c) multilingual finetuning via distant supervision with many languages (section 5.3). following previous work (section 2), we use the question and answer as inputs. we conduct our experiments with two multilingual transformers: xlm-roberta @xcite and xlm-align @xcite . both are obtained from huggingface @xcite and were pretrained on hundreds of languages. we split our benchmarks into validation (20%) and test (80%), and report results with the test split. cross-lingual learning was conducted with a model trained and validated exclusively in english (with circa and swda-ia). monolingual and multilingual fine-tuning use the validation split in each language for hyperparameter tuning, the test set is always the same. we will discuss results with xlm-roberta, as it outperforms xlm-align by a small margin. appendix f and g detail the results with xlm-align and the hyperparameters. ## conclusions we have tackled for the first time the problem of interpreting indirect answers to yes-no questions in languages other than english. these kinds of answers do not include yes or no keywords. their interpretations, however, are often either yes or no. indeed, middle accounts for between 12.7% and 33.7% depending on the language ( we have created new evaluation benchmarks in eight languages: hindi, korean, chinese, bangla, turkish, spanish, nepali, and persian. additionally, we present a distant supervision approach to identify yes-no questions and direct answers. training with this new, noisy, language-specific data is always beneficial-and significantly better in five languages. cross-lingual learning from many languages to a new language is the best strategy. this is true even for the three languages for which distant supervision was not explored because of difficulties finding large unannotated corpora (bangla, nepali, and persian). our approach successfully learns to interpret indirect answers from direct answers obtained via distant supervision. our future work includes exploring more robust multilingual fine-tuning @xcite . we also plan to explore applications of the fundamental research presented here. in particular, we are interested in improving dialogue consistency and avoiding inconsistencies by ensuring that generated turns are compatible with the correct interpretation of indirect answers to yes-no questions.