A Dataset and Baselines for Multilingual Reply Suggestion (2021)

Abstract: Reply suggestion models help users process emails and chats faster. Previous work only studies English reply suggestion. Instead, we present MRS, a multilingual reply suggestion dataset with ten languages. MRS can be used to compare two families of models: 1) retrieval models that select the reply from a fixed set and 2) generation models that produce the reply from scratch. Therefore, MRS complements existing cross-lingual generalization benchmarks that focus on classification and sequence labeling tasks. We build a generation model and a retrieval model as baselines for MRS. The two models have different strengths in the monolingual setting, and they require different strategies to generalize across languages. MRS is publicly available at https://github.com/zhangmozhi/mrs .

## dataset construction to study reply suggestion in multiple languages, we build mrs, a dataset with message-reply pairs based on reddit comments. the dataset is available at https://github.com/zhangmozhi/mrs . we download reddit comments between january 2010 and december 2019 from the pushshift reddit dataset @xcite . 1 we extract message-reply pairs from each thread by considering the parent comment as an input message and the response to the comment as the reference reply. we remove comments starting with [removed] or [deleted], which are deleted messages. we also skip comments with a rating of less than one, since they are likely to contain inappropriate content. after extracting examples, we identify their languages with fasttext language detector @xcite . for each example, we run the model on the concatenation of the message and the reply. we discard low-confidence examples where none of the languages has a score higher than 0.7. for the remaining examples, we use the highest-scoring label as the language. we only use english data from 2018 because english data is abundant on reddit. non-english examples are much more scarce, so we use data from the last ten years. we select the top ten languages with at least 100k examples. we create three splits for each language: 80% examples for training, 10% for validation, and 10% for testing. we also build response sets and filter out toxic examples. we describe these steps next. ## experiment settings after presenting the dataset, we explain how we use mrs to compare reply suggestion models. we describe the two frameworks for reply suggestion, our experiment settings, and evaluation metrics. ## baseline models this section introduces the two baseline models: a retrieval model and a generation model. ## results and discussion we experiment with the two baselines from section 4 on mrs. we first compare the models in english, where we have enough training data and human referees. we then build models for other languages and compare training settings listed in section 3.2. ## conclusion we present mrs, a multilingual dataset for reply suggestion. we compare a generation and a retrieval baseline on mrs. the two models have dif-ferent strengths in the english monolingual setting and require different strategies to transfer across languages. mrs provides a benchmark for future research in both reply suggestion and cross-lingual transfer learning.