READIN : A C hinese Multi-Task Benchmark with Realistic and Diverse Input Noises (2023)

Abstract: For many real-world applications, the user-generated inputs usually contain various noises due to speech recognition errors caused by linguistic variations or typographical errors (typos). Thus, it is crucial to test model performance on data with realistic input noises to ensure robustness and fairness. However, little study has been done to construct such benchmarks for Chinese, where various language-specific input noises happen in the real world. In order to fill this important gap, we construct READIN: a Chinese multi-task benchmark with REalistic And Diverse Input Noises. READIN contains four diverse tasks and requests annotators to re-enter the original test data with two commonly used Chinese input methods: Pinyin input and speech input. We designed our annotation pipeline to maximize diversity, for example by instructing the annotators to use diverse input method editors (IMEs) for keyboard noises and recruiting speakers from diverse dialectical groups for speech noises. We experiment with a series of strong pretrained language models as well as robust training methods, we find that these models often suffer significant performance drops on READIN even with robustness methods like data augmentation. As the first large-scale attempt in creating a benchmark with noises geared towards user-generated inputs, we believe that READIN serves as an important complement to existing Chinese NLP benchmarks. The source code and dataset can be obtained from https://github.com/thunlp/READIN .

## introduction user-generated inputs in real-world applications often contain noises where wrong characters or words are used instead of the intended ones @xcite . this is especially true when users type fast or are using speech input in noisy environments or with less common accents that cause errors in postprocessing systems. however, most benchmarks used in academic research do not explicitly try to capture such real-world input noises @xcite , leaving the doubt whether models performing well on standard clean test sets can transfer well onto real-world user-generated data. to evaluate the performance on noisy data for languages like english, existing work typically generates typos via character-level perturbation such as randomly sampled or adversarial character swap or deletion @xcite @xcite , automatic back-translation and speech conversion @xcite . however, there are many factors not considered in the automatic approaches, for example, the keyboard design of users' devices and speakers' phonetic and phonological variations. these overlooked factors have a large impact on the types of noises possible in keyboard and speech inputs. one notable exception to the above is noiseqa @xcite . apart from automatic approaches, they also collected test sets with noises produced by annotators. their dataset only considered the question answering task and is only in english. in this paper, we focus on chinese instead and present a multi-task benchmark with realistic and diverse input noise, named readin. compared to the case of english, chinese input noises have very different patterns due to the very different nature of the two languages. chinese is a pictographic language without morphological inflections that are common in indo-european languages. also, the tone system is a unique and integral part of chinese phonology but not in english. such differences cause different types of input noises in both keyboard typing and speech input. to compre-original 花呗怎么不能提额了(1a) huā bei zěn me bù néng tí é le why can't i raise my quota on huabei? keyboard 花呗怎么不能贴了(1b) huā bei zěn me bù néng tiē le speech 画呗怎么不能提饿了(1c) huà bei zěn me bù néng tí è le we consider noises occurring in two widely used chinese input methods, keyboard input and speech input, and provide an example in for keyboard input, chinese users need to use an input method editor (ime) to convert the raw transliteration foot_1 sequences into chinese characters. in such cases, noises can either occur in the transliteration input, or occur when users are choosing the intended word from the candidate list suggested by the ime. it is different from the case of english where typos and spelling variations are expected to happen on the character level. the noise patterns are further coupled with the typing habits of individual users, for example, whether they type the full pinyin transliteration or just the abbreviations results in different noise patterns. in order to capture these nuances, we recruit annotators with different typing habits and instruct them to use different imes for typing. for speech input, noises could arise when the speakers' accents or background noises lead to failures of the post-processing automatic speech recognition (asr) systems. to capture these, we recruit 10 speakers from different regions of china to cover diverse accents and use a commonly used chinese commercial asr system for post-processing. for instance, in we take many additional measures in the annotation process in order to capture the real-world input noise distribution, as detailed in section 2. in section 3, we provide more statistics and analysis of the collected data. in section 4, we train strong baseline models on the clean training data and test the models on our readin test sets. the results indicate that these models suffer significant performance drops on the real-world input noises, leaving ample room for future improvement. ## annotation process our annotation asks crowdworkers to re-enter clean test data from these existing nlp datasets. our goal is to induce realistic and diverse input noises in the annotation. we collect data using two different types of input methods: keyboard (pinyin) input and speech input, both are commonly used among chinese users @xcite . all examples are annotated with both input methods and we keep two separate tracks for data collected with these two different input methods. in the following subsections, we first introduce the four tasks and the original datasets that our annotations are based on, and then introduce the annotation process for keyboard input and speech input respectively. ## dataset overview in this section, we analyse the annotated noisy test sets, including data statistics, our proposed metrics for robustness evaluation, a manual quality assessment of the annotated data as well as a qualitative analysis of the diverse types of input noises. ## experiments we benchmark several pretrained language models and examine whether their performance stays strong on readin. ## conclusion in this work, we present readin -the first chinese multi-task benchmark with realistic and diverse input noises. our annotation is carefully designed to elicit realistic and diverse input noises for both keyboard pinyin input and speech input. through both quantitative and qualitative human evaluation, we show that our crowdsourced input noises are much more plausible and diverse than existing automatically created ones. our experiments on strong pretrained language model baselines show that models suffer significant drops on our noisy test sets, in-dicating the need for more robust methods against input noises that would happen in the real world.