Don’t Rule Out Monolingual Speakers: A Method For Crowdsourcing Machine Translation Data (2021)

Abstract: High-performing machine translation (MT) systems can help overcome language barriers while making it possible for everyone to communicate and use language technologies in the language of their choice. However, such systems require large amounts of parallel sentences for training, and translators can be difficult to find and expensive. Here, we present a data collection strategy for MT which, in contrast, is cheap and simple, as it does not require bilingual speakers. Based on the insight that humans pay specific attention to movements, we use graphics interchange formats (GIFs) as a pivot to collect parallel sentences from monolingual annotators. We use our strategy to collect data in Hindi, Tamil and English. As a baseline, we also collect data using images as a pivot. We perform an intrinsic evaluation by manually evaluating a subset of the sentence pairs and an extrinsic evaluation by finetuning mBART (Liu et al., 2020) on the collected data. We find that sentences collected via GIFs are indeed of higher quality.

## introduction machine translation (mt) -automatic translation of text from one natural language into anotherprovides access to information written in foreign languages and enables communication between speakers of different languages. however, developing high performing mt systems requires large amounts of training data in the form of parallel sentences -a resource which is often difficult and expensive to obtain, especially for languages less frequently studied in natural language processing (nlp), endangered languages, or dialects. for some languages, it is possible to scrape data from the web @xcite , or to leverage existing translations, e.g., of movie subtitles @xcite or religious texts @xcite . however, such sources of data are only available for a limited number of languages, and it is impossible to collect large mt corpora for a diverse set of languages using these methods. professional translators, which are a straightforward alternative, are often rare or expensive. in this paper, we propose a new data collection strategy which is cheap, simple, effective and, importantly, does not require professional translators or even bilingual speakers. it is based on two assumptions: (1) non-textual modalities can serve as a pivot for the annotation process @xcite ; and (2) annotators subconsciously pay increased attention to moving objects, since humans are extremely good at detecting motion, a crucial skill for survival @xcite . thus, we propose to leverage graphics interchange formats (gifs) as a pivot to collect parallel data in two or more languages. we prefer gifs over videos as they are short in duration, do not require audio for understanding and describe a comprehensive story visually. furthermore, we hypothesize that gifs are better pivots than images -which are suggested by @xcite for mt data collection -based on our second assumption. we expect that people who are looking at the same gif tend to focus on the main action and characters within the gif and, thus, tend to write more similar sentences. this is in contrast to using images as a pivot, where people are more likely to focus on different parts of the image and, hence, to write different sentences, cf. figure 1 . we experiment with collecting hindi, tamil and english sentences via amazon mechanical turk (mturk), using both gifs and images as pivots. as an additional baseline, we compare to data collected in previous work @xcite . we perform both intrinsic and extrinsic evaluations -by manually evaluating the collected sentences and by training mt systems on the collected data, respectively -and find that leveraging gifs indeed results in parallel sentences of higher quality as compared to our baselines. foot_0 ## related work in recent years, especially with the success of transfer learning @xcite and pretraining in nlp @xcite , several techniques for improving neural mt for low-resource languages have been proposed @xcite @xcite @xcite . however, supervised methods still outperform their unsupervised and semi-supervised counterparts, which makes collecting training data for mt important. prior work scrapes data from the web @xcite , or uses movie subtitles @xcite , religious texts @xcite , or multilingual parliament proceedings @xcite . however, those and similar resources are only available for a limited set of languages. a large amount of data for a diverse set of low-resource languages cannot be collected using these methods. for low-resource languages, hasan et al. (2020) propose a method to convert noisy parallel documents into parallel sentences. @xcite filter noisy sentence pairs from mt training data. the closest work to ours is @xcite . the authors collect (pseudo-)parallel sentences with images from the flickr8k dataset @xcite as a pivot, filtering to obtain images which are simplistic and do not contain culture-specific references. since flickr8k already contains 5 english captions per image, they select images whose captions are short and of high similarity to each other. culture-specific images are manually discarded. we compare to the data from @xcite in section 4, denoting it as m20. ## conclusion in this work, we made two assumptions: (1) that a non-textual modality can serve as a pivot for mt data collection, and (2) that humans tend to focus on moving objects. based on this, we proposed to collect parallel sentences for mt using gifs as pivots, eliminating the need for bilingual speakers and reducing annotation costs. we collected parallel sentences in english, hindi and tamil using our approach and conducted intrinsic and extrinsic evaluations of the obtained data, comparing our strategy to two baseline approaches which used images as pivots. according to the intrinsic evaluation, our approach resulted in parallel sentences of higher quality than either baseline.