The Multilingual A mazon Reviews Corpus (2020)

Abstract: We present the Multilingual Amazon Reviews Corpus (MARC), a large-scale collection of Amazon reviews for multilingual text classification. The corpus contains reviews in English, Japanese, German, French, Spanish, and Chinese, which were collected between 2015 and 2019. Each record in the dataset contains the review text, the review title, the star rating, an anonymized reviewer ID, an anonymized product ID, and the coarse-grained product category (e.g., ‘books’, ‘appliances’, etc.) The corpus is balanced across the 5 possible star ratings, so each rating constitutes 20% of the reviews in each language. For each language, there are 200,000, 5,000, and 5,000 reviews in the training, development, and test sets, respectively. We report baseline results for supervised text classification and zero-shot cross-lingual transfer learning by fine-tuning a multilingual BERT model on reviews data. We propose the use of mean absolute error (MAE) instead of classification accuracy for this task, since MAE accounts for the ordinal nature of the ratings.

## introduction text classification is one of the fundamental tasks in natural language processing, and research in this area has been accelerated by the abundance of corpora across different domains (e.g., twitter sentiment @xcite , movie ratings @xcite , textual entailment @xcite , restaurant reviews (yelp inc., 2019), among many others). the construction of multilingual classification systems which handle inputs from different languages has been studied extensively in previous work (e.g., @xcite . more recently, researchers have observed 'zero-shot' cross-lingual behavior @xcite where { "review_id": "en_0000258", "reviewer_id": "reviewer_en_0010355", "product_id": "product_en_0000097", "language": "en", "stars": 5, "review_title": "salad spinner", "review_body": "perfect for herbs and leafy vegetables!", "product_category": "kitchen" } classification performance in one language can be transferred to the same task in another, without target language supervision, as long as the encoder was pretrained on a machine translation task. in addition, contextual embeddings have shown unexpected cross-lingual behavior in classification, ner, and dependency parsing tasks @xcite @xcite . as with all other areas in nlp, progress in multilingual research relies on the availability of highquality data. however, large-scale multilingual text classification datasets are surprisingly rare, and existing multilingual datasets have some notable deficiencies. the proprietary reuters rcv1 @xcite and rcv2 @xcite corpora and its derivatives like mldoc @xcite are relatively small; in rcv2, each language has ∼37,000 training examples on average, and the smallest language only has 1,794 examples. rcv1 and 2 are not easily accessible; a researcher who wishes to acquire the data would need to work with an organization that has obtained legal approval from reuters ltd. the xnli dataset @xcite was designed for evaluating zero-shot cross-lingual transfer and does not contain training data for nonenglish languages. the yelp corpus (yelp inc., 2019) contains reviews from international marketplaces, but the reviews from each marketplace can be written in multiple languages and the language identity is not provided. furthermore, the yelp corpus itself is refreshed from time to time, and previous versions are not made available for download, which affects the reproducibility of published results. several versions of the amazon reviews corpus exist today. neither the version from @xcite nor amazon inc. (2015) provide training, development, and test splits, and neither version focuses on the multilingual aspect of the reviews. @xcite provide amazon reviews in 4 languages (i.e., 2,000 training and test reviews, along with a variable number of unlabeled reviews), but the dataset is small by modern standards. we address many of the above-mentioned limitations by releasing a subset of amazon reviews specifically tailored for the task of multilingual text classification: the multilingual amazon reviews corpus (marc) can be found at https://registry . the dataset description, code snippets, and license agreement can be retrieved at https: //docs.opendata.aws/amazon-reviews-ml/ readme.html. ## baseline results in and y i , ŷi ∈ {1, 2, 3, 4, 5} are the true star rating and the predicted rating for the i-th review respectively. all of our baseline models are initialized with the cased multilingual bert (mbert) base model @xcite , which has 110m parameters. note that the star ratings for each review are ordinal, and a 2-star prediction for a 5-star review should be penalized more heavily than a 4-star prediction for a 5-star review. however, previous work on amazon reviews classification (e.g., @xcite used the classification accuracy as the primary metric, which ignores the ordinal nature of the labels. we use mae in our baselines as the primary metric instead. we also report the classification accuracy for completeness ( ## conclusion we present a curated subset of amazon reviews specifically designed to aid research in multilingual text classification. to the best of our knowledge, this is the largest public benchmark dataset for the training and evaluation of multilingual text classification models. with this work, we systematically address various gaps that we identified in existing multilingual corpora: we apply careful sampling, filtering, and text processing to the documents to minimize noise in the dataset, and we provide a large number of samples for training models in six languages with well-defined training, development, and test splits. we discuss the data preparation steps, analyze the distribution of the important characteristics of the corpus, and present baseline results for supervised and zero-shot cross-lingual text classification. with these contributions, we hope that this corpus will be an important resource to the research community.