Identifying Complaints from Product Reviews: A Case Study on H indi (2020)

Abstract: Automatic recognition of customer complaints on products or services that they purchase can be crucial for the organisations, multinationals and online retailers since they can exploit this information to fulfil their customers’ expectations including managing and resolving the complaints. Recently, researchers have applied supervised learning strategies to automatically identify users’ complaints expressed in English on Twitter. The downside of these approaches is that they require labeled training data for learning, which is expensive to create. This poses a barrier for them being applied to low-resource languages and domains for which task-specific data is not available. Machine translation (MT) can be used as an alternative to the tools that require such task-specific data. In this work, we use state-of-the-art neural MT (NMT) models for translating Hindi reviews into English and investigate performance of the downstream classification task (complaints identification) on their English translations.

## introduction almost all online retailers allow users to freely express their opinions and thoughts on products via their websites and relevant social media platforms. customers who intend to purchase a product may take purchasing decisions based on the reviews of the product. accordingly, commercial and retail companies consider product reviews as an important source of information, and could exploit this information to build their marketing tools and strategy, and to resolve any issues in relation to the product. this could also benefit users with suggestions on the quality of the products or services that they want to purchase. as for the number of reviews of a product posted by the users, they could range from several hundreds to tens of thousands. e-commerce companies and online retailers want to identify complaints given the reviews of a product for their own benefit. likewise, customers who want to buy a product or service may need such information while avoiding having to consult thousands of reviews about the product. in this context, @xcite identified the relationship between users' purchase intent from their social media forums such as quora foot_0 and yahoo! answers, 2 and wang et al. (2015) investigated the problem of identifying purchase intent with using a list of seed intent-indicators (e.g. 'want to'). @xcite extend the work of @xcite while increasing the coverage of the purchase intent indicators with the distributed vector representation of words using the continuous skip-gram model @xcite . recently, @xcite automatically identified complaints from tweets posted by social media users and potential customers. in @xcite , we conducted a similar study in an attempt to identify complaints from opinionated texts (reviews) about products posted in a low-resource language, hindi, from the the websites of the retail giant amazon india foot_2 and the popular social media platform youtube. 4 for investigating this problem in hindi @xcite by employing a number of human annotators. the process of creating such a data set is not easy; it is not only time-consuming and laborious but also a very expensive task. in this context, @xcite investigated possibility of exploiting mt in a specific nlp task in a language for which dedicated tools are not available due to the scarcity of task-specific training data. as in tebbifakhr et al. (2019) , in this work, we considered hindi, an under-resourced indic language, and investigate whether mt can play a role in complaint identification and eliminate the requirement for complaint identification tools for hindi, which require labeled data for training, which is expensive to create. accordingly, we study the following two scenarios while considering reviews about a variety of products from the the websites of amazon india and youtube expressed in hindi as the test examples in our experiments, namely performance of the classifiers (complaints identifiers) built for english on the english translations of the hindi reviews by (i) our mt systems and (ii) human translators. note that as a part of our investigation, we created a labeled training dataset of english reviews about products posted in amazon, and detail the data creation process and statistics in section 2.2. unlike @xcite who focus on improving a downstream task (i.e. sentiment classification) by controlling translations of an mt system but at the expense of translation quality, we customise our neural mt systems using the standard and commonly-used data augmentation and terminology-aware domain adaptation techniques @xcite @xcite so that the translations produced by the mt systems can retain source-side stylistics property and semantics as much as possible. in other words, in this study, we aim to observe the performance of the english classifiers (complaint identifiers) on the translations of the hindi reviews by the baseline, adapted/customised neural mt systems, and human translators. the remainder of the paper is organised as follows. in section 2, we detail how we created training data for our experiments. sec-complaints-mining-from-hindi-product-reviews tion 3 describes our mt system building and setups. in section 4, we present our experimental methodology for complaint classification. section 5 presents our evaluation results, with some discussion. section 6 concludes and provides avenues for further work. ## dataset creation this section details the creation of training data that has been used in this task. ## the hindi-to-english mt systems our mt systems are transformer models @xcite which were trained using the marian-nmt toolkit. 10 the tokens of the training, evaluation and validation sets are segmented into sub-word units using byte-pair encoding (bpe) @xcite , and bpe is applied individually on the source and target languages. from our experiences @xcite @xcite in the participation in the recent shared translation tasks @xcite @xcite involving low-resource language pairs and domains, we found that the following configuration usually leads to the best results in the low-resource translation settings: (i) the bpe vocabulary size: 6,000, (ii) the sizes of the encoder and decoder layers: 4 and 6, respectively, and (iii) learning-rate: 0.0003. as for the remaining hyperparameters, we followed the recommended best setup from @xcite . the early stopping criterion is based on cross-entropy; however, the final nmt system is selected as per the highest bleu score on the validation set. the beam size for search is set to 6. we make our final nmt model with ensembles of 8 models that are sampled from the training run. for building our baseline models (forward and backward), we used the iit bombay english-hindi parallel corpus 11 (kunchukuttan et al., 2017) that is compiled from a variety of existing sources, e.g. opus 12 @xcite . after applying standard cleaning procedures including applying a language identifier 13 we are left with just over 1.1 million parallel sentence pairs. as for hindi and english monolingual sentences for forward-translation and back-translation, respectively, we sampled them from the ai4bharat-indicnlp corpus @xcite and amazon review dump (cf. section 2.2), respectively. we present the performance of our mt systems in terms of the automatic evaluation metric bleu @xcite . additionally, we performed statistical significance tests using bootstrap resampling methods @xcite . we obtained the bleu scores of our mt systems on the test set, and the scores are reported in we observed that the review texts generally contain terms or product names, and terminology translation is a challenging task in mt @xcite . in order to adapt our best mt system, base+bt+ft, to the task, we adopted the terminology-aware onthe-fly adaption method jooste et al. ( 2020 ); haque et al. (2020c); nayak et al. (2020b); parthasarathy et al. (2020), and mine those sentences from large monolingual datasets that could be beneficial for fine-tuning the original nmt model. as in jooste et al. (2020); haque et al. (2020c); nayak et al. (2020b); parthasarathy et al. ( 2020 ), we first identified terms in the review test set (cf. the bleu scores of the adapted mt system (base+bt+ft+da) on the test set are shown in the last row of ## results and discussion we evaluate the performance our classifiers on the gold-standard test set (cf. for comparison, we also measured the performance of the hindi classifiers that were built on the hindi training data released by @xcite on the original reviews (i.e. the hindi-side of the test set; cf. we clearly see from the scores presented in ## conclusion in this paper, we presented a strategy in which mt can be used to eliminate the requirement for expensive task-specific data creation for low-resource languages or domains. we investigated our strategy on complaint identification from reviews about products posted in hindi. we used state-of-the-art nmt models for translating the hindi reviews into english, and investigate the performance of the english complaint identifiers on the translations of the hindi reviews by the hindi-to-english mt systems. for comparison, we tested the performance of the english classifiers on two setups: (i) english translations of the hindi reviews by the mt systems, and (ii) gold-standard english translations of the hindi reviews. we also compared the performance of the hindi classifiers built on a publicly available hindi review training data set on the original hindi reviews (i.e. the hindi-side of the review test set). in our experiments, we also aimed at preserving source-language stylistic properties and semantics in translation. for this, we applied standard and commonly-used data augmentation techniques and terminology-aware domain adaptation method @xcite @xcite for building our hindi-to-english nmt systems, and used task-specific target-language monolingual data. these strategies were found to be effective in this task. we demonstrated that the nmt systems when customised or trained to translate texts of specific styles (e.g. usergenerated content or reviews) can act as an alternative to those tools that require taskspecific (i.e. complaint identification) training data which are expensive to create. we believe that this work would bring additional value to the social media analytics research and practice given the fact that many task-specific data are available in english only and does not exist in many low-resource and even some high-resource languages. in future, we intend to test our method on different low-resource and high-resource non-english languages. we also plan to investigate this method on different nlp tasks.