Bilingual Transfer Learning for Online Product Classification (2020)

Abstract: Consumer Price Indices (CPIs) are one of the major statistics produced by Statistical Offices, and of crucial importance to Central Banks. To calculate CPIs, statistical offices collect a large amount of individual prices of goods and services. Nowadays prices of many consumer goods can be obtained online, enabling a much more detailed measurement of inflation rates. One major challenge is to classify the variety of products, from different shops and languages into the given statistical schema consisting of a complex multi-level classification hierarchy - the European Classification of Individual Consumption according to Purpose (ECOICOP) for European countries, since there is no model, mapping or labelled data available. We focus in our analysis on food, beverage and tobacco which account for 74 of the 258 ECOICOP categories and 19 % of the Euro Area inflation basket. In this paper we build a classifier on web scraped, hand-labeled product data from German retailers and test the transfer to French data using cross lingual word embedding. We compare its performance against a classifier trained on the single languages and a classifier with both languages trained jointly. Furthermore, we propose a pipeline to effectively create a data set with balanced labels using transferred predictions and active learning. In addition we test how much data it takes to build a single language classifier from scratch an if there are benefits from multilingual training. Our proposed system reduces the time to complete the task by about two thirds and is already used to support the analysis of inflation.

## introduction consumer price inflation in the euro area is measured by the harmonised index of consumer prices (hicp), which is calculated based on a basket of goods and services. while prices for some goods, like energy prices, are easy to observe, prices for many product groups, like e.g. food, are often collected manually. this survey-based approach is relatively expensive and slow. nowadays we have access to data on prices of individual products sold on the internet. this kind of data can help to improve the quality of the data and monitor it at a higher frequency. there are various initiatives working on the usage of alternative data sources for the calculation of price statistics, but these are mainly countryspecific. examples are the use of scanner data @xcite and web scraped food data @xcite to measure polish inflation, the measurement of the cpi in finland @xcite or forecasting daily cpi in the united kingdom @xcite . the billion prices project @xcite extensively researched this topic and validated the usefulness by backtesting against traditional measured price indices. furthermore, they use online prices also to answer macroeconomic research questions, like the verification of inflation statistics from argentina or the review of effects from changes in us trade policy @xcite . there is an important problem related to online price data, which is rarely addressed but always encountered in the process: the classification of millions of online products into categories. this is crucial for analysing the individual components underlying inflation as required by consumer price indexes. those classification systems are rather complex with hundreds of categories within several layers of hierarchy. usually experts in the specific classification system are needed and the classification cannot be done by people without this expertise. additionally, the product set is not static. during our investigation of scraped web product data, we found out that products are phased out and new products appear frequently. thus over time much more different products with the same or completely different characteristics appear than originally available and a lot of them disappear again, meaning that constant monitoring of the system is necessary. using supervised machine learning methods to automatize a classification task of this type often requires a large amount of labeled training data and labeling by hand is very time consuming as it needs to be done for every language individually. moreover, all the texts (product names, descriptions and categories) have to be represented by numerical features or feature vectors, which leads into the field of natural language processing (nlp). many of the previously mentioned initiatives have neglected a detailed classification or only offer very broad classifications. this makes them less suitable for analysing individual inflation components. furthermore, they do not take into account the fact that product data can be available in different languages, which is especially important for the polyglot european market. in this study we address how contemporary nlp and machine learning techniques can be used to automate the classification of online products, and what kind of effort is needed to build a model from scratch without having labeled training data. the underlying classification system is the european classification of individual consumption according to purpose (ecoicop) (european union, 2016) used by the european statistic offices and the european union. we found the vocabulary of the product texts to be very domain specific and pretrained language models only to cover parts. for this reason we have been working on ways to extend the vocabulary of existing models, which limited the methods we could use. in particular, we look at ways of transferring knowledge contained in a classification model for product data in one language to another using cross-lingual word embeddings. in addition, we propose different neural architectures for monolingual training, bilingual transfer and bilingual training. we also report how much labeled data is needed for a decent model and compare the results of zero-shot bilingual transfer to bilingual training. on the basis of our findings we propose an active learning pipeline to create balanced training data in a target language. the goal is to build a tool-set for a multi-lingual collection of web-scraped product corpora to make product classification accessible in many languages. ## related work although there is interest for detailed inflation monitoring, only a small amount of research exists about classification into the ecoicop or similar schemes for online product data. for example, the billion prices project emphasize the use of big data for inflation measurement @xcite ). they collected web scraped data from large multi-channel retailers in the united states and built a naive bayes classifier on "language-specific, hand-categorized items". to our knowledge there is just a single study on using vector representations of words in the context of coicop classification: @xcite take web scraped clothing data and use a semi-supervised approach to create training labels for their products. they useed various methods to spread existing labels. they found them to be correctly classified in 70% of all cases. text classification. text classification is one of the core nlp tasks with a huge number of applications (e.g., sentiment analysis, spam filtering, and intent detection in chatbots to mention some of the most important ones). specifically, the classification of short texts that do not necessarily consist of a wellformed sentence or sentences is also a much-studied problem @xcite @xcite . state-of-the-art short text classification models typically consist of a deep neural architecture on top of a static word or subword embedding layer. while recurrent neural nets (rnns) using long short-term memory (lstm) @xcite are often the first choice for sequential data like time series or texts, one-dimensional convolutional neural networks (cnns) were found to outperform lstm-based rnn variants in many settings, especially in the case of short or unstructured texts @xcite . transfer learning. using representations learned for a supervised task on a large data set as a basis for building and training a model for a different data set is a frequently used method for addressing data scarcity problems. one of the first models widely used for this kind of transfer learning was the alexnet convolutional image classifier by @xcite . for the transfer the upper layers are cut off while lower layers get frozen and on top of them new layers are trained on the new task. word embeddings are another example where representations trained on one task (to predict a word given its surrounding) are used for many other tasks. originally, an important limitation of pre-trained word embeddings was their limited vocabulary, since they provided representations only for words that occurred in the training corpus. to overcome this limitation, facebook ai research additionally trained their embedding model on certain subwords occurring in the corpus as well @xcite . the resulting framework (fasttext) provides models pre-trained on wikipedia for a large amount of languages. @xcite trained word embeddings jointly for english and german to obtain representations in one common vector space. transferring knowledge from a classifier only trained on english data to a classifier for german data using these cross-lingual word vectors outperformed a simple translation approach. in the following section we use a common vector space for french and german to transfer the ecoicop classification. a very successful approach for cross-lingual zero shot learning was introduced by @xcite . they used state-of-the-art transformer-based language models trained on multiple languages. only the encoder part of the model was transferred, why the decoder was replaced with their own classification layer with impressive results in transferring the learned information between english and french. earlier, @xcite already showed that a single neural machine translation model trained on multiple languages can generalize to some extent, allowing for zero shot prediction on unseen data. ## framework we propose a cnn architecture (cf. figure 2 ) on top of word vector representations exemplified in figure 1 . ## conclusion and further research we investigated in this work how web scraped product data ca be classified in to the ecoicop classification schema and how multilingual transfer learning can be applied to transfer results between languages. we demonstrated that zero-shot learning can be very useful, especially in the early phases when no manually labeled data is available. in the future we want to extend this work to further european languages and explore if some language combinations benefit more than others from the transfer from a pretrained single-language classification model and if a multilingual model generalizes for transfer learning. we found the predicted scores for the categories to be a good proxy for the certainty of the model and we expect that the generic framework introduced here can be reused for many different languages by changing only language-specific parts. this would help to focus the manual classification effort for millions of products that needs to be enhanced on a continuous basis. furthermore, we want to include all ecoicop categories in a generalized manner.