Multi-label classification of promotions in digital leaflets using textual and visual information (2020)

Abstract: Product descriptions in e-commerce platforms contain detailed and valuable information about retailers assortment. In particular, coding promotions within digital leaflets are of great interest in e-commerce as they capture the attention of consumers by showing regular promotions for different products. However, this information is embedded into images, making it difficult to extract and process for downstream tasks. In this paper, we present an end-to-end approach that classifies promotions within digital leaflets into their corresponding product categories using both visual and textual information. Our approach can be divided into three key components: 1) region detection, 2) text recognition and 3) text classification. In many cases, a single promotion refers to multiple product categories, so we introduce a multi-label objective in the classification head. We demonstrate the effectiveness of our approach for two separated tasks: 1) image-based detection of the descriptions for each individual promotion and 2) multi-label classification of the product categories using the text from the product descriptions. We train and evaluate our models using a private dataset composed of images from digital leaflets obtained by Nielsen. Results show that we consistently outperform the proposed baseline by a large margin in all the experiments.

## introduction the latest advances in artificial intelligence (ai) have provided new tools to enhance the automation of different recognition problems. we are witnessing a clear trend to merge different domains within ai to obtain better representations for the most complex problems. many recent approaches merge textual and visual information by applying natural language processing (nlp) and computer vision (cv), with the aim of solving problems that involve both text and images @xcite . within this context, structured knowledge extraction from unstructured text is an open problem in the e-commerce literature @xcite . regardless the source of the information (e.g. product websites, product images captured from stores or digital leaflets), it refers to a unified concept that can be denoted as "automated product coding", i.e. the extraction of attribute values of e-commerce products (see fig. 1 ). our present work focuses on the case of knowledge extraction from digital leaflets. most retailers are replacing physical leaflets, that are directly collected from the stores, with digital leaflets that are uploaded in the cloud on the retailers websites. comparing to real-world e-commerce platforms that contain billions of products with detailed descriptions (including ratings and opinions), digital leaflets include concise textual and visual information of promotions that applies to some of the products of the store assortment for a short period of time and thus it has to be updated regularly. the knowledge extraction of digital leaflets is of great interest for the e-commerce business as it impacts not only on several aspects of the consumers behaviour or seasonality, but also modifies relevant attributes of the products periodically, e.g. price or volume. in this paper, we bring for the first time the problem of automated product coding in digital leaflets for e-commerce. in particular, we present an approach to predict the product categories for each of the promotions within the leaflet that serves as a strong baseline for future works. technically, this is a multi-label text classification problem as some promotions can potentially apply to several product categories. for that purpose, we hypothesize that most of the information of the promotions is selfcontained in the products descriptions, so we first detect all these regions within the image that contain textual descriptions and extract the text using optical character recognition (ocr) techniques. in fig. 1 , we show a visual representation of the three key components in the proposed approach: 1) regionbased detection of the promotions description, 2) text recognition and extraction, and 3) multi-label text classification. the building blocks depicted in fig. 1 also relate to the different domains covered in the approach and they can be divided into the following three categories: 1) cv: a region detection architecture based on deep learning and image processing to detect the descriptions of each individual promotion within the digital leaflet, 2) cv+nlp: a text recognition method for extracting the textual information contained into the detected descriptions based on ocr and 3) nlp: a multi-label text classification model based on sub-word text embeddings and a shallow neural network. the main contributions of the paper are three fold: the contents of the paper are structured as follows: related works are described in section 2. the technical proposal presented in this paper is described in section 3. the data used in the evaluations of our proposal, the experiments carried out to validate it and several comparative results are reviewed in section 4. the final conclusions derived from this paper are discussed in section 5. ## related work prior works related to the fusion of nlp and cv have experienced an important growth in the last years due to the advances in deep learning and its influence in both domains. it covers several fields such as text retrieval @xcite , image detection and classification @xcite or automated item coding @xcite . regarding the first step of the system proposed in this paper, approaches based on region detection over images have received an incredible attention in the last few years. the popularization of deep learning jointly with convolutional neural networks (cnn) @xcite has completely changed the traditional paradigm in cv. standard cnns are commonly applied only for image classification. however, r-cnns (r stands for region-based) are focused on object detection, which combines both detection and classification. nowadays, techniques such as faster r-cnn @xcite are broadly extended to localize and classify objects over images. in this method, the regions in the r-cnn are detected by a selective search algorithm based on a region proposal network (rpn). yolo @xcite is also a technique very popularized for object detection which is focused on single shot detection (ssd) @xcite . similar proposals based on r-cnn architectures can be used in our approach to initially detect the regions where the text of the leaflets descriptions is located over the images. following with the second stage for the recognition of the texts contained in descriptions regions, optical character recognition (ocr) is a broad topic covered in the ai community and it aims at extracting text from images, thus working in the intersection between cv and nlp. the most recent approaches use deep learning techniques @xcite to examine images pixel by pixel, looking for shapes that match the character traits. available ocr engines comprise implemented solutions that are open-source and proprietary. calamari @xcite or tesseract @xcite are some of the most effective open-source approaches, with lots of users around the world. however, proprietary solutions such as google ocr foot_0 are currently obtaining better results in text recognition, including support for a larger number of languages. our goal is to apply ocr-based algorithms over the regions previously detected using a r-cnn architecture in order to obtain the product descriptions over the images of leaflets. in the final stage of our described approach, the textual information extracted from the detected regions is classified into their corresponding product categories. the state of the art in short text classification is recently moving to approaches based on dnns (deep neural networks). on the one hand, in @xcite the authors proposed to incorporate sub-word level information to train textual embeddings very efficiently for text classification. on the other hand, the bert architecture described in @xcite also supposed a great milestone in natural language modeling introducing a self-supervised learning strategy that is able to incorporate an architecture based on transformers @xcite leveraging a large corpus for training. the embeddings obtained using bert approaches highly correlate with the linguistic context within a sentence. thus, proposals based on sub-word level information are very competitive compared to bert models on those cases where we have unstructured textual information and probably ocr errors, such as the descriptions processed in most of the leaflets. it must be also noted that although bert is focused on standard text processing, there are derived approaches that are also diving into text processing associated with images, such as vilbert @xcite or vl-bert @xcite . the difference is that these recent approaches are not directly applied to classification over text contained in images, they are used for tasks that involve images and related external text, such as vqa (visual question answering) @xcite . ## our proposal for digital leaflets categorization the leaflets categorization proposal presented in this paper is focused on the prediction of multiple product categories from images such as the depicted in fig. 1 , which is showing a part of a catalog representing specific products. the solution can be divided into the following three main parts: in this section, we introduce these three main components of the proposed approach, jointly with the whole leaflets categorization pipeline that combines them to obtain the final output. ## experiments and results with the aim of validating the proposed approach in a specific leaflets categorization use case, we prepared a set of experiments with leaflets data captured by nielsen. in this section, we describe the experimental setup: the datasets, the hyperparameters used for training and the comparative results with different data and approaches.