Does Named Entity Recognition Truly Not Scale Up to Real-world Product Attribute Extraction? (2023)

Abstract: The key challenge in the attribute-value extraction (AVE) task from e-commerce sites is the scalability to diverse attributes for a large number of products in real-world e-commerce sites. To make AVE scalable to diverse attributes, recent researchers adopted a question-answering (QA)-based approach that additionally inputs the target attribute as a query to extract its values, and confirmed its advantage over a classical approach based on named-entity recognition (NER) on real-word e-commerce datasets. In this study, we argue the scalability of the NER-based approach compared to the QA-based approach, since researchers have compared BERT-based QA-based models to only a weak BiLSTM-based NER baseline trained from scratch in terms of only accuracy on datasets designed to evaluate the QA-based approach. Experimental results using a publicly available real-word dataset revealed that, under a fair setting, BERT-based NER models rival BERT-based QA models in terms of the accuracy, and their inference is faster than the QA model that processes the same product text several times to handle multiple target attributes.

## introduction to serve better product search and recommendation to customers on e-commerce sites, industry researchers have studied attribute value extraction (ave) to organize hundreads of millions of products in terms of their attribute values. in the literature, ave has been formalized as sequence tagging similar to named entity recognition (ner), which recognizes attribute values in the given product text while classifying them to corresponding attributes defined by an e-commerce site (figure 1 ) @xcite @xcite @xcite @xcite @xcite . when we apply ner-based sequence tagging to ave, a larger number of classes (attributes), which can exceed a thousand, poses a challenge. to make ave scalable to thousands of attributes, @xcite have proposed models based on question-answering (qa) to reduce the number of classes by additionally inputting the target attribute and extract only values for that attribute. they reported that the ner-based model, open-tag @xcite performed poorly on rare attributes due to a data sparseness problem and thus did not scale to the diverse attributes. following this study, recent researchers focus on the qa-based approach @xcite @xcite . in this study, we re-evaluate the scalability of the ner-based approach to real-world ave against the qa-based approach in a more fair setting, in terms of efficiency in inference as well as accuracy. in the above comparison @xcite , open-tag is based on a bidirectional lstm @xcite trained from scratch (figure 2 (a) ), whereas the qa-based approaches leverage a pre-trained bert, which remedies the data sparseness problem. meanwhile, the ner-based approaches may classify recognized values as irrelevant attributes and have issues in recognizing overlapping values @xcite , whereas the qa-based approaches bypass these issues by explicitly giving a single target attribute. we should also consider the scalability to a number of products on e-commerce sites, since the ave model will be applied to hundreds of millions of product text on major e-commerce sites. the ner-based the common input to the models is text @xmath0, . . . , t n }. ⊕ represents the concatenation operation. opentag @xcite and bert-ner/crf @xcite introduce a set of chunk tags for each attribute (e.g., b a ). meanwhile, su opentag @xcite and bert-qa @xcite take a target attribute @xmath1, . . . , a m } as an additional input; su opentag and bert-qa thereby predicts a single set of chunk tags and starting and ending positions, respectively, to extract a value corresponding to the given attribute. in our experiments, to enable bert-qa to extract multiple values for a given attribute, we replace a decoder part in the model with a feed forward layer used in (b). our contribution is as follows. traditionally, most previous studies formulated ave as a sequence tagging problem and adapted ner techniques @xcite @xcite @xcite @xcite @xcite @xcite . these studies introduce a set of chunk tags (e.g., bio tags) for each attribute and classify each token in text into one of the chunk tags. therefore, ner-based models can extract values for multiple attributes at the same time. however, since the number of attributes in real-world ave easily exceeds a thousand @xcite , the models are required to perform a large-scale multiclass classification at the token level. to address a large number of attributes in the ave task, recent studies @xcite @xcite adopted a qa-based approach (e.g., figures 2 (c ) and (d)). these qa-based approaches take an attribute as query and a product title as context, and extract attribute values from the context as answer for the query. by taking attributes as the input, qa-based models achieved the best performance on publicly available ave datasets @xcite . on the other hand, unlike ner-based models, qa-based models cannot extract values for multiple attributes at the same time. this is because the models jointly encode a given title and attribute, and it is necessary to perform extraction multiple times when there are values for multiple attributes in the title. hence, the qa-based models are more time-consuming than ner-based models, which incurs a critical issue in business contexts. previous studies @xcite @xcite reported that ner-based models did not scale up to large-sized attributes in ave through the evaluation of opentag @xcite , which was referred to as the state-ofthe-art ner-based model. however, since open-tag relies on bidirectional lstms @xcite and glove @xcite , it is debatable whether ner-based models are truly unscalable, as large-scale pre-trained language models such as bert @xcite have become the de-facto standard as a text encoder. although @xcite verified the performance of bert-based ner models using their own dataset consisting of 12 attributes, the size of attributes is far from the ave task in the real-world scenario. this paper is the first work that reports the performance of bert-based ner models on a publicly available real-world dataset for ave. ## attribute value extraction we formalize ave as a sequence labeling problem. let a be the set of all possible attributes in training data and y be the tag set containing all the tags. if we choose bio as our chunk tag scheme, then @xmath2, i}}∪ o}. given a product data (text) @xmath3, t 2 , . . . , t n } where n is the number of tokens in t, the model is trained to return @xmath4, y 2 , . . . , y n } where y @xmath5. in short, the model performs multiclass classificaition over each token. in the inference, attributes and values are decoded from the sequence of predicted tags. in what follows, we describe bert @xcite and bert-based ner models. ## experiments we evaluate bert-based ner models on a publicly available real-world ave dataset, namely the mave dataset @xcite . 1 the statistics of the dataset are listed in all attributes to evaluate the capability of scaling up to large-sized attributes, we evaluate the models on all attributes in the dataset. selected attributes to demonstrate the performance on individual attributes, we evaluate the models on a set of selected attributes; five head attributes that contain a large number of attributevalue annotations in the dataset and five tail attributes that have very few examples in the dataset. those attributes were selected by @xcite . ## conclusions in this study, we have revisited the ner-based approach to attribute-value extraction (ave) from e-commerce sites, and evaluated the scalability of bert-based ner models on the ave task. we performed experiments using a publicly available real-world dataset and confirmed that even nerbased models scaled up to large-sized attributes. these results showed that experiments with open-tag are insufficient to verify the scalability and performance of ner-based models in real-world ave. we observed that the bert-based ner model rivals the bert-based qa model in terms of accuracy for test examples in which the model does not predict attributes other than the target attributes and examples do not include overlapping values for multiple attributes; these issues are bypassed in the qa-based approach by explicitly giving a single target attribute for extraction. even in the nerbased approach, limiting the output tag space to the target attributes will remedy the first issue, and the use of nested ner models (surveyed in @xcite ) will remedy the second issue. moreover, qa models are more time-consuming since the models must be applied to the same product text for each target attribute (for thousands of attributes when a comprehensive attribute taxonomy narrows down the candidates). we thus conclude that the ner-based models still can be a practical solution to ave, and worth a target for future research.