Combining semantic search and twin product classification for recognition of purchasable items in voice shopping (2021)

Abstract: The accuracy of an online shopping system via voice commands is particularly important and may have a great impact on customer trust. This paper focuses on the problem of detecting if an utterance contains actual and purchasable products, thus referring to a shopping-related intent in a typical Spoken Language Understanding architecture consist- ing of an intent classifier and a slot detec- tor. Searching through billions of products to check if a detected slot is a purchasable item is prohibitively expensive. To overcome this problem, we present a framework that (1) uses a retrieval module that returns the most rele- vant products with respect to the detected slot, and (2) combines it with a twin network that decides if the detected slot is indeed a pur- chasable item or not. Through various exper- iments, we show that this architecture outper- forms a typical slot detector approach, with a gain of +81% in accuracy and +41% in F1 score.

## introduction spoken language understanding (slu) constitutes the backbone of voice-controlled devices such as amazon alexa or google assistant. slu is typically divided into two sub-tasks, intent classification (ic) and slot filling (sf). ic determines the user's intent and slot filling (sf) extracts the semantic constituents. for instance, if a user says buy batteries, ic should classify the intent as buyitem and sf should label the utterance as buy|other batteries|itemn ame. the slot in this example is itemname and the slot value is batteries. this constitutes a way to map the utterance on a semantic space. throughout this paper the term carrier phrase denotes all tokens that are not part of the slot, e.g. buy in the previous example. in typical slu architectures, the final confidence score is obtained through multiplication of the two scores from the ic and the sf model. for each utterance, the model produces n-best hypotheses ranked by the final confidence score. one hypothesis consists of an intent and slot labels. problem statement -the complexity for both ic and sf tasks is continuously increasing due to the growth of voice-controlled device functionalities. investigations have revealed that the model tends to put more weight on the carrier phrase than the slot value. this bias implies that particular slot labels are assigned based on the carrier phrase rather than the slot value. since with a rising amount of functionalities carrier phrases can be identical across functionalities and the correct class then solely depends on the slot value. label confusions are particularly observed for shopping intents, e. g. buyitem. for example, how much is {toilet paper} / was kostet {klopapier} is a request that falls into the realm of shopping functionalities because the user could actually purchase this item through the device. in contrast, if the user asks how much is {ronaldo} worth / was kostet {ronaldo} the request needs to be processed by qa functionalities. note that in german the two sentences have an identical pattern how much is {slot value} / wie viel kostet {slot value} while the carrier phrase slightly differs in english. shopping functionalities in a voice-controlled device have a particular importance due to their high potential impact on customer trust and experience. if a user has the feeling they could accidentally buy something, this may reduce trust in the system and drive customers away. on the contrary, not recognizing shopping intents is equally harmful as users will not shop with the device anymore. contribution and approach -in this paper, we focus on solving the problem of label confusions, where detected slots are mistaken as itemname, hence are routed to shopping (false accept) and the other way around, when an item name is not identified as itemname, hence are not correctly handled as shopping (false reject). in order to correctly determine if an item is purchasable or nonpurchasable, one needs to match the itemname candidates to products sold on the respective platform. if the item is similar (enough) to products in the product catalog, it is purchasable. a straightforward way of solving this problem is to search through the whole shopping catalog to identify if the detected slot is found within the catalog or not through simple string or substring matching. however, this approach has two main drawbacks: first, it is extremely expensive to search through the full catalog for all slot candidates and is therefore prohibitive in real applications. second, the approach is not precise and does not utilize meaning at all. take the non-purchasable itemname candidate lamborghini as an example. a lamborghini cannot be purchased on e-commerce platforms and is therefore non-purchasable in this context. string or substring matching however returns matches like lego lamborghini, toy lamborghini from the product catalog and as such would indicate that a lamborghini could be bought through the device. instead, we need an approach that is able to detect the difference between a lego lamborghini and a lamborghini. to improve the accuracy of the detected slots with respect to the itemname entities, we propose to combine a retrieval module with a twin product classifier. instead of searching through all billions of possible products, this approach uses a twin network @xcite to compare the candidate slot only with the semantically most similar products returned by the retrieval module from the product catalog. by looking at negative (non-purchasable) and positive (purchasable) examples of itemname candidates and their matched (retrieved) items from the product catalog, the twin network learns to pull together pairs that result in a correct match (positive) and push apart pairs that are not a match (negative). the twin product classifier can be used as a signal to adjust the final confidence scores for the n-best hypotheses from ic-sf. this framework overcomes the problem of high numbers of false accept and false reject for shopping functionalities in voice-controlled devices, while taking into account the efficiency and speed requirement for a real world application. ## related work pre-trained transformer @xcite models have proven successful in many language applications including spoken language understanding (slu) @xcite @xcite . transformers are typically used to create an embedding of input tokens that is then fed to a downstream task. however, as explained in more detail by @xcite , transformer models such as bert are unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. @xcite therefore proposed sentence-bert (sbert). sbert is a modification of the pretrained bert network based on a twin network to derive semantically meaningful sentence embeddings that can be compared via cosine similarly. in e-commerce, product retrieval plays a key role and is a well-researched topic @xcite ). e-commerce systems typically perform two steps: a retrieval and a ranking step. in the retrieval step relevant products are retrieved from the product catalog. in the second step, the retrieved products are ranked by relevance with respect to the query. semantic search @xcite @xcite is state-of-the-art for product retrieval and replaced previously prevalent keyword matching @xcite . in this paper we propose a product retrieval stage in the context of slu to retrieve potential matches for tokens labelled as itemname by the sf model (itemname candidate) from a product catalog. we then replace the ranking stage with a similaritybased classification to identify if the itemname candidate is indeed a purchasable item or not. the goal of this approach is to reduce false accepts and false rejects for voice commerce functionalities as explained in section 1. ## twin product classifier in order to solve the problem of deciding if an item-name candidate is purchasable or non-purchasable, we leverage information from the product catalog and employ a twin network to differentiate between these two categories. figure 1 illustrates the general architecture of the proposed approach, which consists of two main components: (1) the catalog retrieval module and (2) the twin product classifier. in the retrieval module, the detected itemname candidate will be matched with similar items indexed from the product catalog in order to find the best matching items with respect to the candidate. after that, we employ a twin network architecture @xcite and pass the original utterance on the left side and the set of matching items on the right side. instead of learning to classify the inputs, twin networks aim at learning to differentiate between two inputs, i.e., to learn the similarity between them and to identify pairs that don't match. the similarity between these two parts (the original utterance vs. the matching items) will be computed. after that, the twin network is trained using a contrastive loss to learn to differentiate between pairs of utterances and matching catalog items to finally decide if the given item name is purchasable, i.e., could be matched with items from the product catalog. let u be an input utterance and c u be the item-name candidate that this utterance contains. our task is to decide if u should be classified as a shopping intent or not based on checking if c u is a purchasable product. note that the candidate c u is provided by the n-best hypotheses from the ic-sf architecture described in the previous section. from the product catalog, we retrieve a set of products @xmath0, p 2 , .., p k } that are semantically most similar to c u , with @xmath1. u and p are passed through the twin network, which yields two embeddings e u and e p respectively. we define a pair of utterance u and matched products p to be positive if the item-name candidate c u contained in u can be potentially found in the product catalog p and negative if c u is a non-purchasable item. once p is retrieved from the catalog, we compute the distance between e u and e p using the euclidean distance and the contrastive loss l: n is the embedding size and τ the margin parameter which determines the minimum distance a pair should have in order to be classified as negative. the final loss is the sum of both positive and negative loss. contrastive loss has been shown to be very effective for training twin networks @xcite , which pulls together similar pairs and push dissimilar pairs apart. the gradient of the constrastive loss is computed as: in practice, to accelerate the training process and improve the performance, we employ online contrastive loss, which selects only hard positive and hard negative samples in each mini-batch. in particular, the distance between each pair is computed and the loss is only added for pairs with the smallest or biggest distances. with the aim of finding the best set of matching products p for an itemname candidate c u , we argue that the top products returned by fast semantic search provide essential information to the twin network to learn the similarity between c u and p for purchasable items and generalize on unseen items. keyword exact match search is usually faster using a reverse indexing system, but is limited since it cannot find items that are differently spelled, written or semantically close. we use faiss @xcite , a library that has shown to be fast and efficient in similarity search for billion-scale data sets. faiss takes embedded catalog vectors as inputs and starts the indexing process, which involves clustering the data into different clusters represented by their centroids -which are used as inverted file or index. when a query vector comes in, the most suitable cluster found based on its similarity with the centroids is returned together with the top k nearest items. faiss is implemented for running both in cpu and gpu, supporting single and multi-gpus together with batch processing. the encoding and indexing of p is done once completely offline and could be stored in memory for instant ram search of incoming itemname candidates c u . integration of the twin product classifier into the slu system finally, the integration of the twin product classifier in the whole slu architecture is displayed in figure 2 . it illustrates the previous slu architecture (ic-sf) on the left and the added twin product classifier on the right. the ic-sf component illustrated on the left side is an independent part that one can integrate together with the twin product classifier proposed here. the flow would be as follows: the utterance is first encoded using character, positional and catalog embeddings. the catalog embedding is a fixed-size real vector for each token that indicates if the token or a substring of the token is present in one of the catalogs. note that the product catalog used for generating the catalog embedding is a superset of the product catalog used in the twin product classifier (see next section 4.2). this first-stage embedding is then fed through a tinybert encoder block. the resulting encoding is then passed to the respective decoders, the sf and the ic decoder. since the model as is has difficulties to distinguish between purchasable and non-purchasable items, we pass the utterances with itemname candidates in the n-best hypothesis through the twin product classifier. dependent on the classifier feedback, the shopping ic-sf score can be adjusted accordingly. ## conclusion we have presented an architecture with a retrieval module and a twin product classification module to verify if a detected itemname slot from the sf model contains a purchasable item or not. the experimental results have shown the effectiveness of the framework to increase accuracy by +81% for purchasable item detection. for the retrieval module, we have experimented with different numbers of matching products returned by semantic search. we showed that using the top five most relevant product names yields the best results. moreover, adding the whole utterance in the twin network and using an online contrastive loss function resulted in the best performance. this approach can be lever-