Conceptual Similarity for Subjective Tags (2022)

Abstract: Tagging in the context of online resources is a fundamental addition to search systems. Tags assist with the indexing, management, and retrieval of online products and services to answer complex user queries. Traditional methods of matching user queries with tags either rely on cosine similarity, or employ semantic similarity models that fail to recognize conceptual connections between tags, e.g. ambiance and music. In this work, we focus on subjective tags which characterize subjective aspects of a product or service. We propose conceptual similarity to leverage conceptual awareness when assessing similarity between tags. We also provide a simple cost-effective pipeline to automatically generate data in order to train the conceptual similarity model. We show that our pipeline generates high-quality datasets, and evaluate the similarity model both systematically and on a downstream application. Experiments show that conceptual similarity outperforms existing work when using subjective tags.

## introduction as products and services proliferated the internet in recent years, tagging came into prominence to facilitate the consumption of online information @xcite . tagging is the practice of assigning labels and keywords to online resources. it plays a pivotal role in the indexing, management and retrieval of factual information. on the other hand, recent years have witnessed a major shift in people's expectations when searching online @xcite . beside the factual data such as a restaurant's cuisine type or a camera's resolution, the search trend evolved to be more experiential @xcite . common search queries include attributes such as delicious food for restaurants or long-lasting battery for cameras. previous work @xcite called this new set of attributes as subjective tags because they are short phrases that hint towards the subjective quality of products and services. subjective tags are particularly useful in enhancing online experiential search. in this context, users who are seeking subjective experiences, include sets of tags they care about in their queries, and it is the search system's responsibility to fetch products and/or services that have been previously described with matching tags. deciding whether two given subjective tags match or not implies using a similarity measure, for which cosine similarity remains a convenient, yet arbitrary default @xcite @xcite . recent search systems such as opinedb @xcite or searchlens @xcite rely mostly on cosine similarity when it comes to comparing tag-like short phrases, since it is easy to use and provides simple geometric interpretations @xcite . however, recent studies @xcite argue that this interpretability becomes fogged when dealing with sentences or phrases, and cosine similarity suffers from severe limitations when used to compare multi-word textual inputs. a lot of research has been directed toward proposing supervised methods for textual similarity, spanning a diverse set of paradigms, e.g. siamese networks @xcite , aggregation-matching models @xcite @xcite , or the recent cross-sentence attention paradigm which was made possible by the advent of the transformer architecture @xcite . although these models work fairly well on syntactically-correct sentences @xcite , they lack effectiveness when used with shorter-spanned phrases such as subjective tags. a reason behind this is that subjective tags do not share the same structure of full sentences and hence require different treatment. as will be discussed later in this paper, our experiments confirm this limitation. a second drawback is that current similarity models are not explicitly trained to recognize conceptual similarities between the compared textual entities (e.g., meal and pizza share the concept of food; or background music and lighting share the concept of ambiance). therefore, all conceptual reasoning is disregarded. in this work, we compel our own similarity model to encode more conceptual relationships as provided by a human (whom we call the designer) and further expanded by popular knowledge bases such as wordnet @xcite or conceptnet @xcite . to illustrate the importance of capturing conceptual similarities between subjective tags, suppose a user searches for a restaurant serving delicious meals. a search system should be able to suggest a restaurant which has been tagged with tasty chicken wings among its search results, because meal and chicken wings share the same concept (that of food) even though meal and chicken wings are not semantically similar. as a result, traditional semantic similarity models @xcite @xcite usually fail to meet this expectation and provide low similarity scores for the tags in the example. the same reasoning applies to other subjective tags, like high-autonomy camera and long-lasting battery, or romantic ambiance and low-beat music bar. aiming to solve the aforementioned drawbacks, we propose a new similarity model that focuses on learning and then using conceptual relationships as reflected in the training data. given the new nature of subjective tags @xcite , we are not aware of the existence of datasets that suit our needs. besides, manually annotating data is expensive, and extending to other application domains (e.g. from restaurants to electronics) usually necessitates re-annotating from scratch. therefore, the main contribution of this paper is a pipeline to automatically generate large synthetic datasets for the conceptual similarity task. first, we prompt the dataset designer to provide seed words for the concepts she needs her conceptual similarity model to learn about. second, we exploit the simple structure of subjective tags @xcite to expand the seeds with conceptually related terms using knowledge bases, or the implicit knowledge encoded in existing language models to automatically generate large training data. our second contribution is the similarity model itself. capitalizing on the latest advances in se-mantic similarity research @xcite @xcite , we propose a new similarity model by combining insights from aggregation-matching and cross-sentence attention paradigms. we show that conceptual similarity is better than cosine similarity with a margin of 17.42% in terms of pearson correlation, or bertbased similarity models through systematic evaluations. we also plug different similarity models into a tag-based search system and show that conceptual similarity outperforms them all. also, we evaluate the quality of the automatically generated dataset through various experiments. we release our code and data in github foot_0 . ## pipeline to generate training datasets borrowing from the aspect-based sentiment analysis literature @xcite , we define a subjective tag as the concatenation of an aspect term with an opinion term. the aspect term designates the component or the feature being described and the opinion term characterizes this feature. for example, delicious food is a subjective tag wherein food is the aspect while delicious is the opinion. this definition is sufficiently expressive to allow a wide range of subjective tags such as romantic ambiance, clean hotel rooms, long-lasting battery, great camera or amiable dentist. specific to this work, we define a concept as a set of aspect terms conceptually related to each other. for example, the concept of food can be described with the following set of terms: {food, plates, dishes, pizza, chicken wings, meal, pasta} while the concept of ambiance can be defined with {ambiance, atmosphere, lighting, background music, dance floor}. the goal of conceptual similarity is to consider the aspects belonging to the same concept as similar when described with similar opinions. we cast conceptual similarity as a binary classification problem, where the positive label denotes similarity. these specifications enable automatic generation of high-quality labeled datasets for conceptual similarity of subjective tags, with minimal costs. to do so, the dataset designer provides a list of concepts. we then leverage seed expansion techniques to generate the dataset, through the pipeline illustrated in figure 1 . in the following, we describe each step of the pipeline in detail. ## conceptual similarity model in this section, we present our approach to compute conceptual similarity for a pair of subjective tags. following guidelines from the aggregationmatching paradigm @xcite , our model encodes explicit interactions between tags, e.g. whether the tags correspond to the same concept; whether they use the same opinions but with different aspects; whether the choice of words in the tags is similar but the tags themselves are not. to this end, we propose a novel bilateral matching model which automatically encodes such interactions and relationships before making a similarity decision. given two subjective tags t 1 and t 2 , this model estimates their similarity by computing their probability of being perfectly similar p (@xmath0, t 2 ). figure 3 illustrates the different layers of this model. we begin by feeding t 1 and t 2 into bert @xcite . this serves two purposes: first, we get word embeddings for each word in the tags; second, we have a cls vector that captures the relationship between t 1 and t 2 as a vector. given bert embeddings [u 1 , ..., u m ] and [v 1 , ..., v n ], we utilize mean pooling to obtain fixed-sized embeddings for each tag (u all and v all ). the next layer in the network matches each word embedding of one tag with all the word embeddings of the other tag. the matching is done in two directions (hence the bilateral aspect): (1) we match each u i with v all to compare each word u i in t 1 with all the words in t 2 , and encode their relationship. (2) we match each v i with u all to do the same in the reverse direction. the matching function we use is the elementwise multiplication which has long been used in the nlp community as a proxy for similarity. thus, we use it to match word embeddings of t 1 and t 2 . after the matching layer, we aggregate [u ′ 1 , ..., u ′ m ] and [v ′ 1 , ..., v ′ n ] to obtain fixed-length vectors for each tag via bidirectional lstm (bilstm) layers @xcite , taking the last hidden states as tag embeddings u and v. at this step, we have encoded the relationship between t 1 and t 2 using two different paradigms: (1) aggregation-matching through the use of elementwise multiplication for matching and bilstm for aggregation (vectors u and v), and (2) the crosssentence attention paradigm through cls vector, because bert uses self-attention @xcite to compute its vectors. we concatenate u, v and cls and feed it to a classification head (ffnn layer) to estimate similarity. ## experiments we use restaurants as the test domain. we consider nine concepts that we use to automatically generate the training dataset: food, service, price, atmosphere, location, cleaning, environment, menu and parking. each concept consists of one set of aspect terms, and two to three sets of different opinion terms. the choice of concepts, and seed words for aspects and opinions was inspired by previous work @xcite who conducted surveys and qualitative experiments on many restaurant-seeking participants, and identified the most important factors taken into account by these same participants in their decision-making process for choosing a restaurant. the full list of concepts and their seeds is in section a.3, while the hyperparameter details for the similarity model are in section a.1. in the following, we first compare conceptual similarity to various baselines. next, we evaluate the quality of the automatically generated dataset. finally, we assess the practical value of conceptual similarity by measuring its impact on a downstream search system proposed by @xcite that uses subjective tags. ## conclusion in this work, we propose conceptual similarity for subjective tags. we also propose a methodology to automatically generate training datasets for conceptual similarity with minimal effort given a domain and a set of concepts. unlike traditional semantic similarity, our model is trained with conceptual signals as reflected in the generated dataset. intrinsic and extrinsic experiments demonstrate the superiority of our approach on subjective tags. on the other hand, we acknowledge the following limitations. although the method is independent from the application domain, we constrained our evaluations to the restaurants domain for reasons related to unavailability of test data. so we were forced to create our own test benchmark by asking three participants to give ground truth labels for 500 pairs of subjective tags. this may seem small-scale, which risks putting into question the conclusions regarding the superiority of our similarity approach. however, the extrinsic experiment that we conduct by using relatively larger crowdsourced data shows that our approach is efficient and outperforms other similarity models, which assuages our concern. as future work, we plan to apply our methods on other domains, e.g. hotels, or electronics. in this paper, we build the whole argument of our contributions against the blind use of cosine similarity in tag-based search systems, and to replace it with our newly proposed conceptual similarity. however, we employ bert and lstms in our model which incur a much higher computational cost than cosine similarity. the adoption of our model in practice depends on whether efficiency is a major concern in the downstream search application, i.e. whether a poor search inflicts major negative consequences in critical domains such as finances or regulations. it also depends on the underlying infrastructure into which conceptual similarity will be deployed, e.g. are there any gpus in use? is memory space enough to hold bert and lstms? so whether to adopt our contributions in practice is a compromise between cost and efficiency.