Commonsense Knowledge Salience Evaluation with a Benchmark Dataset in E -commerce (2022)

Abstract: In e-commerce, the salience of commonsense knowledge (CSK) is beneficial for widespread applications such as product search and recommendation. For example, when users search for “running” in e-commerce, they would like to find products highly related to running, such as “running shoes” rather than “shoes”. Nevertheless, many existing CSK collections rank statements solely by confidence scores, and there is no information about which ones are salient from a human perspective. In this work, we define the task of supervised salience evaluation, where given a CSK triple, the model is required to learn whether the triple is salient or not. In addition to formulating the new task, we also release a new Benchmark dataset of Salience Evaluation in E-commerce (BSEE) and hope to promote related research on commonsense knowledge salience evaluation. We conduct experiments in the dataset with several representative baseline models. The experimental results show that salience evaluation is a hard task where models perform poorly on our evaluation set. We further propose a simple but effective approach, PMI-tuning, which shows promise for solving this novel problem. Code is available in https://github.com/OpenBGBenchmark/OpenBG-CSK.

## introduction commonsense knowledge (csk) reflects our natural understanding of the world and human behaviors, which are shared by all humans. in ecommerce, previous study @xcite have provided large-scale commonsense knowledge in e-commerce. however, existing csk collections @xcite @xcite rank statements solely by confidence scores, and there is no information about which ones are salient from a human perspective. intuitively, the salience of commonsense knowledge is important and cannot be ignored. to be figure 1 : an excerpt from an e-commerce network of commonsense knowledge. although all the commonsense knowledge is plausible, only the knowledge with a green face is salient, which is beneficial for product search and recommendation. specific, salience reflects that a property is characteristic of the concept in the sense that most humans would list it as a key trait of the concept. for example, lions hunt in packs, bicycles have two wheels, and rap songs have interesting lyrics and beats (but no real melody). in e-commerce, as shown in figure 1 , "running" requires "running shoes" is salient (by most people), while "running" requires "shoes" is not a salient commonsense because for users searching "running" in e-commerce, they would like to find products highly related to running, such as running shoes rather than shoes @xcite . note that salience captures if the triple is characteristic for the concept, salient common sense can help ai agents understand users' behaviors more explicitly. nevertheless, there are few resources and methods regarding commonsense knowledge salience evaluation. to this end, we release a commonsense dataset in e-commerce and present a new shared task of commonsense knowledge salience evaluation. unlike previous works, which mostly focus on plausibility, we aim at salience evaluation of every triple in the commonsense knowledge graph. specifically, we invite annotators to manually annotate the salience for commonsense assertions. as salience is highly subjective, it is hard to annotate directly. hence we interpret the connotation of salience in terms of necessity and sufficiency and provide annotators with several options standing for the specific grades. to evaluate the salience evaluation ability of machine learning models, we conduct experiments with several representative baseline models based on our benchmark data. the experimental results show that salience evaluation is a challenging task where models perform poorly on our evaluation set and far below human performance. furthermore, we propose a novel plm-based model, pmi-tuning, to solve this kind of problem. we introduce the technique of pointwise mutual information (pmi) to estimate the strength of salience. for calculating pmi, we utilize the masked language model to approximate the probabilities of the subject and object. we manually define the prompt template for each predicate type and introduce soft prompts based on the idea of prompt tuning in order to probe more focused knowledge from the language model. ## task design the task requires recognizing, given a commonsense triple, whether it is salient from the human perspective. more concretely, the applied notion of salience is defined as the strength of relatedness between the subject s and the predicate p and the object o in a triple (s, p, o). statistically speaking, if the co-occurrence of the s and po is high and the co-occurrence of the s and other po is low, people would list the po as a key trait of the s. we say that (s, p, o) is salient if, typically, most humans thinking of s would associate it with p, o. this somewhat informal definition is based on (and assumes) common human selectional preference @xcite as well as commonsense knowledge. as in other evaluation tasks, the annotators are required to decide whether a triple is salient or not based on the given judgment criteria. since the definition of salience is marginally vague and hard to annotate, we propose to harness the concepts of sufficiency and necessity to discern salience. the relationship between the salience s i of instance i and the necessary factor nec i and the sufficient factor suf i is expressed by the following equation: the necessity nec represents that the subject s is almost the only cause for predicate p and the object o to hold. while the sufficiency suf represents that the predicate p and the object o hold for most instances of the subject s. for example, the triple (running s , requires p , shoes o ) encodes more sufficiency, since in most situations running requires shoes. similarly, (running s , requires p , weight running vest o ) encodes more necessity, as running is almost the only cause for requiring weight running vest. when both necessity and sufficiency are strong, the triple should be salient. the reasons for leveraging both sufficiency and necessity factors to model the salience is demonstrated in appendix a. ## datasets construction to obtain labeled data for training models and as ground truth for evaluation, we construct a dataset in e-commerce. the construction process comprises knowledge acquisition and annotation. ## the proposed method driven by the analysis of salience, we propose a simple model to learn salience by utilizing continuous trainable prompt embedding @xcite @xcite . figure 2 presents the framework of the proposed model. ## experiments 6.1 experimental setup the model needs to determine whether the triple (s, p, o) is salient or not, which is essentially a triple classification task. thus, we use the f1, accuracy, and area under the roc curve (auc) as the metrics for all the test sets. with two types of split strategy (i.e., random split and concept split), we evaluate all the models on the test set and tune hyper-parameters in the development set. all baseline models are trained based on the simplified dataset. besides baseline models, we conduct experiments of pmi-tuning in the simplified training set and original training set. the textual encoder of all the models are all base models, i.e. bert-base @xcite and bart-base @xcite , from the transformer library foot_4 . we use batch size 8 and adam (kingma and ba, 2015) with an initial learning rate of 1 × 10 -5 for all models. for pmi-tuning, the template of the prompt is s [p 0:2 ] p [p 3:6 ] o [p 7:11 ]. for hyperparameters, we set γ = 0.1 and λ = 0.5. ## conclusion and future work we propose a framework of pmi-tuning that could deal with salience scores by probing the latent knowledge distribution in masked language models. besides, we annotate data for quantitatively evaluating the necessity, sufficiency, and salience of common sense. the experimental results show that evaluating the salience score remains a technical challenge, where most models exploit spurious lexical cues while prompt learning is beneficial to learning the intrinsic salience of a triple. along with this paper, we publish the annotated dataset and hope that this dataset will facilitate future research on salient commonsense knowledge. in addition, our future work plans include leveraging this salient csk for product recommendation and search in e-commerce. moreover, we found a positive correlation exists between click-through rate (ctr) and salience of triple. it indicates the potential of salience to play a greater role in search. we put the detailed verification process and exploratory applications in appendix g.