Going “Deeper”: Structured Sememe Prediction via Transformer with Tree Attention (2022)

Abstract: Sememe knowledge bases (SKBs), which annotate words with the smallest semantic units (i.e., sememes), have proven beneficial to many NLP tasks. Building an SKB is very time-consuming and labor-intensive. Therefore, some studies have tried to automate the building process by predicting sememes for the unannotated words. However, all existing sememe prediction studies ignore the hierarchical structures of sememes, which are important in the sememe-based semantic description system. In this work, we tackle the structured sememe prediction problem for the first time, which is aimed at predicting a sememe tree with hierarchical structures rather than a set of sememes. We design a sememe tree generation model based on Transformer with adjusted attention mechanism, which shows its superiority over the baselines in experiments. We also conduct a series of quantitative and qualitative analyses of the effectiveness of our model. All the code and data of this paper are available at https://github.com/thunlp/STG .

## introduction a word is the fundamental element of natural languages, but its meaning can be further divided. to explore semantics atomically, linguists define a sememe as the minimum semantic unit @xcite . it is even believed that the meanings of all words in any language can be represented by a limited set of sememes, which is closely related to the idea of semantic primitives @xcite . hownet @xcite is the most well-known sememe knowledge base (skb). it comprises more than 100,000 english and chinese words and phrases manually annotated by about 2,000 sememes that are defined by linguistic experts. multiple senses of a polysemous word are independently annotated, and the sememes annotated to a sense are hierarchically organized as a sememe tree. figure 1 illustrates the sememe annotations of two english words in hownet. different from other lexical knowledge bases, skbs like hownet define words intensionally with a limited set of semantic units (sememes), thus have some unique strengths. for example, skbs can be combined with neural network models smoothly by regarding sememes as the external semantic labels of words @xcite . moreover, thanks to the limitedness of sememes, skbs have been proven very useful in the lowdata regimes, e.g., improving the representation learning of rare words by transferring knowledge from frequent words via sememes @xcite . as a result, skbs have been widely utilized in many nlp tasks @xcite . however, most languages have no skbs like hownet, and it is too expensive to manually build an skb for a new language from scratch. 1 in addition, even for the languages covered in hownet (english and chinese), new words are emerging every day and the meanings of existing words keep changing. it is also costly to expand and update hownet. to solve these issues, a series of studies have been conducted, trying to automatically predict sememes for monolingual or cross-lingual words @xcite @xcite @xcite . for simplicity, all previous sememe prediction studies ignore the hierarchical structures of sememes. they simplify sememe prediction as a multi-label classification task, and their models output a structureless set of sememes. however, the structures of sememes are very important. for one thing, the structural information is indispensable in the sememe-based semantic description system, as it carries semantics, and branches of sememe trees stand for the relations of sememes. as shown in figure 1 , the difference in sememe structure results in the different meanings of the second sense of "throne" and "emperor", although they have four identical sememes. for another, the structures of sememes are necessary for many sememe-based applications @xcite @xcite . in this paper, we try to tackle structured sememe prediction, which is aimed at predicting sememes together with their hierarchical structures rather than the structureless sememes only. this task is essentially a kind of tree generation task but is more challenging than other tree generation tasks. first, the size of its node type is more than 2,000 (i.e., over 2,000 sememes), which is much larger than that of most tree generation tasks, e.g., less than 100 for code generation and semantic parsing (rab). second, the structures of sememe trees are extremely diverse -almost any sememe can be the child node of another sememe, and one sememe node can have an arbitrary number of children. many of the existing tree prediction methods depend on the certain number of children of a node and perform strongly correlated with the number of candidates @xcite , thus are not applicable. to handle this difficult task, we conduct further formalization. different from most structureless sememe prediction studies whose input is merely a word, inspired by @xcite , we regard a sentence of definition as the input, and the task is formalized as a sequence-to-tree task. we do this for two reasons. first, sememe prediction can be conducted at the sense level (one definition corresponds to one sense of a word). second, definitions can provide more useful information than single words for structured sememe prediction. further, we propose a model based on transformer @xcite especially designed for the task of sememe tree generation (stg). we decompose the attention in transformer into two parts that capture the semantic similarity and topological relations between sememes, respectively, in order to better represent the characteristics of sememe trees. experimental results show that our method outperforms baseline methods including the vanilla tree transformer model. we also conduct quantitative and qualitative analyses of the results of our method. ## methodology in this section, we first detail two straightforward sememe tree generation (stg) models, which will serve as the baselines. then, we describe the modification of tree attention and introduce a novel stg model. ## analysis in this part, we further discussed the efficiency of positional attention and analyzed the performance of our model in different tree complexity, and make a case analysis of our models. ## conclusion and future work in this paper, we handle the structured sememe prediction task for the first time. we propose a transformer-based tree generation model by adapting the attention mechanism to trees. experimental results show that our model outperforms baselines including the general tree transformer. we also conduct extensive experiments and detailed analyses to demonstrate the different properties of our models and the challenges of the task. we will explore the following research directions in the future: (1) we will better measure the semantic similarity of tree nodes. in this paper, the strict-f1 score only focuses on the structure and ignores the semantic similarity of generated sememe pairs with the answer. (2) we will further explore to import the tree-attention mechanism in all sublayers of the decoder and figure out the influence.