Random Entity Quantization for Parameter-Efficient Compositional Knowledge Graph Representation (2023)

Abstract: Representation Learning on Knowledge Graphs (KGs) is essential for downstream tasks. The dominant approach, KG Embedding (KGE), represents entities with independent vectors and faces the scalability challenge. Recent studies propose an alternative way for parameter efficiency, which represents entities by composing entity-corresponding codewords matched from predefined small-scale codebooks. We refer to the process of obtaining corresponding codewords of each entity as entity quantization, for which previous works have designed complicated strategies. Surprisingly, this paper shows that simple random entity quantization can achieve similar results to current strategies. We analyze this phenomenon and reveal that entity codes, the quantization outcomes for expressing entities, have higher entropy at the code level and Jaccard distance at the codeword level under random entity quantization. Therefore, different entities become more easily distinguished, facilitating effective KG representation. The above results show that current quantization strategies are not critical for KG representation, and there is still room for improvement in entity distinguishability beyond current strategies.

## introduction knowledge graphs (kgs) comprise (head entity, relation, tail entity) triplets. they are crucial external knowledge sources for various natural language processing tasks @xcite . learning representations on kgs is necessary for expressing complex semantics and supporting downstream tasks. the most dominant paradigm, kg embedding (kge), maps entities and relations to a vector space @xcite @xcite . despite the popularity, kge models need to represent each figure 1 : the process of parameter-efficient compositional kg representation. entities are quantized to entity codes, which are encoded to represent entities. each dimension of an entity code stands for a codeword, and indicates whether current entity matches this codeword (set to 1) with optional weights or not (set to 0). entity with an independent vector, which leads to a linear increase in the number of parameters with the number of entities. consequently, scalability becomes a challenge for these models, posing difficulties in their implementation and deployment @xcite , especially for large-scale kgs @xcite . a recently proposed parameter-efficient kg representation method uses compositional entity representations to reduce parameters @xcite . instead of learning separate representations like kge, it represents entities by composing their matched codewords from predefined codebooks through an encoder, and requires fewer parameters since codewords are much fewer than entities. see fig 1 for an illustration of this method. we refer to the process of obtaining corresponding codewords to each entity as entity quantization due to its similarity to vector quantization @xcite . specifically, existing methods construct two codebooks, in which codewords are the entire relation set and a selec-tive subset of entities, i.e., anchors, respectively. from these two codebooks, each entity matches two groups of codewords: connected relations and anchors nearby @xcite or with similar adjacent relations @xcite . @xcite also regards matching degrees as codeword weights for more expressiveness. matched results are denoted as entity codes, including matched codewords and optional weights. a subsequent encoder uses entity codes to compose corresponding codewords and generate entity representations. this approach performs closely to kge with fewer parameters, making kg training and deployment more efficient. the key to entity quantization lies in two steps: we design several model variants for experiments. first, to investigate the effectiveness of matching codewords with connectivity, we randomize the codeword matching step by randomly selecting codewords as matched results of entities. moreover, we set codeword weights randomly or equally for @xcite to verify whether designed weights from matching are critical. finally, to explore whether mapping codewords to actual elements in the kg is critical, we randomly construct codebooks with codewords that have no actual meaning. we adopt random codeword matching to the randomly constructed codebook, to provide a fully random entity quantization. counterintuitively, empirical results show that the above operations achieve similar results compared to complicated quantization strategies and may even improve the model performance. moreover, we have verified that random entity quantization can better distinguish entities than current quantization strategies, which leads to more expressive kg representations @xcite . under the strategies designed by previous works, different entities could match the same codewords, making their code similar or identical. in contrast, random quantization leads to a lower possibility of matching same codewords and distributes entity codes more uniformly across a wide range. we prove this claim by analyzing the properties of entity codes. at the code level, we consider entity code as a whole and treat it as one sample of a random variable. the entropy of this variable can be derived from its distribution across all entity codes. we prove that random entity quantization has higher entropy and maximizes it with high probability, thus producing more diverse and unique entity codes. at the codeword level, each entity code indicates a set of matched codewords. we analyze the jaccard distance between different sets and find that it is significantly increased by random entity quantization. as a result, different entities will have a more obvious dissimilarity when randomly quantized, making them easier to distinguish. in summary, the contributions of our work are two-fold: (1) we demonstrate through comprehensive experiments that random entity quantization approaches perform similarly or even better than previously designed quantization strategies. (2) we analyze that this surprising performance is because random entity quantization has better entity distinguishability, by showing its produced entity codes have higher entropy and jaccard distance. these results suggest that current complicated quantization strategies are not critical for the model performance, and there is potential for entity quantization approaches to increase entity distinguishability beyond current strategies. ## experimental setup though previous methods have designed complicated strategies for entity quantization, whether these strategies are critical for the model performance remains to be explored. our experiments are to empirically test the effect of these quantization strategies. we therefore design a series of model variants using random entity quantization. we focus on showing the effectiveness of random entity quantization, rather than proposing new stateof-the-art models. below we introduce the model variants we design, the datasets we test, and the training/evaluation protocols. ## random entity quantization this section details the random variants we design and their experimental results. we design model variants by randomizing different steps of existing entity quantization strategies, including codeword matching and codebook construction. we find that these variants achieve similar performance to existing quantization strategies. these results suggest that the current entity quantization strategies are not critical for model performance. ## why random quantization works this section analyzes the reasons for the surprising performance of random entity quantization. the entity codes directly affect entity representation and model performance. we analyze the entity codes produced by different entity quantization approaches to compare their ability to distinguish different entities. we find random entity quantization obtains entity codes with greater entropy at the code level and jaccard distance at the codeword level. thus, it can better distinguish different entities and represent kgs effectively. ## conclusion in conclusion, this paper demonstrates the effectiveness of random entity quantization in parameterefficient compositional knowledge graph representation. we explain this surprising result by illustrating that random quantization could distinguish different entities better than current entity quantization strategies. thus, we suggest that existing complicated entity quantization strategies are not critical for model performance, and there is still room for entity quantization approaches to improve entity distinguishability beyond these strategies.