L ib KGE - A knowledge graph embedding library for reproducible research (2020)

Abstract: LibKGE ( https://github.com/uma-pi1/kge ) is an open-source PyTorch-based library for training, hyperparameter optimization, and evaluation of knowledge graph embedding models for link prediction. The key goals of LibKGE are to enable reproducible research, to provide a framework for comprehensive experimental studies, and to facilitate analyzing the contributions of individual components of training methods, model architectures, and evaluation methods. LibKGE is highly configurable and every experiment can be fully reproduced with a single configuration file. Individual components are decoupled to the extent possible so that they can be mixed and matched with each other. Implementations in LibKGE aim to be as efficient as possible without leaving the scope of Python/Numpy/PyTorch. A comprehensive logging mechanism and tooling facilitates in-depth analysis. LibKGE provides implementations of common knowledge graph embedding models and training methods, and new ones can be easily added. A comparative study (Ruffinelli et al., 2020) showed that LibKGE reaches competitive to state-of-the-art performance for many models with a modest amount of automatic hyperparameter tuning.

## introduction knowledge graphs (kg) @xcite ) encode real-world facts as structured data. a knowledge graph can be represented as a set of (subject, relation, object)-triples, where the subject and object entities correspond to vertices, and relations to labeled edges in a graph. kg embedding (kge) models represent the kg's entities and relations as dense vectors, termed embeddings. kge models compute a score based on these embeddings and are trained with the objective of predicting high scores for true triples and 1 https://github.com/uma-pi1/kge low scores for false triples. link prediction is the task of predicting edges missing in the kg @xcite . some uses of kge models are: enhancing the knowledge representation in language models @xcite , drug discovery in biomedical kgs @xcite , as part of recommender systems @xcite , or for visual relationship detection @xcite . kge models for link prediction have seen a heightened interest in recent years. many components of the kge pipeline-i.e., kge models, training methods, evaluation techniques, and hyperparameter optimization-have been studied in the literature, as well as the whole pipeline itself @xcite @xcite . @xcite argued that it is difficult to reach a conclusion about the impact of each component based on the original publications. for example, multiple components may have been changed simultaneously without performing an ablation study, baselines may not have been trained with state-of-the-art methods, or the hyperparameter space may not have been sufficiently explored. libkge is an open-source kge library for reproducible research. it aims to facilitate meaningful experimental comparisons of all components of the kge pipeline. to this end, libkge is faithful to the following principles: modularization and extensibility. libkge is cleanly modularized. individual components can be mixed and matched with each other, and new components can be easily added. ## modularization and extensibility libkge is highly modularized, which allows to mix and match training methods, models, and evaluation methods (see figure 1 ). the modularization allows for simple and clean ways to extend the framework with new features that will be available for every model. for example, libkge decouples the relation-alscorer (the kge scoring function) and kgeembedder (the way embeddings are obtained) as depicted in figure 1 . in other frameworks, the embedder function is hardcoded to the equivalent of libkge's lookupembedder, in which embeddings are explicitly stored for each entity. due to libkge's decoupling, the embedder type can be freely specified independently of the scoring function, which enables users to train a kge model with other types of embedders. for example, the embedding function could be an encoder that computes an entity or relation embedding from textual descriptions or pixels of an image @xcite . search:job +device_pool: str[...*] +process_pool: trainingjob +... 1 * trainingjob:job +model: kgemodel +loss: loss +optimizer: optimizer +valid_job: evaluationjob +... gridsearch axsearch quasi random search, bayesian optimization kgemodel +scorer: relationalscorer +entity_embedder: kgeembedder +relation_embedder: kgeembedder +... negativesampling kvsall onevsall 1 1 complexscorer transescorer ... relationalscorer +score_emb(s:emb,p_emb,o_emb) kgeembedder +embed(indexes) 1 1 lookupembedder projectionembedder ... evaluationjob entityranking 1 1 1 1 ... ... ... ## configurability and reproducibility reproducibility is important, which means that configuration is important. to enable reproducibility, it is key that the entire configuration of each experiment be persistently stored and accessible. while this sounds almost obvious, the crux is how this can be achieved. typically, source code can and will change. therefore, to make an experiment in a certain setting reproducible, the configuration for an experiment has to be decoupled from the code as much as possible. in libkge all settings are always retrieved from a configuration object that is initialized from configuration files and is used by all components of the pipeline. this leads to comprehensive configuration files that fully document an experiment and make it reproducible as well. to make this comprehensive configurability feasible-while also remaining modular-libkge includes a lightweight import functionality for configuration files. in figure 2 , we show an (almost) minimal configuration for an experiment for training a complex kge model @xcite . the main configuration file my experiment.yaml in figure 2 will automatically import the model-specific configuration complex.yaml, which in turn imports the configuration lookup embedder.yaml. the latter defines the default configurations of the lookupembedder for entities and relations, which associates every entity and relation identifier with its respective embedding. all configurations are merged into a single configuration object. during merging, the settings in the main configuration file always have precedence over the settings from imported files. the resulting single configuration will be automatically saved in the experiment directory along with the checkpoints and the log files. as an example of how configurability also helps modularization, we come back to the example of switching the lookupembedder with an encoder that computes entity embeddings from string tokens. for this purpose, one may implement a tokenpoolembedder. the simple changes to the configuration that uses the new embedder type are demonstrated in figure 3 the currently known best practices, libkge also includes-and makes configurable-some settings that might not be considered best practice, e.g., different tie breaking schemes for ranking evaluations @xcite . therefore, with regards to configurability, the goal is not only that the framework reflects best practices, but also reflects popular practices that might influence ongoing research. ## hyperparameter optimization hyperparameter optimization is crucial in empirically investigating the impact of individual components of the kge pipeline. libkge offers manual search, grid search, random search, and bayesian optimization; the latter two provided by the hyperparameter optimization framework ax. 2 in this context, libkge further benefits from its configurability because everything can be treated 2 https://ax.dev/ as a hyperparameter, even the choice of model, score function, or embedder. the example in figure 4 shows a simple hyperparameter search with an initial quasi-random search, and a subsequent bayesian optimization phase over the learning rate, batch size and negative samples for the complex model. the trials during the quasi-random search are independent, which can be exploited by parallelizing their runs over multiple devices. in this way, a comprehensive search over a large space of hyperparameters can be sped up significantly (also shown in the example; for more details, please refer to the documentation). ## profiling and metadata analysis libkge provides extensive options for profiling, debugging, and analyzing the kge pipeline. while most frameworks print the current training loss and some frameworks also record the validation metrics, libkge aims to make every moving part of the pipeline observable. per default, libkge records during training things such as runtimes, training loss and penalties (e.g., from the entity and relation embedders), relevant meta data such as the pytorch version and the current commit hash, and dependencies between various jobs. we show an example logging output during training one epoch in appendix b. for more fine-grained logging, libkge also can log at the batch level. during evaluation, the framework records many variations of the evaluation metrics, such as grouping relations by relation type, relation frequency, head or tail. additionally, users can extract and add information by adding their custom function to one of multiple hooks that are executed before and after all relevant calls in the framework. in this way, users can interact with all components of the pipeline, without risking divergence from libkge's master branch. finally, libkge provides convenience methods to export (subsets of) the logged meta data into plain csv files. ## comparison to other kge projects in this section, we compare libkge to other open source software (oss) that provides functionality around training and evaluating kge models for link prediction. the assessments are a snaphot taken at the end of may 2020. all model-specific comparisons have been evaluated w.r. in the focus on configurability and reproducibility in libkge is reflected by the large amount of configurable keys. for example, in contrast to other projects, libkge does not tie the regularization weights of the entity and relation embedder to be the same. for entity ranking evaluation, only libkge and pykeen transparently implement different tie breaking schemes for equally ranked entities. this is important, because evaluation under different tie breaking schemes can result in differences of ≈ .40 mrr in some models and can lead to misleading conclusions, as shown by @xcite . openke, for example, only supports the problematic tie breaking scheme named top by @xcite . libkge and pykeen are the only frameworks that provide machine-readable logging. only libkge offers resuming from a checkpoint for training and hyperparameter search. libkge, ampligraph, and pykeen are the most active projects in terms of amount of commits during the past 12 months. ## conclusions in this work, we presented libkge, a configurable, modular, and efficient framework for reproducible research on knowledge graph embedding models. we briefly described the internal structure of the framework and how it facilitates libkge's goals. the framework is efficient and yields stateof-the-art performance. we hope that libkge is a helpful ingredient to gain new insights into knowledge graph embeddings, and that a lively community gathers around this project to improve and extend it further.