zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: Using a saved model to train new observations
Root cause: Unimplemented Feature Gap

**Please also take a look at the main discussion on inductive link prediction at https://github.com/pykeen/pykeen/issues/720**

---
Is there a way to train on pykeen with a saved model (from a previous pykeen pipeline run)? My dataset gets new items added to it at set intervals and I'm looking for a way to add embeddings without retraining over the whole dataset.

The scenario you're describing is pretty tricky as far as machine learning goes - PyKEEN does not support it directly, currently.

One tricky thing you could do is add more entities with randomly assigned vectors, then train a bit more, but I'm not sure if that works really well in practice. I think @mali-git, @mberr , or @migalkin could give a better answer about how this is sort of like the "inductive" learning setting, which they're very interested in at the moment.

Thanks for getting back to me! Interested to hear other's ideas on this. 

As of now, I don't see how I'd get around this without retraining.

Follow-up question: I assume by new items in the dataset you mean triples. But do you know if they have new entities, or they're only between entities already in the graph?

Inductive NodePiece was added in https://github.com/pykeen/pykeen/pull/722. We'll make a 1.8 release soon!

If you're able to answer my follow-up question, we might be able to suggest alternate ways to do this kind of "online learning". Please feel to clarify and re-open this issue or start a new one :)
Repository: pykeen/pykeen
Issue: RGCN experiment
GitHub labels: question

Why does only the RGCN **WN18** experiment use the `schlichtkrull` sampler and the **FB15k** dosn't?

I believe this is a mistake, and FB15k should also use `schlichtkrull`.

The authors' implementation can be found here, https://github.com/MichSchli/RelationPrediction, but is not an easy read :slightly_smiling_face: 

Hi @moritzblum ,

the experiment configuration should now be fixed. It will be distributed via `pypi` with the next release. You can get earlier access to it by installing `pykeen` from source, cf. https://github.com/pykeen/pykeen#installation--
Repository: pykeen/pykeen
Issue: Improve docs on default metric for early stopper
GitHub labels: documentation

Hi folks,

[Here](https://pykeen.readthedocs.io/en/stable/reference/stoppers.html#pykeen.stoppers.EarlyStopper.metric), I see the default monitored metric for the early stopper is `'hits_at_k'`.

Which `hits@` is considered? Like, @1, @3, @5?

Thanks. 

The default is Hits@10.

The documentation is hidden here:
https://github.com/pykeen/pykeen/blob/7a87578f9bdb8fef28307bac1c037e0c177eee7a/src/pykeen/evaluation/rank_based_evaluator.py#L228-L240

@cthoyt maybe we can link this method to the documentation of the early stopper?
Repository: pykeen/pykeen
Issue: Memory error while attempting to train RGCN embeddings on my knowledge graph
Root cause: Resource Mishandling

Hello,

I tried generating RGCN knowledge graph embeddings and these are the parameters that I passed while invoking the model through pipeline.

```python
results = pipeline(
    training=training,
    testing=testing,
    validation=validation,
    model='RGCN',
    training_kwargs=dict(num_epochs=50, sampler="schlichtkrull", batch_size=8),
    random_seed=1235,
    device='cuda:0',
    stopper='early',
)
```

However regardless of whatever batch size that I set, I always end up getting this memory error that I have pasted below for your reference. 

```
raise MemoryError("The current model can't be trained on this hardware with these parameters.")
MemoryError: The current model can't be trained on this hardware with these parameters.

```
My knowledge graph has about 5 million triplets and I am training it on 80% of those 5 million triplets while 10 % has been allocated for testing and 10% has been allocated for validation. Can you please tell me a solution for this ? I have a Tesla V100 GPU as my CUDA device with about 16 gigs available on it. Thanks in advance.



That’s a pretty tiny GPU. You should probably decrease the dimensions of the embeddings as well 

I see. Yes I tried that just now but that affects the quality of the representation because I am trying to compress a lot  within lesser dimensions than needed to effectively represent the embeddings. So the spatial orientation of these embeddings is affected. I think I will get a bigger GPU. I do have multiple GPUs though. Is there a dataparallel wrapper for pykeen that allows me to parallelize this training across multiple CUDA devices ?

There's an experimental solution for multi-GPU support in https://github.com/pykeen/pykeen/pull/393 but it's not for allowing distributed storage of the representations. It's pretty well-known that GNN methods don't scale well on real datasets, so if you must have larger representations, maybe try a different kind of model in the mean time (they will also be much faster)

Alright. Thank you very much.

See pytorch lightning support for multi gpu in https://pykeen.readthedocs.io/en/stable/contrib/lightning.html
Repository: pykeen/pykeen
Issue: Compute embedding for new data
Root cause: Unimplemented Feature Gap

Hi, everyone!
I was wondering whether it is possible to compute the embedding for a completely new entity with a trained model, if you have triplets with the nodes the model already know. Or I should re-train the model and that is the only way?

I've seen this thing possible for text embedding model, i.e. FastText, so it got me wondering the same issues for graph embeddings

Hi @tatiana-iazykova ,

this is usually referred to as *inductive* link prediction as opposed to *transductive* link prediction where all entities are known at training time, cf., e.g., https://arxiv.org/abs/1706.02216

The short answer is that we currently do not have built-in support for these kind of models. There is some ongoing work, cf. #556 , but this will likely take some time to be available.

If you are interested in more works on this topic you can also search for keywords such as "emerging entities" or "out-of-vocabulary entities".

You may also want to take a look into this repo: https://github.com/mali-git/hyper_relational_ilp where we used some components of pykeen alongside custom code to do inductive link prediction on (hyper-relational) KGs.

Wow, thanks for the quick response! Will be looking forward for your build-in support :)
Repository: pykeen/pykeen
Issue: TranSparse model
Root cause: Unimplemented Feature Gap

**On which paper(s) is your requested model based.**
[Knowledge Graph Completion with Adaptive Sparse Transfer Matrix](https://ojs.aaai.org/index.php/AAAI/article/view/10089)

**List the original and other existing implementations of this model**
[One implementation on Github](https://github.com/FrankWork/transparse)

**Relevance of this model**
This model aims to solve the heterogeneity problem that some relationships connect to a large number of entities, while others may connect to only a small number of entities, which is very interesting and practical. 



@Hao-666 do you think you might want to try providing a PR for this?

Hi @cthoyt , sorry that I may not be able to implement a new model for now. I am not from computer science and just started touching knowledge graph embedding models recently. I came across this model and think it is interesting and practical, so I shared it here. But I am trying :)
Repository: pykeen/pykeen
Issue: Loss function value
Root cause: Documentation Deficiency

Hi, 

I was using DGLKE to train embeddings and would like to switch to PYKEEN. 

I noticed the loss function reported by PYKEEN is very different from DGLKE even if I used the same model (TransE) and the same loss function (MarginLoss).


This is an example output from DGLKE.
[proc 0][Train](100000/100000) average pos_loss: 0.34241078025102617
[proc 0][Train](100000/100000) average neg_loss: 0.4835104523599148
[proc 0][Train](100000/100000) average loss: 0.41296061608195306
[proc 0][Train](100000/100000) average regularization: 0.06404572251439095
[proc 0][Train](100000/100000) average loss: 0.41296061608195306
[proc 0][Train](100000/100000) average regularization: 0.06404572251439095
[proc 0][Train](100000/100000) average regularization: 0.06404572251439095


This is the output from PYKEEN. 
Training epochs on cuda:  65%|███▉  | 13/20 [12:25<06:40, 57.22s/epoch, loss=4.36e-6, prev_loss=4.44e-6]

Could you help me understand the difference?


Best Regards,
Jing

Hi @JXT218 ,

I am not very experienced with the DGL-KE codebase. You can find the implementation of pykeen's margin loss here: https://github.com/pykeen/pykeen/blob/3976d36bfef6e9aea05ceb2bd8fcf998f6ffda82/src/pykeen/losses.py#L529-L531

Some general things you may want to check are:
* [ ] are you using the same margin value?
* [ ] do you compare the models at similar points of the training?
* [ ] are the models trained by the same optimizer and with the same configuration (learning rate, dampening factors, etc.)
* [ ] did you use the same form of regularization for training both models?
* [ ] did you train both models using the same training assumption / loop (sLCWA vs. LCWA, same batch size, number of negative samples etc)

All of the aforementioned factors (and likely even some other factors) can influence the exact loss value.
Repository: pykeen/pykeen
Issue: Colab and Pykeen
Root cause: Environment Incompatibility

**Describe the bug**
I try to run this latest version of pykeen on colab but it caused me error as mentioned in PyKEEN issue #373. However, then I downgraded the scipy to 1.4.1 (as hinted by colab error log), then it worked, but then I find many other missing packages, 

Please note running this command 
from pkg_resources import require
 require('pykeen')
was not possible without downgrading scipy to 1.4.1

thanks
**To Reproduce**
Steps to reproduce the behavior:
1. [First step]
2. [Second step]
3. [...]
4. [See error]

**Expected behavior**
[A clear and concise description of what you expected to happen.]

**Environment:**

Please copy the table output by `pykeen version` here (and delete this text).
This includes information about your system and installation of PyKEEN.

**Additional information**
[Add any other information about the problem here.]
![image](https://user-images.githubusercontent.com/32249936/132759822-4cac8f12-b57c-4264-adde-9a23b61b4930.png)


I don’t understand why you downgraded scipy - the error shows that PyKEEN requires at minimum version 1.5.0. Could you please explain?



> I don’t understand why you downgraded scipy - the error shows that PyKEEN requires at minimum version 1.5.0. Could you please explain?

thanks for reply, later I didn't downgrade it, actually the problem is that it is not working with Colab, it does not read import pipeline and dataset line. then I used the package import (as mentioned in thread 373 but then I get additional issues like scipy package mismatch.



I fixed it using these commands

```
!python3 -m pip uninstall pykeen
! python3 -m pip install pykeen
```

thanks

Great! not sure what was going on - perhaps some old caches. If there's any more issues, please feel free to reopen this or send a new issue
Repository: pykeen/pykeen
Issue: How to implement my own model to leverage numerical triples
Root cause: Documentation Deficiency

Hello ! I am currently working on a way to adapt KG embedding models so they can handle numerical triples too.
The idea is different from LiteralE (already implemented in pykeen), but simple. I have a transformation that takes as input a number `t`, and outputs a vector `e(t)`, then I can use this vector in the scoring function `f(h, r, t)` of any model, as if it were the embedding of a discrete entity.

I am trying to build this directly in pykeen, so I can use all the models and nice features it provides, but I don't really know how to start. Any advices on that ?

Hi, @alexis-cvetkov 

I not really sure i understood the idea of the model you are trying to implement, however if you are trying to implement an alternative transformation to what LiteralE does I would recommend you to look at the `src/pykeen/nn/combinations.py` file. You can easily add a combination that fits what you are trying to implement, and then it is just a matter of creating a `LiteralModel` derivative and tell it to use your combination (see `src/pykeen/models/multimodel/distmult_literal.py` to better understand the structure).

I don't know if this answered your question.
Repository: pykeen/pykeen
Issue: Question about high GPU memory usage on the evaluation step.
Root cause: Documentation Deficiency

I have a question relative to the evaluate module. Is it normal for the evaluation step to have a much higher GPU memory usage than the training phase even when the evaluation batch size is smaller and has less triples?

More concretely, I am working with a dataset with the following sizes:

Training: 500459 triples
Validation: 59651 triples  
Test: 59986 triples

I am using DistMult as the model to fit this data.

During the training phase I find that my GPU is using ~1GB of its memory. However as soon as the evaluation step starts the memory usage shoots up to ~12GB even though the batch size used in the evaluation step is lower compared to the training one (512 vs 128).

Lowering the evaluation batch size obviously lowers the usage (~3 GB when using 32 batch size on evaluation), but even so I can't think of a reason for this discrepancy. I would really appreciate it if someone could clarify why this happens.


**Note:** Bellow i have a graph that shows the evolution of memory usage in MB (y axys) along the time in minutes (x axis). The moment where the memory shoots up is when the evaluation step starts. (graph scale can be difficult to see if you have github on dark mode)

![download (2)](https://user-images.githubusercontent.com/69594660/135064580-0c377cd7-04ec-4c61-81e4-89e584e2e0a1.png)


Hi @Rodrigo-A-Pereira ,

depending on which training loop you use, this is expected: The evaluation always uses 1:n scoring, i.e., for a triple `(h, r, t)` computes scores for all possible entities `t'`. Hence, the evaluation's memory requirements are roughly similar to training with the LCWA loop (`O(batch_size * num_entities)` when assuming a constant cost of computing one score). A longer description of the evaluation setting can be found here: https://pykeen.readthedocs.io/en/stable/tutorial/understanding_evaluation.html

btw, there is a builtin automatic batch size optimization for evaluation, which automatically infers the maximum possible batch size for evaluation. This is generally desirable, since it ensures that you utilize the GPU as much as possible. You can find more information about this here: https://pykeen.readthedocs.io/en/stable/tutorial/performance.html#automated-memory-optimization

Hi @mberr,

Thank you for the quick reply. It now makes sense, i completely forgot about the impact of the training training loop (I am using the sLCWA).

Also thank you for pointing out the existence of the "automatic batch size optimization", i didn't realize its existence, it seems very helpful for my usecase.
Repository: pykeen/pykeen
Issue: How to know the adopted hyper parameter values in a model?
GitHub labels: question

Hi, take the TransR as an example, how do I know all the adopted values of hypter parameters in this model? Thanks!

The built-in HPO, [pykeen.hpo.hpo_pipeline](https://pykeen.readthedocs.io/en/stable/reference/hpo.html#pykeen.hpo.hpo_pipeline), uses [optuna](https://optuna.readthedocs.io/en/latest/index.html). The returned [HpoPipelineResult](https://pykeen.readthedocs.io/en/stable/reference/hpo.html#pykeen.hpo.HpoPipelineResult) offers an attribute `study` to access the optuna [study](https://optuna.readthedocs.io/en/latest/reference/generated/optuna.study.Study.html#optuna.study.Study), which in turn has an attribute `best_params`.

The study object also allows to inspect *all* tried parameter configurations and their objective values.

Thanks @mberr. What I mean is not about the parameters in a HPO study. I mean that when I call the model in this library, where can I know the parameters? Let say I call TransR using the codes below, then what are the parameters set in it? How can I check?

```
tf = TriplesFactory.from_path(...)
training, testing, validation = tf.split([.6, .2, .2],random_state=42)

results_transR = pipeline(
    training=training,
    testing=testing,
    validation=validation,
    model='transR',
    training_kwargs=dict(num_epochs=300,batch_size=512),
    random_seed=42
)
```


Hmm I’ve written code that introspects on the signature of a given model class to build the command line Interface and also to run the pipeline itself, but this isn’t exposed cleanly to the user. It might take a bit of work to do that, can you explain what your use case is, then we could prioritize writing this code or making an example for you to use 

As a rule of thumb, if you do not provide `model_kwargs` they should mostly corresponds to their default values from them model's constructor, e.g., for [TransR](https://pykeen.readthedocs.io/en/stable/api/pykeen.models.TransR.html#pykeen.models.TransR). Notice that there are `**kwargs` passed to the super class, some of which are automatically filled.

Thanks @cthoyt . My case is that I am writing a report where TransR in this library is used (many thanks to this great library). I need to write clearly about the adopted hyper parameter values. So I wondered about the checking of the hyper parameter values in the model. 

Thanks @mberr . There are some default values such as embedding dim. But it seems that not all of them are clearly written in the doc such as batch size.
Repository: pykeen/pykeen
Issue: Can't get prediction for nodes that were in numeric triplets
Root cause: Documentation Deficiency

I was trying to get relation prediction for nodes that were in training data in ComplExLiteral model. Is this supposed to be that numeric triples only used in training process but not during the inference?

Code to replicate:
```python

TRIPLES = np.array(
    [
        ['peter', 'likes', 'chocolate_cake'],
        ['chocolate_cake', 'isA', 'dish'],
        ['susan', 'likes', 'chocolate_cake'],
        ['susan', 'likes', 'pizza'],
        ['peter', 'likes', 'susan'],
        ['peter', 'likes', 'mango'],
        ['mango', 'isA', 'fruit'],
        ['peter', 'isA', 'person'],
        ['susan', 'isA', 'person'],
    ],
    dtype=str,
)

NUMERIC_TRIPLES = np.array(
    [
        ['peter', 'age', '12'],
        ['peter', 'has_apples', '15'],
    ],
    dtype=str,
)


tf = TriplesNumericLiteralsFactory(triples=TRIPLES, numeric_triples=NUMERIC_TRIPLES)

results = pipeline(
    training=tf,
    testing=tf,
    model='ComplExLiteral',
    training_loop='sLCWA',
    negative_sampler='basic',
    training_kwargs = dict(num_epochs=1),
    model_kwargs = dict(embedding_dim=2),
    random_seed= 42
)

predict.get_relation_prediction_df(results.model, head_label='peter', tail_label= '12',  
                               triples_factory=results.training,)

KeyError                                  Traceback (most recent call last)
/tmp/ipykernel_1478/3281283891.py in <module>
----> 1 predict.get_relation_prediction_df(model = results.model, head_label='peter', tail_label='12',
      2                                    triples_factory=results.training)
 
~/jupyter_hub_graph/pykeen-master/src/pykeen/models/predict.py in get_relation_prediction_df(model, head_label, tail_label, triples_factory, add_novelties, remove_known, testing)
    198     """
    199     head_id = triples_factory.entity_to_id[head_label]
--> 200     tail_id = triples_factory.entity_to_id[tail_label]
    201     batch = torch.as_tensor([[head_id, tail_id]], dtype=torch.long, device=model.device)
    202     scores = model.predict_r(batch)
 
KeyError: '12'
```

This isn’t really how the literal models work. The literal triples aren’t considered as nodes, so there’s no way to predict a connection to “12”

Ok, got it. Thanks a lot!


I'm not sure why you would want to do a relation prediction task for this, but perhaps it would be interesting to do a tail inference for a given node / numeric predicate. I don't think there are any models published on this, so it would be a cool area of research

Hi @tatiana-iazykova, I'm going to close this issue, but if there's any follow-up, please feel free to post here or re-open this issue.
Repository: pykeen/pykeen
Issue: Question about documentation link
GitHub labels: question

https://pykeen.readthedocs.io/en/latest/api/pykeen.models.TorusE.html

In this link, the "TorusE" should be TransE, right?

No, this is indeed the page for TorusE. You can find the documentation for TransE here: https://pykeen.readthedocs.io/en/latest/api/pykeen.models.TransE.html

@Diison I'm going to close this one - if you have some further questions about the differences between TorusE and TransE after looking into their respective papers (both linked in the docs and here), please feel free to open a new issue or discussion.

| Model | Docs | Paper |
|-------|-------|-------|
| TransE | https://pykeen.readthedocs.io/en/latest/api/pykeen.models.TransE.html | http://papers.nips.cc/paper/5071-translating-embeddings-for-modeling-multi-relational-data.pdf |
| TorusE | https://pykeen.readthedocs.io/en/latest/api/pykeen.models.TorusE.html | https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16227 |
Repository: pykeen/pykeen
Issue: Change file format of experiment configurations
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
JSON files do not allow for comments. However, when describing an experimental setup, comments are often very helpful, e.g., to describe how a particular value was found, and e.g. how concepts of other libraries / code bases are mapped onto pykeen's.

**Describe the solution you'd like**
**Describe alternatives you've considered**
Change the file format to something which allows comments, e.g., [TOML](https://toml.io/en/) or [YAML](https://yaml.org/). The format should be human-readable, easy to edit without any specialised tools, well-suited for version control (text-based, meaningful diffs, one line per setting), and supported by Python.


**Additional context**
motivated by the work done in #610
Repository: pykeen/pykeen
Issue: Introduce BoxE
GitHub labels: 💃 Model

💎 New Component

Hello, I would like to recommend to add BoxE (NeurIPS 2020 spotlight paper, https://proceedings.neurips.cc/paper/2020/file/6dbbe6abe5f14af882ff977fc3f35501-Paper.pdf) into PyKEEN. Its official implementation is in TF 1.x, and is available at https://github.com/ralphabb/BoxE.

BoxE is a model that naturally scales to higher-arity knowledge bases (which, to my knowledge, is not something currently supported by PyKEEN), and which can be injected with rules, as well as capture a rich rule language of inference patterns. 
I am happy to help first build a KG version of BoxE (i.e., that applies to binary facts only), and subsequently extend PyKEEN (and the BoxE implementation) towards the higher-arity setting and include all the paper datasets.   I already have built a preliminary interaction-based implementation of KG BoxE, and would really appreciate some help iterating over this and validating it (e.g., reproducing results, etc.)



@ralphabb thanks for moving discussion over to GitHub. Please feel free to make a draft PR as soon as you start writing code for this (don't wait until you're finished - discussion and iteration is much more practical to start as early as possible)