zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: Improve documentation on training assumptions
GitHub labels: documentation

Pykeen uses only 2 training models: **sLCWA** and **LCWA**.

**How can I change it so that it works with Open World Assumptions?**

It doesn't really make sense to train under the open world assumption since that means there are no negative examples. The stochastic local closed world assumption (sLCWA) is the appropriate analog. We wrote about this in the PyKEEN software and benchmarking papers, but the philosophy behind it never seemed to make it into the PyKEEN documentation.

I'll start a new section in the tutorial called "Understanding the Training Assumptions"

Hi @apoorva2810 
In addition to what @cthoyt said. When your Knowledge Graph(KG) follows the Open World Assumption(OWA) you don't have negative samples in most cases, so when training without negative triples your model will overgeneralize, cf. [Nickel et al.](https://ieeexplore.ieee.org/abstract/document/7358050). Therefore, you usually want to sample negative triples, which is what the stochastic Local Closed World Assumption (sLCWA) does. It samples only some negative triples from the LCWA approach, hence the name as described in our benchmarking paper https://arxiv.org/abs/2006.13365. This is also what other publications or software packages falsely call OWA.

In case you really want to train without generating negative samples, i.e. training under the true OWA, you would need to make some changes to PyKEEN. However, this doesn't really make sense as indicated by @cthoyt 

Thanks, @cthoyt and @lvermue

But how can I avoid "key error" for an entity which has never occurred before during training but can be true, I think this can be considered as a typical read world situation where a knowledge graph need not be complete and, entities and relation which knowledge graph has not seen before can also be true.

Hi @apoorva2810 ,

the models in pykeen are trained in the *transductive* setting, where each entity has to been seen during training. This is explicitly enforced to make sure that you see an error when you try to get an embedding from a non-existing entity.

We plan to support inductive settings for the future, but now we do not have code for this. 

For a discussion of the transductive and inductive setting, you can, e.g., look at https://arxiv.org/abs/1706.02216 .

Thanks @mberr for clarifying.

This was addressed in #534, please let me know if the new documentation at https://pykeen.readthedocs.io/en/stable/reference/training.html is unclear
Repository: pykeen/pykeen
Issue: Tutorial on using PyKEEN with a GPU in Google Colab
GitHub labels: documentation

needs step-by-step tutorial with screenshots

bump

This look interesting. Is there a sample of this?

Not yet, but thanks for the bump ;) I will try and make one over the weekend or early next week. It's actually a lot easier than I had originally thought since you can integrate Google Colab with GitHub and directly open a Jupyrer notebook that's in a GitHub repo, then commit changes back to GitHub (Colab itself is a modified version of Jupyter notebook/lab). The trick is to make sure all of the installation code is at the top of the notebook
Repository: pykeen/pykeen
Issue: Is there a way to visualize the embeddings?
GitHub labels: question

I'm new to using Pykeen. Is there a way to visualize the embeddings? Tensorboard seemed like an option but it requires a tensorflow variable.

As far as I know, you can load TSV files into the embedding projector, cf. https://www.tensorflow.org/text/guide/word_embeddings#retrieve_the_trained_word_embeddings_and_save_them_to_disk

Here is a tutorial for hpw you can get the embedding matrices: https://pykeen.readthedocs.io/en/stable/tutorial/first_steps.html#using-learned-embeddings

Awesome thanks! One more question. I'm having trouble saving the embedding from PairRE (I get the error that 'PairRE' object has no attribute 'entity_embeddings' ). How would I save PairRE embeddings to a tsv file?

Example for plotting: 

https://pykeen.readthedocs.io/en/stable/tutorial/translational_toy_example.html

Various methods for plotting embeddings are built in to the PipelineResult class https://pykeen.readthedocs.io/en/stable/api/pykeen.pipeline.PipelineResult.html#pykeen.pipeline.PipelineResult, specifically https://pykeen.readthedocs.io/en/stable/api/pykeen.pipeline.PipelineResult.html#pykeen.pipeline.PipelineResult.plot_er

I'm trying to use the embeddings as an input for another model so something on how to save PairRE embedding would be helpful! 

There are several sections in the docs about that and other issues asking the same question. Please check those out and ask again if they’re not sufficient 

Thanks! The old posts helped me figure this out
Repository: pykeen/pykeen
Issue: BYOD for ablation
Root cause: Documentation Deficiency

For the ablation pipeline, have there been improvements to using custom triples for experiments? I noticed that as of last year the method was to load the data in a config file before the ablation study.

Please see https://pykeen.readthedocs.io/en/stable/tutorial/running_ablation.html#run-an-ablation-study-with-your-own-data for an example on how to do this. Let me know if that's not sufficient to get yourself up and running
Repository: pykeen/pykeen
Issue: What is the proper n_trials when using hpo_pipeline to optimize hyper-parameters?
GitHub labels: question

Hi, I am following the [doc](https://pykeen.readthedocs.io/en/stable/tutorial/running_hpo.html) to obtain optimized hyper-parameters as follows.

```
results_transR_optim = hpo_pipeline(
    n_trials=30,
    training=training,
    testing=testing,
    validation=validation,
    model='TransR',
)
results_transR_optim.save_to_directory('results_transR_optim')
```

I am wondering what is the proper `n_trials` to try to find the optimized hyper-parameters? Thanks! 

Hi @Hao-666 ,

`n_trials` defines how many different configuration will be explored in your HPO. There is no perfect value for this, but it rather depends on the size of search space, how thoroughly you want to explore it, and how much resource you have at hand / want to spent on HPO.

Thanks @mberr ,

I want to explore it as thoroughly as possible but certainly I don't want to let say set it as 80 with 100 configurations in total if 80 has very minor differences with 60. It seems that the total configurations by default is large (sorry I didn't find the exact number, grateful if you could point this out), for example I saw the [doc](https://pykeen.readthedocs.io/en/stable/tutorial/running_hpo.html#:~:text=the%20TransE%20model%E2%80%99s,size%20of%2016.) says that `the TransE model’s default HPO strategy for its embedding_dim argument is to search between  [16,256] with a step size of 16.` So I am wondering are there suggested numbers according to experiences? Assume no worries about the resources.

In our benchmarking study, cf. [here](https://github.com/pykeen/benchmarking), we used a maximum of 100 trials or 24 hours for each sub-configuration, whatever came earlier. Due to the large number of subconfigurations, we still ended up with more than ~25,000 GPU hours.

Notice that due to the large number of hyperparameters not only of the interaction function, but also e.g., the optimizer, the training loop or the loss function, a grid-search with all possible combinations is generally not advisable, but other search strategies, e.g. random search, cf. [here](https://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf), or more advanced methods such as the sampling and pruning algorithms from [optuna](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/003_efficient_optimization_algorithms.html) are more resource-efficient. Moreover, for continuous hyperparameters such as the learning rate, exploring *all* possible values is impossible. Using optimization frameworks aware of the continuity is generally speaking better than manually arbitrarily discretizing these parameters' search space.

My knowledge graph is small (hundreds of entities, no more than 10 relations, no more than 10,000 triples) so I intended to use a large `n_trials` to do the HPO. Thanks for your informative reply @mberr  :)
Repository: pykeen/pykeen
Issue: Lack of random_seed parameter in hpo_pipeline?
Root cause: Documentation Deficiency

Hi , when I used the codes below to obtain optimized hyper-parameters, it showed that `No random seed is specified. Setting to 1470630583`. But there is no random seed parameter in [hpo_pipeline](https://pykeen.readthedocs.io/en/stable/reference/hpo.html#pykeen.hpo.hpo_pipeline). Does it mean that the optimized results maybe different each time the codes are run? Thanks!
```
results_transR_optim = hpo_pipeline(
    n_trials=30,
    training=training,
    testing=testing,
    validation=validation,
    model='TransR',
)
results_transR_optim.save_to_directory('results_transR_optim')
```


Hi @Hao-666 ,

each trial of the hpo study uses a fixed, but randomly chosen random seed. The used random seeds will be stored (@mali-git correct me if I am wrong here), so you can deterministically re-run the best configuration. The HPO study itself is non-deterministic, i.e. running the search multiple times will probably sample different configurations each time.

Got it. Thanks @mberr! BTW I suggest supplementing this explanation in the doc to avoid such confusion :) 

I don't think you can set a random state for HPO to make it deterministic. If anyone has seen this in Optuna, I would definitely look into adding a random_seed parameter to the hpo_pipeline function that uses it
Repository: pykeen/pykeen
Issue: RGCN with pipeline
Root cause: Validation Gap

I get 

`TypeError: RGCN: __init__() missing 1 required keyword-only argument: 'interaction'`

when I do:

```
tf = TriplesFactory.from_path(args.data)
training, testing, validation = tf.split([.8, .1, .1])
result = pipeline(
    training=training,
    testing=testing,
    validation=validation,
    model='RGCN',
    stopper='early',
)
```

How can I use RGCN with `pipeline`? Thanks.

Hi @flpgrz, this seems like a bug more than a question. It would be good to know a bit more about your environment before looking into this. Can you please copy-paste the result of running `pykeen version` in the shell into this issue?

Also Grüß aus Bonn ;) 

Version 1.5.0

Grüsse aus Heidelberg! :) 

The reason seems to be that RGCN does not define a default interaction [here](https://github.com/pykeen/pykeen/blob/bc7f2f0703ffd86b7b885be4c19f9893ba625b1f/src/pykeen/models/unimodal/rgcn.py#L74).

@cthoyt I think it would be safe to put DistMult there, since it was also used in experiments by the original authors.

@flpgrz To get it running before we change this, you can explicitly specify the interaction function you would like to use as a decoder by the `model_kwargs`:
```python
result = pipeline(
    ...,
    model_kwargs=dict(
        interaction="DistMult",
    ),
)
```

Another comment: By default, pykeen uses a random batch sampler. For GNN-based models this may be undesired since the resulting batch subgraphs are easily disconnected. You can change this by adapting the code to
```python
result = pipeline(
    ...,
    training_kwargs=dict(
        sampler="schlichtkrull",
    ),
)
```
Be warned that creating these batches is somewhat slow. We recommend using larger batch sizes, and multiple data loader threads. The latter can be achieved by
```python
result = pipeline(
    ...,
    training_kwargs=dict(
        num_workers=8,
    ),
)
```
Depending on the speed differences between your GPU/CPUs you may still experience slowdown though. We very much would welcome contributions in accelerating this!


@mberr dismult should definitely be the default (surpised it isn't) and we should emit a warning when not using the Schlichtkrull sampler with R-GCN

> we should emit a warning when not using the Schlichtkrull sampler with R-GCN

this is already done :slightly_smiling_face: 

@mberr @flpgrz I've made a PR https://github.com/pykeen/pykeen/pull/547 to improve the exception so it has more pykeen-specific information on how to fix this (this also involved updating the class_resolver!)

Let me know what you think. This PR could also be extended to give more information on any of the other components, too (or maybe we should get more clever about generalizing this)

@flpgrz in #548 we made DistMult the standard in R-GCN. The other PR, #547 is interesting but I think it's actually better that you should never _have_ to use the `model_kwargs`, so I might close that one without merge.
Repository: pykeen/pykeen
Issue: Could not find a coverage of all entities and relation
GitHub labels: bug

**error faced**
i am trying to train a new model with a custom kg, but I keep coming across this error:
```python-traceback
  File "C:\Users\sudha\anaconda3\envs\kg_embedding\lib\site-packages\pykeen\triples\splitting.py", line 329, in split
    raise ValueError(f"Could not find a coverage of all entities and relation with only {sizes[0]} triples.")
ValueError: Could not find a coverage of all entities and relation with only 4369 triples.
```
 
any possible solution?


Hi @sudhanvarajesh, we’ll need a bit more context to give assistance. Can you please share the version information about pykeen (as requested in the bug issue template), the exact code you ran, and possibly the custom data?

I'm going to close this since there wasn't a response. Please feel free to reopen if you can provide some more context
Repository: pykeen/pykeen
Issue: HPO strategies
GitHub labels: question

What is the difference between:

[stepping](https://pykeen.readthedocs.io/en/stable/tutorial/running_hpo.html?highlight=optuna#stepping)

```
negative_sampler_kwargs_ranges=dict(
        num_negs_per_positive=dict(type=int, low=10, high=100, step=10),
    ),
```

and [custom strategy with q=](https://pykeen.readthedocs.io/en/stable/tutorial/running_hpo.html?highlight=optuna#custom-strategies)

```
model_kwargs_ranges=dict(
        embedding_dim=dict(type=int, low=100, high=500, q=100),
    ),
```

?

Also, since [power scale ](https://pykeen.readthedocs.io/en/stable/tutorial/running_hpo.html?highlight=optuna#power-scale-type-int-only) only works with int, how can sample e.g. a learning rate from the set of options {1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1}?

I tried using 

```
optimizer_kwargs_ranges=dict(
    lr=dict(type=float, low=1e-6, high=1, log=True)
),
```
but I observed unexpected values, e.g. 0.03434, which is not in the set {1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1}.

Thanks. 

q and step are synonyms in this case, just for flexibility since this kwarg was renamed in a recent version of Optuna.

The log scale affects the probability distribution, not the values themselves as described in the [Logarithmic Reweighting](https://pykeen.readthedocs.io/en/stable/tutorial/running_hpo.html?highlight=optuna#logarithmic-reweighting) section of the tutorial.

We talked about having something like what you described, but haven't implemented it for floats, just for ints using the [Power Scale](https://pykeen.readthedocs.io/en/stable/tutorial/running_hpo.html?highlight=optuna#power-scale-type-int-only) for ints . The quick version is to just use the categorical type.

Thanks @cthoyt
Repository: pykeen/pykeen
Issue: What kind of loss does PipelineResult.plot_losses() plot? Training loss, Validation loss, or Testing loss?
Root cause: Documentation Deficiency

Hi, the [doc](https://pykeen.readthedocs.io/en/stable/api/pykeen.pipeline.PipelineResult.html?highlight=plot#pykeen.pipeline.PipelineResult.plot_losses) says that the `PipelineResult.plot_losses()` plot the losses per epoch. But what set is the loss for? Is it the loss on training set, validation set, or testing set? Thanks!

These are the training losses. 

While it is easy to get validation metrics using an early stopper, getting the validation loss is under implementation, cf. https://github.com/pykeen/pykeen/issues/304

The test loss is not computed automatically in order to not accidentally use it for model selection / hyperparameter tuning.

Thanks @mberr :)
Repository: pykeen/pykeen
Issue: Add CITATION.cff file
GitHub labels: documentation

For easy citing: https://twitter.com/natfriedman/status/1420122675813441540

@cthoyt the issue description required a `.cff` instead of a `.bib` one.

Even after reading through the documentation again I didn’t find a very good explanation of how to do a CFF file automatically. It also suggested bibtex as an alternative 

This comment to the tweet linked in the original issue description describes a way how to generate them from a DOI. I have not really tried it myself though.

> ah, nice! [@WDScholia](https://twitter.com/WDScholia) can create CFF for you (feedback welcome).
>
> 1. type https://scholia.toolforge.org/doi/$DOI and replace $DOI with the actually DOI and hit enter
> 2. on the resulting page, add "/export" to the URL (e.g. https://scholia.toolforge.org/work/Q107586230/export)
>
> thx to [@citation_js](https://twitter.com/citation_js)
>
> Egon Willighⓐgen (@egonwillighagen) - [July 28, 2021](https://twitter.com/egonwillighagen/status/1420278201130049537)
Repository: pykeen/pykeen
Issue: NameError: name 'Dataset' is not defined
Root cause: Environment Incompatibility

I am not entirely sure why this is an error, but this is the message i get 

```python-traceback
Traceback (most recent call last):
  File ".\test.py", line 1, in <module>
    from pykeen.pipeline import pipeline
  File "C:\Users\nari9\Anaconda3\envs\CV\lib\site-packages\pykeen-1.5.1.dev0-py3.6.egg\pykeen\pipeline\__init__.py", line 5, in <module> 
    from .api import (
  File "C:\Users\nari9\Anaconda3\envs\CV\lib\site-packages\pykeen-1.5.1.dev0-py3.6.egg\pykeen\pipeline\api.py", line 200, in <module>    
    from ..datasets import get_dataset
  File "C:\Users\nari9\Anaconda3\envs\CV\lib\site-packages\pykeen-1.5.1.dev0-py3.6.egg\pykeen\datasets\__init__.py", line 17, in <module>
    from .base import (  # noqa:F401
  File "C:\Users\nari9\Anaconda3\envs\CV\lib\site-packages\pykeen-1.5.1.dev0-py3.6.egg\pykeen\datasets\base.py", line 67, in <module>    
    class Dataset:
  File "C:\Users\nari9\Anaconda3\envs\CV\lib\site-packages\pykeen-1.5.1.dev0-py3.6.egg\pykeen\datasets\base.py", line 115, in Dataset    
    def triples_sort_key(cls: Type[Dataset]) -> int:
NameError: name 'Dataset' is not defined
```

This looks like a problem of eager evaluation of type annotations. @cthoyt may be able to help :slightly_smiling_face: 

How do I fix this?

Looks like you're on Py36 (would have been nice to have filed this under the "bug" where it instructs to show the `pykeen version` output)

We don't support py36 since it's effectively outdated (end of life is in 4 months, see https://endoflife.date/python). Please upgrade to a more recent version of Python and the problem should resolve itself
Repository: pykeen/pykeen
Issue: When are constrainers and normalizers applied?
GitHub labels: question

I can see that each model's entities and relations have constrainers defined. Where are these constraints applied? In what stage of the training are these constraints applied. For example, bordes et al. states that entities embeddings are normalized in the beginning of each epoch. Does PyKEEN follow the same process? 

The difference between contrainers and normalizers is

https://github.com/pykeen/pykeen/blob/5a24de7d6c42af36ea1a546c10e68634da817852/src/pykeen/nn/emb.py#L204-L206

If you are referring to [Bordes et al., 2013](https://papers.nips.cc/paper/2013/hash/1cecc7a77928ca8133fa24680a88d2f9-Abstract.html), this is corresponding to an L2-constrainer in our terminology.

Great, thank you very much.

On Thu, Jul 29, 2021 at 5:20 PM Max Berrendorf ***@***.***>
wrote:

> The difference between contrainers and normalizers is
>
>
> https://github.com/pykeen/pykeen/blob/5a24de7d6c42af36ea1a546c10e68634da817852/src/pykeen/nn/emb.py#L204-L206
>
> If you are referring to Bordes et al., 2013
> <https://papers.nips.cc/paper/2013/hash/1cecc7a77928ca8133fa24680a88d2f9-Abstract.html>,
> this is corresponding to an L2-constrainer in our terminology.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pykeen/pykeen/issues/557#issuecomment-889467389>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/APAYUWJZWRXCYNIBREPBKSDT2HAX3ANCNFSM5BHHKPDQ>
> .
>
Repository: pykeen/pykeen
Issue: Save best epoch functionality
Root cause: Documentation Deficiency

Is there a "save best epoch" option in pykeen? Basically, I want to keep training but only save the best-performing epoch on the validation set. Alternatively, is it possible to save every epoch with a different name instead of overwriting the old ones? Perhaps a call back is an option? Is there an option that I can use in conjunction with the "pipeline"?

Hi @pitehu,

One approach would be to define an `early stopper` with `patience=num_epochs` and access after finishing training the stopper's `best_epoch` attribute:

```python
stopper = EarlyStopper(
 patience=num_epochs,
 frequency=1,
)

# After training 
best_epoch = stopper.best_epoch
```

If you use the pipeline, you can access the stopper vie the `PipelineResult` that the pipeline returns:

```python
pipeline_result = pipeline(...)
best_epoch = pipeline_result.stopper.best_epoch
```

Alternatively, you could implement your own callback by extending `TrainingCallback`:

```python
class MyCallback(TrainingCallback):
 def post_epoch():
   #TODO
```


More information is also posted here with a more specific solution for this, since a lot of people requested similar things https://pykeen.readthedocs.io/en/stable/tutorial/first_steps.html#training-callbacks
Repository: pykeen/pykeen
Issue: RGCN relation embeddings
Root cause: Documentation Deficiency

I used the RGCN FB15k example and stored the model. How do I get the relation embeddings? Getting the node embeddings is straight forward.
```
my_pykeen_model = torch.load('trained_model.pkl', map_location=torch.device('cpu'))
z = my_pykeen_model.entity_representations[0].entity_embeddings()
```

Same thing but replace `entity_representations` with `relation_representations`. A full guide is available at https://pykeen.readthedocs.io/en/stable/tutorial/first_steps.html#using-learned-embeddings. Let me know if that solves your problem

With `rel_embed=my_pykeen_model.relation_representations[0].relation_embeddings()` I get this error: `torch.nn.modules.module.ModuleAttributeError: 'Embedding' object has no attribute 'relation_embeddings'`

Oh, there was a typo in your original one. The embedding is a module and you just need `my_pykeen_model.relation_representations[0]()`. It should also say this in the docs I sent


Oh thank you, I was a little bit confused since I got some embeddings with my first code.