zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: Better access to and visibility of the hyperparameter values used in a pipeline object
Root cause: Documentation Deficiency

Hi,

I think it would be nice to have more transparency over what hyperparameters were used to train a model when using the `pipline` object -- particularity when no user defined ones are provided and defaults are used. 

This would include everything from model options like embedding dimension, optimizer options like learning rate to things like the number of negative samples used.

Would it be possible to bundle all of these various parameters up in a dictionary and store them in the `PipelineResult` object returned from the `pipeline` function?

As an aside, what is the best way to discovery the default parameters taken by a model/dataset? For example, if I do not set the number of negative samples explicitly, what value is taken? Is it always 1 as suggested by the `BasicNegativeSampler` constructor? Does this change per dataset as well? 

Some of the choices are logged to the result-tracker, e.g., here
https://github.com/pykeen/pykeen/blob/c8f6af64ab1ed75c8ca188eabd8eaa263806057c/src/pykeen/pipeline/api.py#L1032-L1035

A simple work-around would be to always add an additional result tracker, which accumulates all parameters in memory, and adds them to the pipeline result.

Update: here are a few PRs to this end
* [x] #681 
* [x] #682 
* [ ] #685
* [ ] #686 

Thanks @mberr this is pretty much exactly what I was thinking.
Repository: pykeen/pykeen
Issue: Add installation notes for plotting libraries
Root cause: Documentation Deficiency

`matplotlib`, `seaborn` and `word_cloud` are needed for some of the plotting functionality and it would be useful to list them as optional installs in the documentation.

@wwymak documentation updates have made it to the README. when we push the next version to PyPI (before the end of the week) these changes will be reflected for all users
Repository: pykeen/pykeen
Issue: 'TypeError: Object of type TriplesFactory is not JSON serializable' error thrown when saving HPO Pipeline with own dataset
Root cause: Validation Gap


Hi,

I seem to get an error when I am using my own dataset with the `hpo_pipeline` object and attempting to save using the `save_to_directory` function. This error does not seem to happen when using an in-built dataset.  

```python
Traceback (most recent call last):
  File "hpo_run.py", line 80, in <module>
    run_hpo_search(args)
  File "hpo_run.py", line 62, in run_hpo_search
    hpo_pipeline_result.save_to_directory('study')
  File "/home/user/anaconda3/envs/pykeen/lib/python3.8/site-packages/pykeen/hpo/hpo.py", line 327, in save_to_directory
    json.dump(self._get_best_study_config(), file, indent=2, sort_keys=True)
  File "/home/user/anaconda3/envs/pykeen/lib/python3.8/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/home/user/anaconda3/envs/pykeen/lib/python3.8/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/home/user/anaconda3/envs/pykeen/lib/python3.8/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/home/user/anaconda3/envs/pykeen/lib/python3.8/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/home/user/anaconda3/envs/pykeen/lib/python3.8/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/home/user/anaconda3/envs/pykeen/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type TriplesFactory is not JSON serializable
```

What's interesting is that the directory is seemingly actually created as normal after this error message. 

The dataset has been created using the `TriplesFactory.from_path` function as follows:

```python
tf = TriplesFactory.from_path(path=data_location)
training, testing, validation = tf.split([.8, .1, .1], random_state=seed)
```

The `hpo_pipline` has then been run as follows:

```python
hpo_pipeline_result = hpo_pipeline(
    training=training,
    testing=testing,
    validation=validation,
    model=model,
  )
```

**Environment**

python              3.8.5
click               7.1.2
click-default-group 1.2.2
more-click          0.0.1
numpy               1.19.2
pykeen              1.0.6.dev0
torch               1.7.0
tqdm                4.54.1

@sbonner0 thanks for the heads up. Also very much appreciated that your code was sufficient for reproducing the issue. There's a fix implemented in #235 that will get merged after passing CI and review 🥳 

Hi @cthoyt thanks for the prompt fix! 

Would there be any issues for completed runs because of this issue - or is it not something to worry about?

It might be annoying, but this output happens at the end of the entire pipeline so hopefully there's no interference with any other data output
Repository: pykeen/pykeen
Issue: RotatE does not run on gpu
Root cause: Resource Mishandling

RotatE throws a cpu/gpu tensor mismatch error on the optimizer step when running on gpu
```
  File "/home/wwymak/code_experiments/pykeen_expts/hello_world_transe.py", line 19, in <module>
    training_kwargs=dict(num_epochs=100)
  File "/home/wwymak/libraries/pykeen/src/pykeen/pipeline.py", line 722, in pipeline
    **training_kwargs,
  File "/home/wwymak/libraries/pykeen/src/pykeen/training/training_loop.py", line 190, in train
    num_workers=num_workers,
  File "/home/wwymak/libraries/pykeen/src/pykeen/training/training_loop.py", line 370, in _train
    self.optimizer.step()
  File "/home/wwymak/anaconda3/envs/deep_graph/lib/python3.7/site-packages/torch/optim/adagrad.py", line 96, in step
    state['sum'].addcmul_(1, grad, grad)
RuntimeError: expected device cpu but got device cuda:0
```

**To Reproduce**
running this simple code snippet on a gpu machine 
```python
results = pipeline(
        dataset=Kinships,
        model='RotatE',
        random_seed=1235,
        training_kwargs=dict(num_epochs=100)
    )
print(results)
```


**Environment (please complete the following information):**
 - OS: ubuntu 18.04
 - Python version: 3.7
 - Version of this software: 1.0.2dev
 - Versions of required Python packages: # run pip list | grep -Eiw 'torch|numpy|Click|click-default-group|tqdm'
   - Torch: 1.4.0
   - Numpy: 1.18.1
   - Click: 7.1.2
   - Click_default_group:1.2.2
   - Tqdm: 4.46.1

(this error also happens on colab)



Thanks for reporting this issue, @wwymak !

There is now a pull request which will fix this issue: #26.
Repository: pykeen/pykeen
Issue: Add reasonable defaults for NSSA Loss
Root cause: Unimplemented Feature Gap

Right now, the `margin` and `adversarial_temperature` parameters do not have a default for NSSA loss. We need to pick reasonable ones and set it.

From https://arxiv.org/abs/1902.10197, where this loss was proposed, they optimize these hyperparameters in the following ranges:
* self-adversarial sampling temperature `alpha \in {0.5, 1.0}`
* margin `gamma in {3, 6, 9, 12, 18, 24, 30}.`

The chosen parameters by the authors for FB15k, FB15k-237, WN18, WN18RR can be found at https://github.com/pykeen/pykeen/tree/master/src/pykeen/experiments/rotate :

| dataset | alpha | gamma |
| -- | -- | -- |
| FB15k | 1.0 | 24 |
| FB15k-237 | 1.0 | 9 |
| WN18 | 0.5 | 12 |
| WN18RR | 0.5 | 6 |

So any of these configurations may be a good choice for a default value.
Repository: pykeen/pykeen
Issue: Reorganize device handling
Root cause: Architectural Constraint

```python
{
  t.device 
  for t in itertools.chain(self.parameters(), self.buffers())
}
```
yields the set of device on which parameters and buffers bound to this Module are located. This *may* be better since we do not explicitly have a device to pass around, but use the device of the module.

With `model.to(device)` we can automatically move all its parameters and buffers to another device, and get `self.device` updated "for free".

_Originally posted by @mberr in https://github.com/pykeen/pykeen/pull/257#discussion_r565488505_
Repository: pykeen/pykeen
Issue: Add TransG
GitHub labels: 💃 Model

💎 New Component

**On which paper(s) is your requested model based.**

- arXiv: https://arxiv.org/abs/1509.05488
- PDF: [P16-1219.pdf](https://github.com/pykeen/pykeen/files/5888528/P16-1219.pdf)

**List the original and other existing implementations of this model**

Author implementation (numpy): https://github.com/Niubohan/TransG/blob/master/TransG.py

**Relevance of this model**

Tbh i don't think this one is relevant. It was published in 2016 and went directly to obscurity.
Repository: pykeen/pykeen
Issue: Creating Inverse Triples in Validation and Test Factories
Root cause: Validation Gap

It seems that in the current version we by mistake allow to create inverse triples for the validation and test sets:

https://github.com/pykeen/pykeen/blob/4c532b920b66069d63dd1469ec3c310dd34fc1d7/src/pykeen/datasets/base.py#L241-L262

+1 , found it in `1.1.0` when experimenting with a plain DistMult on FB15k237 with default loops/optimizers/losses. Essentially, the numbers show as if the model does not learn anything. Apparently, in the dataset with already created inverse relations the evaluation creates additional `r+1` on the fly? But then I wonder why it does not crash as there should be only 237 * 2 = 474 rows in the embedding matrix 🤔
Repository: pykeen/pykeen
Issue: Improve Modularization of R-GCN
GitHub labels: enhancement

necro

☠️ R-GCN ☠️

Make Block and Bases Decomposition are individual modules so the R-GCN class is agnostic to the chosen relation matrix decomposition.

Move the PR from https://github.com/mali-git/POEM_develop/pull/529


@mberr this is probably done
Repository: pykeen/pykeen
Issue: Improve negative triple filtering performance
Root cause: Algorithmic Error

Filtering is currently painfully slow, when filtering true positives from the sampled negatives.

**Describe the solution you'd like**
Improving the performance of filtering true negatives could help, comparing the evaluation results to f.e. dglke (https://github.com/awslabs/dgl-ke), where the default number of negative samples is the number of unique entities in the graph. Without filtering, evaluation results get significantly distorted, when using this many negative edges.

**Additional context**
As the max limit of integer in python is very generous, positive triples could be hashed into a single integer
``` python
positive_hash = positive[:,1] * (num_rel ** 2) + \
                positive[:,0] * (num_ent ** 1) + \
                positive[:,2] * (num_ent ** 0)
```
where num_rel is the number of unique relations in the graph and num_ent is the number of unique entities. ((num_ent ** 0) is redundant as this is always 1). `positive` and `negative` are tensors of size (n,3).

I've experimented with two versions:

Version 1 uses torch functions:

``` python
negative_hash = negative[:,1] * (num_rel ** 2) + \
                negative[:,0] * (num_ent ** 1) + \
                negative[:,2] * (num_ent ** 0)

version1 = negative_hash.unsqueeze(1).eq(positive_hash).any(-1)
```

The second version uses a standard python set, not very beautiful, but the most performant version:
``` python

negative_hash = negative[:,1] * (num_rel ** 2) + \
                negative[:,0] * (num_ent ** 1) + \
                negative[:,2] * (num_ent ** 0)

set_positive_hash = set(positive_hash.tolist())

version2 = torch.zeros(len(negative), dtype=torch.bool)
for i in range(len(negative)):
    if negative_hash[i].item() in set_positive_hash:
        version2[i] = True
```

Comparison when running on 600,000 positive and 15,000 negative triples:

| Pykeen    | Version 1 | Version 2 |
|-----------|-----------|-----------|
| 231.5241s | 5.7347s   | 0.2646s   |

However, I cannot vouch for the correctness of both versions, as comparison with your filtering method failed, due to (what seems to me) a bug in your code (see #272)

Both above versions would fail if `num_rel^3 + num_ent^2 + num_ent > sys.maxsize` (which is for me 9223372036854775807 and thus could easily support bigger benchmarks like OpenBioLink)






Hi @nomisto ,

thanks again for your interest in negative sampling. We will take a closer look at the suggestions, also benchmarking them for GPU :slightly_smiling_face: ( with @lvermue ).

One small comment to

> [...] where the default number of negative samples is the number of unique entities in the graph. 

This is quite similar to training with 1:n scoring / the local closed world assumption, cf. [here](https://pykeen.readthedocs.io/en/latest/reference/training.html#local-closed-world-assumption). While it not necessarily gives better results than the stochastic LCWA in terms of link prediction quality, it is quite fast (compared to similarly large number of negative samples).
Repository: pykeen/pykeen
Issue: Add DBpedia dataset 
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I have been using TripleFactory and pointing at the local dataset path when I need to train or test on DBpedia50 for the last couple of months. Now I want to run hyperparameter optimization but that requires the dataset to be registered. 

**Describe the solution you'd like**
I'd like to add the dataset to pykeen datasets, but when I looked at this [page](https://pykeen.readthedocs.io/en/latest/reference/datasets.html), I am not sure how to make these setup files, could you please add more detailed instructions so we can add the dataset more easily? 

**Additional context**
DBPedia50 dataset is available [here](https://github.com/ZhenfengLei/KGDatasets) 


Hi @PhaelIshall, I'll look into adding DBpedia50 as a benchmark data set and update the documentation so this is more obvious how to do it yourself

@cthoyt In DBpedia50 the files contain (head, tail, relation) rather than the usual (head, relation, tail). I saw that you were using the data from the link directly and I thought maybe you were not accounting for that. 

@PhaelIshall oh dear, thanks for pointing this out. Will fix immediately

@PhaelIshall everything should be good now, let me know if there are any other problems (I tested it this time 😆 )
Repository: pykeen/pykeen
Issue: Make initialization more configurable
Root cause: Architectural Constraint

> @mali-git no, since we don't want users to specify models this way. The model classes like TransE expose only the things that are considered as hyperparameters to each model. if the user wants full configurability, then they can implement a new model class. However, what you're suggesting becomes much more easy for users with new-style models

This would mean that users for example cannot try different initialization methods. I would opt for as much configuratibility as possible also for existing models.

_Originally posted by @mali-git in https://github.com/pykeen/pykeen/issues/277#issuecomment-776019402_
Repository: pykeen/pykeen
Issue: how to load the model from checkpoint?
Root cause: Documentation Deficiency

Since we could use checkpoint technique, and also get the my_checkpoint.pt this file. I want to know, could I resume the training from this file? That I could load the model trained in the file?

I think there's an example in the tutorial at https://pykeen.readthedocs.io/en/stable/tutorial/checkpoints.html on how to do this. Basically the first time you run the code, it doesn't find the checkpoint file and starts from the beginning. The second time you run the code, it finds the same file (since the config is the same), loads it, and keeps training. Let me know if that works for you. If not, we can ping the PyKEEN team member who's responsible for this feature
Repository: pykeen/pykeen
Issue: Is there anyway to resume an ablation study? 
GitHub labels: documentation

🦆 Ablation


Hi All,

I was wondering is there anyway to resume an ablation study if it gets interrupted for any reason? It seems a shame to have to repeat all the experiments if they have already been completed and logged in the directory structure.

I had a quick look at the code and couldn't see any obvious mechanism in place to handle this....

The ablation code in PyKEEN is taken care of by @mali-git. I think this issue could be a subtask of #116.

@sbonner0 by the way, since you're working in the life sciences, I'm quite interested to hear about what you're up to. If you've got some time to chat next week on zoom/google/etc, shoot me an email cthoyt@gmail.com

Great thanks for the offer, i'll drop you a mail.

Regarding the ablation issue – due to the way all the directories are made beforehand for all the runs, I don’t think it would be too difficult to keep track of what has and hasn’t been run hopefully. It might even be possible to allow optuna to resume trails within a given ablation trial. 

Some related thoughts on the ablation tool as it is.

- Is there a way to set the seed for the dataset? I couldn't see an obvious one as dataset_kwargs doesn't seem to be an argument for the ablation_pipeline function as it is for others. 

- It would be nice if there was a way to pass global settings for parameters like learning rate without having to specify the same thing per model and trainer all the time.
Repository: pykeen/pykeen
Issue: Set default value of Evaluator's parameter filtered to True
Root cause: Configuration Error

**Is your feature request related to a problem? Please describe.**
Since the filtered evaluation is the de facto standard in KG link prediction literature, the default should coincide. Also,  [the documentation](https://pykeen.readthedocs.io/en/latest/tutorial/understanding_evaluation.html#filtering) says so.

**Describe the solution you'd like**
Set the default of `filtered` to `True`, for the evaluator (and thus also for the pipeline).

**Describe alternatives you've considered**
Leaving it as it is, thus maintaining backwards compatibility. However, the old behaviour is less expected, and contradicts (parts of) the documentation.

**Additional context**
Related: #298 


I support this change

Okay, for the rank based evaluator, this seems to be the default

https://github.com/pykeen/pykeen/blob/f793d28b92dbc4478a551e6117ba0fd6d6d84139/src/pykeen/evaluation/rank_based_evaluator.py#L222

It is only the general Evaluator class where it is set to False by default. This may be a reasonable default value.

~Hmm, in the parent Evaluator class, it's set to false, but the only other evaluator class we have is Scikit learn, and it's not used at all. Perhaps we should remove filter from the parent class completely~ sorry this was wrong, there are shared features in the evaluator class that use this.

@cthoyt as far as I see it, everything is correct as it is right now. If you agree feel free to close the issue (and the related PR).
Repository: pykeen/pykeen
Issue: Improve the doc
GitHub labels: documentation

Currently, it is difficult to understand the data structure (i.e. what I am passing to the model). 
It may be useful to add a description page to help in developing custom models and datasets and therefore, extend the framework.

Furthermore, I think that an example based on Graph Neural Networks (maybe with PyTorch Geometric) will be very appreciated.

Hi @robertanto, thanks for being patient. We've just put out the new PyKEEN v1.0 today, with a complete re-write of everything. There's much more documentation (though it still has a long way to go since there are so many features now)

You can check out the [`pykeen.pipeline` tutorial](https://pykeen.readthedocs.io/en/latest/first_steps.html) which will get you training your first model in a few lines of code, then you can start reading the [`pykeen.pipeline`](https://pykeen.readthedocs.io/en/latest/reference/pipeline.html#pykeen.pipeline.pipeline) reference to learn more, and lead yourself to all of the tricky configurations possible for each model, loss function, regularizer etc.

At this point, your feedback will be valued greatly, as we're moving towards doing a documentation sprint.

I'm not sure about Graph Neural Networks... @sharifza has implemented R-GCN for PyKEEN 1.0 so you can direct any specific questions about that to him. Since PyKEEN isn't really a framework for building new tools like PyTorch or PyTorch Geometric, there might be a misunderstanding. You can always use those tools to implement models then make them fit the interface for PyKEEN (in the mean time, PyTorch lightning was published which does something very similar to PyKEEN)

Hi @robertanto, I think you have mistaken Knowledge Graph embedding models with Graph Conv Neural Networks. Although as @cthoyt mentioned, R-GCN is based on Graph Conv Neural Networks, I'm not sure if your question is directed specifically to that model. If my assumption is right, I would direct you to [this](https://arxiv.org/pdf/1503.00759.pdf) and [this](https://arxiv.org/abs/2006.13365) paper.

Thank you both @sharifza and @cthoyt !

Yes, for me the R-GCN model is a sufficient example.
Repository: pykeen/pykeen
Issue: How to get loss on validation data?
Root cause: Unimplemented Feature Gap

I am using an LCWATrainingLoop. After researching the source code I still cannot find a way to get the validation loss during training. Could you provide any hints on how to achieve this?

Currently lcwa_training_loop.train() only returns the loss over the training data for each epoch.


Would you please provide an example of the code you're using so we can give the best answer?

Hi @46692,

Currently, we support computing the ranking metrics, the area under the ROC curve, and the area under the precision-recall curve on the validation set. For now, this can be achieved by using the early stopper (and tracking the results by one of the trackers defined in `pykeen.trackers`) and defining the patience equal to the number of epochs:

```python
stopper = EarlyStopper(
              ...
              evaluation_triples_factory=validation_tf,
              metric='hits_at_10',
              frequency=1,
              patience=num_epochs,
          )
```

We will address your feature request soon (please see  #305).

> Would you please provide an example of the code you're using so we can give the best answer?

I am currently training two models, DistMult and an altered version of DistMult.
I want to compare their trainings via the loss on the validation set.
The following is the code I am using for the training, which is mostly taken from the Pykeen Quick Start tutorial:
```python
def training(model, dataset):
    from torch.optim import Adam
    optimizer = Adam(params=model.get_grad_params())

    from pykeen.training import LCWATrainingLoop
    training_loop = LCWATrainingLoop(model=model, optimizer=optimizer)

    losses = training_loop.train(num_epochs=EPOCHS, batch_size=BATCH_SIZE)

    from pykeen.evaluation import RankBasedEvaluator
    evaluator = RankBasedEvaluator()

    # Evaluate
    mapped_triples = dataset.testing.mapped_triples
    eval_results = evaluator.evaluate(model, mapped_triples, batch_size=128)
```

> Currently, we support computing the ranking metrics, the area under the ROC curve, and the area under the precision-recall curve on the validation set. For now, this can be achieved by using the early stopper (and tracking the results by one of the trackers defined in `pykeen.trackers`) and defining the patience equal to the number of epochs:
> We will address your feature request soon (please see #305).

The EarlyStopper approach is very usefull too, but sadly no replacement for my requirement.
Thanks!


@46692 okay I think we know what needs to be done! we'll keep you updated and link you when we have a prototype PR

The training callbacks are quite extensible for this kind of thing now - see https://pykeen.readthedocs.io/en/stable/tutorial/first_steps.html#training-callbacks