zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: Improve Graph Sampler performance
Root cause: Architectural Constraint

# Problem

* the expensive step in graph sampling is creating the subgraph
* this is currently implemented inside a batch sampler (cf. https://pytorch.org/docs/stable/data.html )
* the batch sampler *always* runs in the main process, irrespective of the number of workers
* only the `__getitem__` method of the dataset is parallelized -> in our case, this operation is very inexpensive (lookup of a single index in a tensor)

I append a small script to observe the behaviour:
<summary>Script</summary>
<details>

```python
# test sampling
import math
import random
from typing import *

import torch
from torch.utils import data

x = torch.empty(100, 10)
y = torch.empty(100, 1)
batch_size = 10


class DummyBatchSampler(data.BatchSampler):
    def __init__(self, n: int, batch_size: int) -> None:
        self.n = n
        self.batch_size = batch_size

    def __iter__(self) -> Iterator[List[int]]:
        for _ in range(int(math.ceil(self.n / self.batch_size))):
            print("iter", data.get_worker_info())
            yield [random.randrange(self.n) for _ in range(self.batch_size)]


class DummyDataset(data.TensorDataset):
    def __getitem__(self, index):
        print("getitem", index, "worker", data.get_worker_info())
        return super().__getitem__(index)


dataset = DummyDataset(x, y)
batch_sampler = DummyBatchSampler(n=len(dataset), batch_size=batch_size)

dl = data.DataLoader(
    dataset=dataset,
    batch_sampler=batch_sampler,
    num_workers=2,
)


for i, batch in enumerate(dl):
    print("batch", i)

```
</details>


# Solution Proposal
This is not a very clean solution, but should work:

* create an iterable-style new dummy dataset, which maps indices to graph batches, i.e., performs the expensive subgraph sampling procedure inside the `__getitem__`
* adjust the number of sampler per epoch to `number of training instances / batch size` to ensure that we still see the same amount of batches per epoch
Repository: pykeen/pykeen
Issue: Predict Head or Tail candidates using GraphSAGE model
Root cause: Unimplemented Feature Gap

Hi

How to train GraphSAGE model for prediction tasks, like **TransE, RotateE, ComplEx models** etc.

**For example:** After training GraphSAGE on KG predict **head** or **tail** entities using `get_head_prediction_df` or `get_tail_prediction_df` 

Thanks

Hi @geo47 ,

we currently do not have the GraphSAGE encoder implemented in our library. For interaction functions / decoders like TransE, RotatE, ComplEx, please take a look at https://pykeen.readthedocs.io/en/stable/reference/nn/modules.html

[This discussion](https://github.com/pykeen/pykeen/discussions/455) about combining [PyTorch Geometric](https://github.com/pyg-team/pytorch_geometric) and pykeen components may also be helpful.

PyTorch Geometric's implementation of GraphSAGE can be found [here](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.SAGEConv).
Repository: pykeen/pykeen
Issue: Fix epoch loss for reduction != "mean"
Root cause: Algorithmic Error

I think this is caused by https://github.com/pykeen/pykeen/blob/cbcc3b7c2ffa008e1403ea34aa7d3217a2eb39cf/src/pykeen/training/training_loop.py#L664
where `num_training_instances` is the number of instances rather than the number of batches. So the epoch loss should be off by roughly `batch_size`. Would this explain the difference in your example?

P.S.: for mean reduction, this error does not occur, since there is some correction code inside the forward method:
https://github.com/pykeen/pykeen/blob/cbcc3b7c2ffa008e1403ea34aa7d3217a2eb39cf/src/pykeen/training/training_loop.py#L785-L788

_Originally posted by @mberr in https://github.com/pykeen/pykeen/pull/618#issuecomment-963170493_
Repository: pykeen/pykeen
Issue: Second epoch is incorrectly displaying NaN loss
Root cause: Algorithmic Error

> @ralphabb I just merged @mberr's fix into this branch so you can give this another shot

@cthoyt @mberr 4-5 epochs in and the behavior seems to be replicating now! :) A very good sign.
P.S.: A very petty bug. At epoch 2 it seems that the previous loss is printing out as nan instead of the first epoch loss (which is reported earlier).

_Originally posted by @ralphabb in https://github.com/pykeen/pykeen/issues/618#issuecomment-964038774_

https://github.com/pykeen/pykeen/blob/313c705aec2497c3699a6694d5e2189ab113c2ba/src/pykeen/training/training_loop.py#L672-L677

this should be `>1` ?

![](https://www.opi.net/wp-content/uploads/2016/07/1852108-1-eng-GB_staples-easy-button-slider-610.jpg)
Repository: pykeen/pykeen
Issue: Update file-based trackers
Root cause: Architectural Constraint

While revisiting the file-based trackers, I noticed a few issues:

1. `kwargs` in base class are unused
https://github.com/pykeen/pykeen/blob/e8225c066b56bcdd3180ba895ce3e153808e7e38/src/pykeen/trackers/file.py#L60-L61

2. `JSONTracker` does not flush
https://github.com/pykeen/pykeen/blob/e8225c066b56bcdd3180ba895ce3e153808e7e38/src/pykeen/trackers/file.py#L149
For me, the whole result file was not written until the very end of the training, which contradicts the concept of a tracker.

3. `CSVTracker` always flushes
https://github.com/pykeen/pykeen/blob/e8225c066b56bcdd3180ba895ce3e153808e7e38/src/pykeen/trackers/file.py#L119
This seems to be harmless, but maybe there is some use-case where you do not want to have this.

I would suggest that we revive the `kwargs` in the base class, and use it to configure flushing consistently for both trackers. By default, I would suggest to flush.

Solution for flushing seems simple, just add `flush=True` in `print()`. Reference: https://stackoverflow.com/questions/230751/how-can-i-flush-the-output-of-the-print-function
Repository: pykeen/pykeen
Issue: Why don't all epoch losses get returned when using the early stopper?
Root cause: Documentation Deficiency

In the training loop, at [line 756](https://github.com/pykeen/pykeen/blob/master/src/pykeen/training/training_loop.py#L756), we reconstruct the state of the best epoch model if an early stopper has derived a best epoch (but didn't stop early due to patience). This changes `self.losses_per_epochs` so that it only goes until the best epoch, which does make sense since we want to reconstruct the state.

However in returning the training losses, I would assume that it is better default behavior for the training loop to return the training losses for all the epochs, rather than just up to the best epoch. I understand I can make a simple change to get the behavior myself, but I think it would be a good idea to either have, in the case of using an early stopper, the option to return all the training losses or have that be the default behavior.


Hi @LostGeorge, even though you said a line number, it's hard to know what line you meant without a commit hash since this file changes every once and a while. Perhaps next time you can use a permalink with a hash so GitHub can render the exact code. Here's the region you're probably looking at:

https://github.com/pykeen/pykeen/blob/139c3d1db77c9e443d03db5383a163297345adfd/src/pykeen/training/training_loop.py#L754-L764

However, I think your issue is a philosophical one. The point of the early stopper is to avoid extra, unnecessary training. Therefore, when it gets to the "best" epoch (which is defined by the rules in the early stopper), training stops. To report all epochs would mean you didn't actually do early stopping.

An alternative, totally valid scenario, is to add a [training callback](https://pykeen.readthedocs.io/en/stable/reference/training.html#module-pykeen.training.callbacks) that does an evaluation every `n` epochs (similarly to the early stopper), then do a post-hoc analysis. Tangentially, we have two long-standing PRs where we want to convert the early stopping functionality to completely work within the callback framework (see #354/#355).

I'll convert this issue to a discussion, and if you try using the training callbacks to implement this functionality, it would be great for you to share some sample code!
Repository: pykeen/pykeen
Issue: Use nn.CrossEntropyLoss label smoothing instead of our own implementation

cf. [nn.CrossEntropyLoss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)  and [release notes of torch 1.10](https://github.com/pytorch/pytorch/releases/tag/v1.10.0), in particular:
>    nn.CrossEntropyLoss: Added support for class probability targets (pytorch/pytorch#61044)
>    nn.CrossEntropyLoss: Added support for label smoothing (pytorch/pytorch#63122)
Repository: pykeen/pykeen
Issue: Problem with pre-trained embeddings for entity emb. initialization
Root cause: Validation Gap

Thank you, @mberr and @cthoyt  🙂 

I tried your suggestions for my pre-trained embeddings, unfortunately, none of them worked.

## RE: First suggestion

I think the embeddings are reset by the `reset_parameters` function somewhere in the pipeline. To avoid any reset, I had to change the `reset_parameters` and `post_parameter_update` functions by adding the `return` statement before other statements in `emb.py` -  I however still doubt that they are not changed somewhere else.

```python
    def reset_parameters(self) -> None:  # noqa: D102
        return
        # initialize weights in-place
        self._embeddings.weight.data = self.initializer(
            self._embeddings.weight.data.view(self.num_embeddings, *self.shape),
        ).view(self.num_embeddings, self.embedding_dim)
        
    def post_parameter_update(self):  # noqa: D102
        return
        # apply constraints in-place
        if self.constrainer is not None:
            self._embeddings.weight.data = self.constrainer(self._embeddings.weight.data)
```

## RE: Second Suggestion

Regarding the second suggestion, which I found more elegant, it seems that the `create_init_from_pretrained` function doesn't work properly - particularly when it is set within a pipeline. Here's what I did:

1. I set `epochs` to `0`
2. After the run, I checked the model's **entity weights**, and they are different from the pre-trained ones. (There was no training since I set the num. of epochs to `0`.)
- For the resolved first suggestion, the embeddings didn't change in the pipeline when `epochs=0`.

_Originally posted by @alipsgh in https://github.com/pykeen/pykeen/issues/637#issuecomment-969881461_

I re-opened the old issue. Let's continue there.
Repository: pykeen/pykeen
Issue: Documentation about dataset split method: coverage and cleanup
Root cause: Documentation Deficiency

I am confused about the dataset `split` method which is not documented clearly. when I handle my own dataset, 

```
from pykeen.triples import TriplesFactory
from pykeen.pipeline import pipeline
tf = TriplesFactory.from_path('/content/drive/MyDrive/Colab Notebooks/KG/data/spo.csv')
training, testing, validation = tf.split([.9, 0.05, 0.05])

print(training)
print(testing)
print(validation)
```

I changed the ratio to 0.9, 0.05, 0.05( total size = 46500), but the message shows that:

> INFO:pykeen.triples.splitting:done splitting triples to groups of sizes [18856, 2326, 2326]

the ratio is also nearly 0.8:0.1:0.1.

And when I check the train, test, valid, it shows that:

> TriplesFactory(num_entities=27596, num_relations=26, num_triples=41862, inverse_triples=False, path="/content/drive/MyDrive/Colab Notebooks/KG/data/spo.csv")
> TriplesFactory(num_entities=27596, num_relations=26, num_triples=2326, inverse_triples=False, path="/content/drive/MyDrive/Colab Notebooks/KG/data/spo.csv")
> TriplesFactory(num_entities=27596, num_relations=26, num_triples=2326, inverse_triples=False, path="/content/drive/MyDrive/Colab Notebooks/KG/data/spo.csv")

So, there exists detailed documentation about the `split` method?
 

Hi @lmy86263 ,

in `pykeen`, we currently support only the transductive setting out-of-the-box (for efforts towards inductive learning cf. #545, #556 ), where all entities are known from the training set.

The split methods we currently provide thus need to make sure that the triples fulfil this. I agree that this is currently not well documented, and try to roughly explain the two methods here:

#### `cleanup`
https://github.com/pykeen/pykeen/blob/8281200931e26e745876362ce9ad5d82c70b0e83/src/pykeen/triples/splitting.py#L329-L338

The triples are first split randomly. After doing so, there may be validation/test-only entities. Thus, the cleanup method moves triples from validation/test into train until all entities occur in at least one training triple. By doing so, the number of training triples may be increased, and thus the split ratio may be tilted towards training.

#### `coverage`
https://github.com/pykeen/pykeen/blob/8281200931e26e745876362ce9ad5d82c70b0e83/src/pykeen/triples/splitting.py#L339-L351

This method first selects one triple for each entity for the training part, and then splits the remaining triples with an adjusted split ratio. Again, depending on the distribution of entities in the triples, it may not be possible to find a split with exactly the given ratio.

#### Discussion
Interestingly, you seem to observe an unexpected behaviour, where the train part is *smaller* than requested. Could you repeat your script with any of the pre-packaged datasets of `pykeen` (preferably one of the smaller ones) and confirm that you can observe a similar behaviour there?
Repository: pykeen/pykeen
Issue: WandB tracker crashes when used with hpo_pipeline
Root cause: Configuration Error

**Describe the bug**
WandB tracker crashes when used with hpo_pipeline
**To Reproduce**
Steps to reproduce the behavior:


```python
pipeline_result = hpo_pipeline(
    dataset="kinships",
    model="TransE",
    epochs=1000,
    result_tracker="wandb",
    result_tracker_kwargs={"project": "scratch"},
    stopper="early",
    stopper_kwargs={"frequency": 10, "patience": 1, "relative_delta": 0.01},
    n_trials=2,
    n_jobs=1,
)
```

Crashes with:
```
wandb.sdk.lib.config_util.ConfigError: Attempted to change value of key "model.kwargs.embedding_dim" from 96 to 48
If you really want to do this, pass allow_val_change=True to config.update()
```


**Environment:**
| Key             | Value                    |
|-----------------|--------------------------|
| **wandb Version** |  0.12.6 |
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.3.18-lp152.63-default  |
| Time            | Thu Nov 18 18:43:37 2021 |
| Python          | 3.8.12                   |
| PyKEEN          | 1.6.0                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.10.0                   |
| CUDA Available? | true                     |
| CUDA Version    | 11.1                     |
| cuDNN Version   | 8005                     |



Hi @leonweber ,

I think the issue should be fixed with #647 but I did not have the chance to check this myself. If you have some time, it would be great if you could check whether the fix works.

Hi @leonweber ,

the issue should now be fixed (on `master` branch) after #647 is merged. Thanks for reporting the issue!
Repository: pykeen/pykeen
Issue: HelloWorld notebook is out-of-date
Root cause: Documentation Deficiency

**Describe the bug**
The HelloWorld notebook is out-of-date. E.g. 
```python
tf = model.triples_factory
```
fails because PairRE does not have this attribute

**To Reproduce**
Steps to reproduce the behavior:
1. Run the cells in the hello world notebook
2. See error

**Expected behavior**
The hello world notebook should work correctly

**Environment:**
Key | Value
-- | --
OS | posix
Platform | Linux
Release | 5.10.0-8-amd64
Time | Thu Nov 25 11:25:43 2021
Python | 3.9.2
PyKEEN | 1.6.0
PyKEEN Hash | 82812009
PyKEEN Branch | master
PyTorch | 1.10.0+cu102
CUDA Available? | false
CUDA Version | 10.2
cuDNN Version | 7605



**Additional information**
I have opened a PR that adapts the notebook using the documentation given in https://pykeen.readthedocs.io/en/stable/tutorial/making_predictions.html solving the issue


Closed by #649
Repository: pykeen/pykeen
Issue: Use composite actions to reduce duplication of GHA workflows

there is quite a bit of duplicated code in the different types of GHA yamls. Maybe we could reduce the duplication via [composite actions](https://docs.github.com/en/actions/creating-actions/creating-a-composite-action)?

_Originally posted by @mberr in https://github.com/pykeen/pykeen/issues/650#issuecomment-979307418_
Repository: pykeen/pykeen
Issue: Error when using labeled triples in predict_triples_df() when CUDA is available
Root cause: Resource Mishandling

**Describe the bug:** 
Gets runtime error when using predict_triples_df() with labeled triples and CUDA enabled.

**To Reproduce Steps to reproduce the behavior:**
Have CUDA enabled.

Run following example given in predict_triples_df() docs:
```python
from pykeen.pipeline import pipeline
result = pipeline(dataset="nations", model="TransE")
from pykeen.models.predict import predict_triples_df
df = predict_triples_df(model=result.model, triples=("uk", "conferences", "brazil"), triples_factory=result.training, )
```
Results in:
`RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper__index_select)`



**Expected behavior:**
Should work as when CUDA is disabled.


**Environment:**

| Key             | Value                             |
|-----------------|-----------------------------------|
| OS              | posix                             |
| Platform        | Linux                             |
| Release         | 5.10.60.1-microsoft-standard-WSL2 |
| Time            | Sat Nov 27 15:54:12 2021          |
| Python          | 3.8.10                            |
| PyKEEN          | 1.6.0                             |
| PyKEEN Hash     | UNHASHED                          |
| PyKEEN Branch   |                                   |
| PyTorch         | 1.10.0+cu102                      |
| CUDA Available? | true                              |
| CUDA Version    | 10.2                              |
| cuDNN Version   | 7605                              |

**Additional information:**
A quick and dirty fix:

Do a check for CUDA availability at: https://github.com/pykeen/pykeen/blob/4d51c0b9105c9ade7baab3299b4083bc1f228f4d/src/pykeen/models/predict.py#L629

And add `triples = triples.to(device="cuda")` if CUDA is enabled.



Hi @Koenkalle thanks for the feedback. Your solution would do the job, but more generally I think we can map the triples to the same device as the model or the triples factory so we don't have to consider the device itself at all in this code. We'll take care of this when Thanksgiving break is over!

Hi @cthoyt thanks for the quick response! Yes, a general solution sounds a lot better :) 
PyKEEN is a joy to play around with - amazing work!

Hi @Koenkalle thanks for being patient - we had a bit of discussion on how to unit test this that ultimately wasn't successful, but Max was able to manually check that #658 does the job. Please let us know if the problem persists, and thanks again for bringing this up :)
Repository: pykeen/pykeen
Issue: None cuda used when peforming Prediction Task but torch.cuda.is_available is True and cuda is free
Root cause: Documentation Deficiency

**Problem:**
When I am training my models, GPU-Utils is high to 90%. But when peforming Prediction, GPU-Utils is 0 and the task needs unreasonable time to complete(700 hours). Four days ago, similar task just needed 8 hours.

torch.cuda.is_available():True
model.device = 'cuda'  before prediction doesn't work

`        models[i].device = 'cuda'`
`        result = predict.get_head_prediction_df(models[i], relation_label=r_label, tail_label=t_label,triples_factory=data.training,add_novelties=True,remove_known=False).values`

**Environment**
- Python 3.7
- Pytorch 1.9.1+cu102
- Cuda 10.2




THANKS FOR YOUR HELP

Hi @iissomeoneelse ,

have you tried using 
https://pykeen.readthedocs.io/en/stable/api/pykeen.models.Model.html#pykeen.models.Model.to_device_
yet?

Just modifying the variable does not make sure that all components are moved correctly.

A couple notes: 
- This is closely related to #654
- I'd suggest upgrading to a more modern version of Python and PyTorch. With the pending release of PyKEEN 1.7, the PyTorch minimum version will become 1.10
- @mberr I'm getting frustrated that people aren't following the instructions on reporting the environment with a full table, I think I will switch this over to using the new GitHub issue template
Repository: pykeen/pykeen
Issue:  Adding Relation Prediction Loss to LP training
Root cause: Unimplemented Feature Gap

UCL & FAIR recently showed that adding Relation Prediction loss to standard LCWA / SLCWA can slightly increase the link prediction performance. Implementation seems quite straightforward

Paper: https://openreview.net/pdf?id=Qa3uS3H7-Le
Code: https://github.com/facebookresearch/ssl-relation-prediction 

@mberr already did some legwork for this in #602

The referenced paper also contains a new dataset, "Aristo-v4" :pick: 

EDIT: opened an issue, cf. #664 

More follow-up in #728
Repository: pykeen/pykeen
Issue: Add Aristo-v4
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

> As these datasets contain a relatively small number of predicates, we also experiment with
Aristo-v4, the 4-th version of Aristo Tuple KB [Mishra et al., 2017], which has more than 1.6k
predicate

from https://openreview.net/pdf?id=Qa3uS3H7-Le

**Dataset is pre-stratified into train/test/valid**
No. https://openreview.net/pdf?id=Qa3uS3H7-Le describe a split, but did not publish it so far.

**Publication(s) or website describing benchmark KG dataset**:
https://aclanthology.org/Q17-1017.pdf

**URL for downloading the benchmark KG dataset**:
https://allenai.org/data/tuple-kb

https://ai2-public-datasets.s3.amazonaws.com/tuple-kb/aristo-tuple-kb-v4-mar2017.zip

How are we supposed to handle the qualifiers and the confidence measures associated with each triple?

Hm, [the paper](https://openreview.net/pdf?id=Qa3uS3H7-Le) does not provide any details here. I suppose they took `COMBINED-KB.tsv` and only used the three columns `Arg1`, `Quantifier`, and `Arg2`.

Hi @cthoyt and @mberr, thank you for your awesome work on this repo. I am one of the authors of the paper. I just wrapped up the dataset and you can download it from [here](https://dl.fbaipublicfiles.com/ssl-relation-prediction/ds/aristo-v4.zip).

@yihong-chen excellent! Will add it to pykeen later this week :) 

However @yihong-chen I'm skeptical about the longevity of the link you gave. Can you post the data to Zenodo instead?

Sure, [here](https://zenodo.org/record/5942560/files/aristo-v4.zip) it is.
Repository: pykeen/pykeen
Issue: Adding PyKEEN to Optuna project list
GitHub labels: enhancement

As suggested, we're adding a list of projects that include Optuna to our docs. Can you confirm you're ok with this reference and the URL here? https://github.com/optuna/optuna/pull/1605

Thanks!

@Crissman I already wrote on https://github.com/optuna/optuna/pull/1605, so i'm going to close this one here