zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: Why FB15k-237 / WN18RR dataset is slightly different from the other versions?
Root cause: Documentation Deficiency

### Why FB15k-237 / WN18RR dataset is slightly different from the other versions?

I find out that the dataset statistics of FB15k-237 is
```
statistics:
    entities: 14505
    relations: 237
    training: 272115
    testing: 20438
    validation: 17526
    triples: 310079
```
However, the number of entities, validation split, and testing split in another version of the dataset (widely used in many recent works) is 14541, 17535, and 20466, respectively (See the [original paper](https://aclanthology.org/W15-4007.pdf)  of FB15k-237). Such a difference also happened on WN18RR. I have checked the source that appeared in the code ([source](https://download.microsoft.com/download/8/7/0/8700516A-AB3D-4850-B4BB-805C515AECE1/FB15K-237.2.zip)), and the dataset statistics is aligned with the widely used version. So, what is the reason causing this difference? Would it influence the reproduction of different models?

### Environment

my pykeen version is "1.8.0"

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Hi @chenxran ,

some of the benchmark datasets contain entities in the validation/test set, which are not present in the training set. For PyKEEN, we remove these entities when mapping the triples. In the output of your script, you should also see a message about this, from here

https://github.com/pykeen/pykeen/blob/8fdceb7e32fb2a3912d09b5c9e694f9a21d2b70d/src/pykeen/triples/triples_factory.py#L121-L132

EDIT: cf. https://github.com/pykeen/pykeen/issues/105 

@mberr  What if we need those filtered testing triples for some additional evaluation. At the moment I am loading them manually after making prediction with pykeen because they get filtered out. Is there a more straightforward way? 

You would need to ensure that here
https://github.com/pykeen/pykeen/blob/63c40ba188c332d47c0beca02233165ea1513c4d/src/pykeen/datasets/base.py#L659

the entity/relation to Id mapping include the non-train entities.
Repository: pykeen/pykeen
Issue: Predict Problem
Root cause: Documentation Deficiency

### What is your question

When I want to get some number of predictions，I found these function really cost a lot.
I tried to get 100000 triples, I tried to use ` predict.get_all_prediction_df`, but it cost  a lot  of gpu memory and time.
I also tried to run `get_tail_prediction_df` or `get_head_prediction_df` for 100000 loop, and also cost a lot of time.
 I wonder if you could know  easy way to achieve my goal.

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 4.15.0-163-generic       |
| Time            | Sat May  7 22:56:48 2022 |
| Python          | 3.9.12                   |
| PyKEEN          | 1.6.0                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.11.0+cu113             |
| CUDA Available? | true                     |
| CUDA Version    | 11.3                     |
| cuDNN Version   | 8200                     |

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Hi @lovefxyo ,

the different prediction methods are summarized here: https://pykeen.readthedocs.io/en/stable/tutorial/making_predictions.html

From your question, it seems like you want to calculate scores _for a given set of evaluation triples_, correct? In that case, https://pykeen.readthedocs.io/en/stable/api/pykeen.models.predict.predict_triples_df.html might be what you are looking for.
Repository: pykeen/pykeen
Issue: Add PrimeKG
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

New biomedical KG from Harvard called the Precision Medicine Knowledge Graph (PrimeKG).

**Dataset is pre-stratified into train/test/valid** []

**Publication(s) or website describing benchmark KG dataset**: https://doi.org/10.1101/2022.05.01.489928

**URL for downloading the benchmark KG dataset**: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/IXA7BM
Repository: pykeen/pykeen
Issue: InfoNCE Loss
Root cause: Algorithmic Error

The InfoNCE loss was defined in ["SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models" from Wang *et al*, 2022](https://arxiv.org/abs/2203.02167) as:

<img width="308" alt="Screen Shot 2022-05-15 at 11 43 05" src="https://user-images.githubusercontent.com/5069736/168466613-232febdf-81af-432f-bafe-4d9f901b6b4a.png">

Context:

> The additive margin γ > 0 encourages the model to increase the score of the correct triple (h,r,t).
φ(h, r, t) is the score function for a candidate triple, here we define φ(h, r, t) = cos(e_hr, e_t) ∈ [−1, 1]

This looks almost like [CrossEntropyLoss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) on transformed scores
```python
score = (score - gamma) / tau
```
except for the last term in the denominator 🤔 

The [official code](https://github.com/intfloat/SimKGC) also uses [CrossEntropyLoss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html), cf.
https://github.com/intfloat/SimKGC/blob/4388ebc0c0011fe333bc5a98d0613ab0d1825ddc/trainer.py#L36
https://github.com/intfloat/SimKGC/blob/4388ebc0c0011fe333bc5a98d0613ab0d1825ddc/trainer.py#L113-L117

Here is the computation of logits:
https://github.com/intfloat/SimKGC/blob/4388ebc0c0011fe333bc5a98d0613ab0d1825ddc/models.py#L86-L115

They also refer to ["A Simple Framework for Contrastive Learning of Visual Representations" by Chen et al. 2020](http://proceedings.mlr.press/v119/chen20j.html) for the loss function; there they do not have an additive `-gamma` term.
Repository: pykeen/pykeen
Issue: Use of LongTensors instead of IntTensor for MappedTriples
Root cause: Architectural Constraint

### Describe the bug

In the [CoreTriplesFactory](https://github.com/pykeen/pykeen/blob/3616656d3a95f6529f81e3e5bf4b81753d2f8cf6/src/pykeen/triples/triples_factory.py#L186) and generally speaking as type for the [MappedTriples](https://github.com/pykeen/pykeen/blob/3616656d3a95f6529f81e3e5bf4b81753d2f8cf6/src/pykeen/typing.py#L47) a [torch.LongTensor](https://pytorch.org/docs/stable/tensors.html) is used.

This allows for up to `2^63` nodes and edge types since it uses a signed integer with 64 bits, using 24 bytes for each row.

This, I believe, is a significant overkill when considering the applications of the library (usually biological graphs have less than a couple of billion nodes, I am not aware of one of that size), and therefore a `torch.IntTensor` object, allowing for up to `2^31` nodes and edge types would be quite sufficient, **requiring half the memory**.

If there is no particular need for using 64-bits, consider switching the tensor type.

### How to reproduce

/

### Environment

/

### Additional information

_No response_

Hi @LucaCappelletti94 (and @zommiommy) ,

thanks for the suggestion. We currently stick with long tensors, since this is the default index data type, PyTorch uses internally, e.g., as return-type of [torch.nonzero](https://pytorch.org/docs/stable/generated/torch.nonzero.html), [torch.Tensor.scatter_](https://pytorch.org/docs/stable/generated/torch.Tensor.scatter_.html#torch.Tensor.scatter_), [torch.nn.functional.one_hot](https://github.com/pytorch/pytorch/issues/53785), to name a few.

While I agree that a different index datatype might save some memory (and thereby may also accelerate operations which need to move index-like data to GPUs), we do not have evidence that this is something which slows PyKEEN down. Moreover, the majority of memory requirement typically stems from the models parameter tensors rather than the triple indices.

Still, if you can demonstrate the advantages in a practical use case, and find a maintainable implementation, we are happy to accept a PR.
Repository: pykeen/pykeen
Issue: Add example of loading a model from a pickle to "Novel Link Prediction" tutorial
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
I am trying to run experiments from a landmark. When the experiment runs, the results are config files and pkl file. There are files that contain MRR and Hits@1,3,10.

**Describe the solution you'd like**
I would like to see the predictions, not just the aggregate numerical results (like predictions for all triples). It is not clear how to do that from this point (especially after spending a lot of time training the model, then not being able to access the predictions).

**Describe alternatives you've considered**
Would it be possible to load from the pkl file and make predictions? I tried this but it did not work out. Please provide an example so this becomes more clear. 

**Additional context**
Add any other context or screenshots about the feature request here.


Hi @PhaelIshall

> Would it be possible to load from the pkl file and make predictions? I tried this but it did not work out. Please provide an example so this becomes more clear.

In https://pykeen.readthedocs.io/en/latest/tutorial/making_predictions.html, we explain how to use a trained model to make predictions. A full example of how to load a serialized model and to make predictions can be found at:

https://github.com/pykeen/pykeen/issues/75#issuecomment-675373744

If things are not clear, please let us know!

@mali-git this tutorial doesn't have an explicit example of loading a pickle. I will add that

> @mali-git this tutorial doesn't have an explicit example of loading a pickle. I will add that

Thanks @cthoyt! 

@PhaelIshall please let me know if the updated documentation from #94 helps you solve your problem. Greetings from Bonn, Germany!

@cthoyt Yes I see it, I think I can use this. Danke Schön!
Repository: pykeen/pykeen
Issue: Device error for filtered negative samples for CrossEntropy and NSSALoss
Root cause: Resource Mishandling

### Describe the bug

Ran into an issue that wasn't fixed with #801. Still getting the two device error referenced in #799 when using CrossEntropyLoss or NSSALoss due to tensor device mismatches at `prepare_negative_scores_for_softmax` in [losses.py](https://github.com/pykeen/pykeen/blob/d7ee986026065997398ab72152f0365224d9e374/src/pykeen/losses.py#L1056)

```
/gpfs/milgram/project/rtaylor/vs428/conda_envs/wikem_env/lib/python3.8/site-packages/pykeen/losses.py in prepare_negative_scores_for_softmax(batch_filter, negative_scores, no_inf_rows)
   1082     # (dense) softmax requires unfiltered scores / masking
   1083     negative_scores_ = torch.zeros_like(batch_filter, dtype=negative_scores.dtype)#, device=negative_scores.get_device())
-> 1084     negative_scores_[batch_filter] = negative_scores
   1085     # we need to fill the scores with -inf for all filtered negative examples
   1086     # EXCEPT if all negative samples are filtered (since softmax over only -inf yields nan)

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
```

### How to reproduce

```python
from pykeen.pipeline import pipeline

result = pipeline(
    dataset="nations",
    model="RotatE",
    loss="CrossEntropyLoss",
    training_loop="sLCWA",
    negative_sampler="basic",
    negative_sampler_kwargs=dict(filtered=True),
    model_kwargs=dict(embedding_dim=16),
)
```

### Environment

| Key             | Value                       |
|-----------------|-----------------------------|
| OS              | posix                       |
| Platform        | Linux                       |
| Release         | 3.10.0-1160.45.1.el7.x86_64 |
| Time            | Sun May 22 21:13:06 2022    |
| Python          | 3.8.12                      |
| PyKEEN          | 1.8.2-dev                   |
| PyKEEN Hash     | UNHASHED                    |
| PyKEEN Branch   |                             |
| PyTorch         | 1.11.0                      |
| CUDA Available? | true                        |
| CUDA Version    | 11.1                        |
| cuDNN Version   | 8005                        |

### Additional information

_No response_

Fixed this, will submit a PR request in a few minutes! 

closed by #936
Repository: pykeen/pykeen
Issue: Load saved training result
Root cause: Documentation Deficiency

### What is your question


1. I've created a new dataset using TriplesFactory : 

```python
dataset = "mydataset.csv"
tf = TriplesFactory.from_path(dataset)
```

2. I've splitted it and saved the result using  

```python
training, testing = tf.split()
training.to_path_binary("training.csv")
testing.to_path_binary("testing.csv") 
```

3. training the model 
```python
result = pipeline(
        training=training,
        testing=testing,
        model='TransE',
        epochs=228, 
    )
result.save_to_directory("results")
```
The results are stored in a "results" directory.

Now I want to load the results from "results" directory but I cannot find a method like `from_path_binary()` to do that. 
I found that it is possible to load the pipeline from the configuration file using `pykeen.pipeline.pipeline_from_path()`. But, how do I generate that file for my custom data set?

### Environment

![2022-05-25_21-16](https://user-images.githubusercontent.com/733580/170350278-64b52cfa-71fd-44d9-a22e-5805e71d7e9b.png)


### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Currently, we do not have a method to load the results back into a `PipelineResults` object, since depending on the choices when saving the result, not all components are saved. However, you can load all individual components with the structure of the serialization format being described here: https://pykeen.readthedocs.io/en/stable/api/pykeen.pipeline.PipelineResult.html#pykeen.pipeline.PipelineResult.save_to_directory

Thank you @mberr.
I already use `torch. load` to load the model, but I want to access to the results also.
I wonder if the checkpoint might be used to reload the results whenever needed?

With results you are referring to the calculated metrics, e.g. hits@k, etc, ? If yes, they should be in `results.json` just next to the saved model.

Thank you. 
For now, I'm using checkpoints to save results.
Repository: pykeen/pykeen
Issue: plot_er reducer arguments crash plots on non-string values
Root cause: Validation Gap

### Describe the bug

In [L181](https://github.com/pykeen/pykeen/blob/a4a150ff40430d5349d982c90efb05f199ab1444/src/pykeen/pipeline/plot_utils.py#L181) of `plot_er`, when a non-string character is passed as a named argument to be sent to the reducer class function (PCA, t-SNE, etc.) the plotting function crashes.

### How to reproduce

```python
import os

import numpy as np
from matplotlib import pyplot as plt

from pykeen.pipeline import pipeline
from pykeen.triples import TriplesFactory

nations_results = pipeline(
    dataset='Nations',
    model='TransE',
    model_kwargs=dict(embedding_dim=8),
    optimizer_kwargs=dict(lr=1.0e-1),
    training_kwargs=dict(num_epochs=80, use_tqdm_batch=False),
    evaluation_kwargs=dict(use_tqdm=False),
    random_seed=1,
    device='cpu',
)

nations_results.plot_er(
    relations={'treaties'},
    apply_limits=False,
    plot_relations=True,
    tol=0.3
)
```

### Environment

| Key             | Value                       |
|-----------------|-----------------------------|
| OS              | posix                       |
| Platform        | Linux                       |
| Release         | 3.10.0-1160.45.1.el7.x86_64 |
| Time            | Fri May 27 17:46:04 2022    |
| Python          | 3.8.13                      |
| PyKEEN          | 1.8.2-dev                   |
| PyKEEN Hash     | UNHASHED                    |
| PyKEEN Branch   |                             |
| PyTorch         | 1.11.0+cu102                |
| CUDA Available? | false                       |
| CUDA Version    | 10.2                        |
| cuDNN Version   | 7605                        |

### Additional information

Opening a PR now to fix this!
Repository: pykeen/pykeen
Issue: Add HousE
GitHub labels: 💃 Model

💎 New Component

### Publication Link

https://arxiv.org/abs/2202.07919

### Reference Implementation

https://github.com/anrep/HousE

### Additional Implementations

_No response_

### Relevance

[accepted](https://icml.cc/Conferences/2022/Schedule?showEvent=17408) at ICML’22

### Additional Context

The model is defined [here](https://github.com/anrep/HousE/blob/main/codes/model.py). In paper the discuss in total 6 different variants: HouseE, HouseE+, HouseE-r, HouseE-r+, HousH, and HousR. From the results, HousE+ seems to be superior in performance (on WN18RR and FB15k-237).
Repository: pykeen/pykeen
Issue: Reproduce Transe experiment
Root cause: Architectural Constraint

### Describe the bug

Transe experiment files not in the correct folder, they are placed with convkb and therefore cannot be executed 

### How to reproduce

```
pykeen experiments reproduce transe nguyen2018 fb15k237
pykeen experiments reproduce transe nguyen2018 wn18rr
```

This will work once the files are moved

### Environment


| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.4.0-45-generic         |
| Time            | Tue May 31 16:10:10 2022 |
| Python          | 3.8.10                   |
| PyKEEN          | 1.8.3-dev                |
| PyKEEN Hash     | 825e3101                 |
| PyKEEN Branch   | master                   |
| PyTorch         | 1.11.0+cu102             |
| CUDA Available? | true                     |
| CUDA Version    | 10.2                     |
| cuDNN Version   | 7605                     |


### Additional information

_No response_

Closed by #957
Repository: pykeen/pykeen
Issue: Add LineaRE
GitHub labels: 💃 Model

💎 New Component

### Publication Link

https://arxiv.org/abs/2004.10037

### Reference Implementation

https://github.com/pengyanhui/LineaRE

### Additional Implementations

_No response_

### Relevance

Has been published on [ICDM2020](http://icdm2020.bigke.org/), and seems very easy to implement.
It can be seen as a mixture of PairRE (relation-specific head/tail Hadamard product) & TransE (additive relation term).

### Additional Context

_No response_
Repository: pykeen/pykeen
Issue: Entity and relation representation advice 
GitHub labels: question

### What is your question

Hi! I'm working on producing a KG embedding on a text corpus. My main doubt is about the best practice for storing/using the embeddings produced by PyKEEN.
Should I:
Save the entity and relation embeddings in a file in the GloVe style? If so how do I recover the mapping between embeddings and words? (This wasn't clear from the documentation)
Use the BERT model to transform labels into embeddings and call the model every time I need an embedding?
Use Node Piece in a similar fashion?
Thanks! 

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.17.5-76051705-generic  |
| Time            | Tue Jun 14 09:30:45 2022 |
| Python          | 3.9.12                   |
| PyKEEN          | 1.8.2                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.11.0+cu102             |
| CUDA Available? | false                    |
| CUDA Version    | 10.2                     |
| cuDNN Version   | 7605                     |


### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

The first steps tutorial has some specific examples on how to save a model https://pykeen.readthedocs.io/en/stable/tutorial/first_steps.html#training-a-model and loading a pre-trained model (i.e., after you saved it) https://pykeen.readthedocs.io/en/stable/tutorial/first_steps.html#loading-a-pre-trained-model

The embeddings are inside the model, but you can feel free to access them and save them directly if you want

@cthoyt :
Thanks for your details.

More interesting to extracting the embedding for each node.
It seems that
  the embeddings are under model.entity/relation_representations
  the thing that lacks is a mapping between which entity/relation corresponds to each embedding.
  
  is there a way to extract it smoothly ? (some idea of code) ?
  
  


I'd highly suggest reading the tutorials in full as well as looking at past questions that match yours.

The section you're probably looking for is https://pykeen.readthedocs.io/en/stable/tutorial/first_steps.html#using-learned-embeddings

This is the part regarding mapping entity/relation labels to their integer IDs: https://pykeen.readthedocs.io/en/stable/tutorial/first_steps.html#mapping-entity-and-relation-identifiers-to-their-names
Repository: pykeen/pykeen
Issue: Add max_id to all representations
Root cause: Unimplemented Feature Gap

another option, I intend to add later on, is to have `max_id` / `shape` be in the `__init__` parameters any time, and use that for verifying inferred ones.

_Originally posted by @mberr in https://github.com/pykeen/pykeen/pull/980#discussion_r900026645_
Repository: pytorch/benchmark
Issue: SRU_Compute_No_Kernel references an undefined variable
Root cause: Validation Gap

Sigh. Don't have time to submit a PR. Looks like `ncols`, used [here](https://github.com/pytorch/benchmark/blob/c33838af059e48741f0e68f2a187c6b44fdcee0a/benchmarks/sru.py#L458), is undefined.

Thanks, we'll take a look.

Cleaning up benchmark repo, closing old issues, please reopen if needed.
Repository: pytorch/benchmark
Issue: V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220630+cu113'
Root cause: External Dependency Breakage


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220628+cu113

Base PyTorch commit: 8a45ef23f564cc092fa0805ea6c0f8f73eed209c

Affected PyTorch version: 1.13.0.dev20220630+cu113

Affected PyTorch commit: 331c0c18033bf4138c9ea468a8c759865dd8ffff

Affected Tests:
- test_train[LearningToPaint-cpu-eager]: +8.56976%
- test_eval[resnet18-cpu-jit]: -20.85417%
- test_eval[shufflenet_v2_x1_0-cpu-eager]: +18.36661%
- test_train[timm_regnet-cpu-eager]: -18.43808%


cc @xuzhao9


Workflow: https://github.com/pytorch/benchmark/actions/runs/2592223087
Empty result, closed.