zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: Losses documentation typos

I think there are some typos in the documentation of the losses.

1. Here I think that $\Delta$ should be defined as $f(\bar{k}) - f(k)$, so that the following formulations in the table would be correct.
![Screenshot from 2023-01-13 12-11-59](https://user-images.githubusercontent.com/48762613/212308326-12803f4c-9d1e-44a1-88d5-b40f61a55760.png)

2. Here the loss is correct, but I think the doc could be improved by expressing $f(h, r, t)$ for TransE as $-||\textbf{e}_h + \textbf{e}_r - \textbf{e}_t||_p$, as it is defined in the TransE model documentation.
![Screenshot from 2023-01-13 12-13-08](https://user-images.githubusercontent.com/48762613/212309086-21aaa90c-712e-4e6e-b33d-a8a2e46ebe38.png)

3. Again, invert $k$ and $\bar{k}$ and adjust the TransE interaction.
![Screenshot from 2023-01-13 12-14-09](https://user-images.githubusercontent.com/48762613/212309795-5dd68de5-aeee-4e98-9a8b-b4db6fa52920.png)

4. Same.
![Screenshot from 2023-01-13 12-14-26](https://user-images.githubusercontent.com/48762613/212310001-69b450a7-5b1c-4c27-88b0-465f60d69f6a.png)

5. Same.
![Screenshot from 2023-01-13 12-14-42](https://user-images.githubusercontent.com/48762613/212310031-185a2e67-1bea-41d5-ba14-30a982d1ef69.png)

I think that the inversion is confirmed by the implementation code:
```
return self._reduction_method(
            self.margin_activation(
                neg_scores - pos_scores + self.margin,
            )
        )
```



Hi @nicolafan ,

thanks for taking time to improve our documentation!

The raised issues seem to be correct to me. Sometimes it is hard to get the correct sign as loss functions are sometimes expressed in terms of score function, or distance-based, with inverted order.

Do you want to submit an own PR with changes to the documentation soure code [here](https://github.com/pykeen/pykeen/blob/3756e4c32b04acb83208d830b123fb0d5770cef6/src/pykeen/losses.py)? Otherwise, I could open a PR for it, too.


Sure! I can open a PR for this.
Repository: pykeen/pykeen
Issue: Add novel link prediction pipeline
Root cause: Unimplemented Feature Gap

Given an entity, its position (either head or tail), and a relation, return a ranked list of all entities and their scores. It should be clear if higher score or lower score means more likely to be a real edge. We might want to have it automatically filter out entities corresponding to edges already in the original knowledge graph.

This should be implemented in the top-level Model class, as all models should have this functionality.

Question: do models trained under LCWA or sLCWA have to behave differently here? Does this functionality have to behave differently based on the loss function?

> Question: do models trained under LCWA or sLCWA have to behave differently here? 

The behavior should be the same for both approaches. But the user should be aware that if the model was trained based on the LCWA training approach (in combination with 1:N-scoring), that for each (h,r)-pair for which a triple exists in the KG, all (h,r,t_i) that are not part of the KG have been considered as false examples during training. Therefore, asking the model to provide predictions for such (h,r)-pairs might not be very useful.

In general, we should just make sure to call the functions `model.predict_scores_all_heads()`/`model.predict_scores_all_tails()`, and not directly `model.score_h()`/`model.score_t()`, because the former make sure that the model uses the appropriate scoring functions in case it was trained with inverse relations.


>  It should be clear if higher score or lower score means more likely to be a real edge.

In our framework, we always treat higher scores as higher likelihood.

> We might want to have it automatically filter out entities corresponding to edges already in the original knowledge graph.

:+1:

> Question: do models trained under LCWA or sLCWA have to behave differently here? Does this functionality have to behave differently based on the loss function?

For point-wise loss functions, the score are calibrated across different `(h, r)` / `(r, t)`-pairs (since we have a fixed "target value" for each triple during training). For pairwise loss functions, this is not the case, and during training we only imposed a meaningful order between tail entities for the same `(h,r)` (and vice-versa).
Repository: pykeen/pykeen
Issue: hpo_pipeline function does not accept evaluation_entity_whitelist nor evaluation_relation_whitelist
Root cause: Interface Contract Mismatch

**Describe the bug**
Probably more a question than a bug again. When I copy the code snippet from your tutorial on https://pykeen.readthedocs.io/en/latest/tutorial/understanding_evaluation.html and run it, I get the error that the hpo_pipeline does not accept the keyword argument "evaluation_relation_whitelist", even though it says so in the tutorial. I have searched the docs for a bit now and couldn't find a way to pass "evaluation_relation_whitelist" to the hpo_pipeline. Is there one?

Error: 
    TypeError: hpo_pipeline() got an unexpected keyword argument 'evaluation_relation_whitelist'

**To Reproduce**
Steps to reproduce the behavior:
(copied the code from https://pykeen.readthedocs.io/en/latest/tutorial/understanding_evaluation.html):
```python
from pykeen.hpo import hpo_pipeline

evaluation_relation_whitelist = {'CtD', 'CpD'}
hpo_pipeline_result = hpo_pipeline(
    n_trials=30,
    dataset='Hetionet',
    model='RotatE',
    evaluation_relation_whitelist=evaluation_relation_whitelist,
)
```
**Expected behavior**
It trains and evaluates only on the whitelist.

**Environment:**

    OS: Ubuntu 18.04.5 LTS
    Python version: 3.8.5
    Version of this software: 1.0.5
    Versions of required Python packages:
        click 7.1.2
        click-default-group 1.2.2
        numpy 1.19.2
        torch 1.6.0
        torch-cluster 1.5.7
        torch-geometric 1.6.1
        torch-scatter 2.0.5
        torch-sparse 0.6.7
        torch-spline-conv 1.2.0
        tqdm 4.51.0


Hi @fratajcz this definitely sounds like a bug! Thank you for pointing it out!  I'm afk but I will take care of it later today and push a new release if all goes well
Repository: pykeen/pykeen
Issue: How to combine multi datasets into one that can be loaded into the triples_factory
Root cause: Unimplemented Feature Gap

### Problem Statement

How can we combine multi datasets/triples into one and can be trained by pykeen, if there are multiple datasets like this

e.g:
```python
dt_list = [
pykeen.datasets.AristoV4,
pykeen.datasets.BioKG,
pykeen.datasets.CKG,
pykeen.datasets.CN3l,
pykeen.datasets.CSKG,
]

# dt_1_2_3_sets = some_func_to_combine(*dt_list)

```

or Do we have a function like pytorch's ConcatDataset ?

```python
#This is a pytorch concat example which can glue multi dataset into one 
dt_1_2_3_sets = torch.utils.data.ConcatDataset([dataset_1, dataset_2,dataset_3])
train_dev_loader = DataLoader(dataset=dt_1_2_3_sets, ...)
```


### Describe the solution you'd like

like pytorch's ConcatDataset 

### Describe alternatives you've considered

A pytorch style [ConcatDataset](https://pytorch.org/docs/stable/data.html#torch.utils.data.ConcatDataset) 

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
Repository: pykeen/pykeen
Issue: Add KEN to leverage numerical attributes in KG embeddings
Root cause: Unimplemented Feature Gap

### Publication Link

https://hal.science/hal-03848124/document

### Reference Implementation

https://github.com/alexis-cvetkov/KEN

### Additional Implementations

_No response_

### Relevance

KEN (Knowledge Embedding with Numbers) extends KG embedding models to numerical triples, e.g. (Paris, HasPopulation, 2.2M). The idea is the following: a numerical value $x$ from an attribute/relation $r$ is embedded into the same space as other entities: $e_r(x) = ReLU(x * w_r + b_r) \in \mathbb{R}^p$.

Before embedding them, we normalize the values $x$ between 0 and 1, using a [quantile transform](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.QuantileTransformer.html#sklearn.preprocessing.QuantileTransformer).

We compared this model to LiteralE, and it performed better. Although we evaluated it on downstream tasks (external to the KG), and not on link prediction. We also did an ablation study to show the importance of the quantile transform and the ReLU activation.

### Additional Context

I am the author of the model and have implemented the approach on top of PyKEEN. Now I would like to add it to the core package.

Hi @alexis-cvetkov ,

we would appreciate your PR, especially as literal KG models have been a niche topic within PyKEEN. I would suggest you go ahead an open a PR with your suggested addition and we can give you feedback over there.

I did not yet have the time to fully read your paper, but from skimming over it it seems like you are also evaluating on the prediction of numerical attributes, which is something we do not yet have within PyKEEN. Maybe we can leave this part for a follow-up PR? Happy to hear your thoughts on this though.

EDIT: It sounds like the quantile transform is a somewhat orthogonal component to the specific prediction, $e_r(x) = ReLU(x * w_r + b_r) \in \mathbb{R}^p$. I assume we could implement this as some generic pre-/postprocessing of the numeric attributes?
Repository: pykeen/pykeen
Issue: EvaluationLoopCallback does not allow to pass `additional_filter_triples`
Root cause: Configuration Error

### Describe the bug

`EvaluationLoop` does not properly forward the `additional_filter_triples` argument to instantiate the `LCWAEvaluationDataset` here:

https://github.com/pykeen/pykeen/blob/ebb5b52b36dec7194c526b68094f82774b058ffc/src/pykeen/training/callbacks.py#L314-L318

### How to reproduce

```python
from pykeen.pipeline import pipeline
from pykeen.datasets import get_dataset

dataset = get_dataset(dataset="nations")
result = pipeline(
    dataset=dataset,
    model="mure",
    training_kwargs=dict(
        num_epochs=5,
        callbacks="evaluation-loop",
        callback_kwargs=dict(
            frequency=1,
            prefix="validation",
            factory=dataset.validation,
            additional_filter_triples=dataset.training,
        ),
    ),
)
```

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | nt                       |
| Platform        | Windows                  |
| Release         | 11                       |
| Time            | Sat Jan 28 12:33:04 2023 |
| Python          | 3.9.13                   |
| PyKEEN          | 1.9.1-dev                |
| PyKEEN Hash     | ebb5b52b                 |
| PyKEEN Branch   | master                   |
| PyTorch         | 1.13.1+cpu               |
| CUDA Available? | false                    |
| CUDA Version    | N/A                      |
| cuDNN Version   | N/A                      |

### Additional information

First discussed here: https://github.com/pykeen/benchmarking/issues/27#issuecomment-1407375480

### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed
Repository: pykeen/pykeen
Issue: Relax Slicing Requirements
Root cause: Validation Gap

> Now in your case, I am not sure why we need to raise that, since the *evaluation* behaviour of batch normalization layers does *not* depend on other batch members (while its training behaviour does). So I think we could relax this check.

_Originally posted by @mberr in https://github.com/pykeen/pykeen/issues/1197#issuecomment-1409235722_
Repository: pykeen/pykeen
Issue: How to use self data in pykeen?
Root cause: Documentation Deficiency

I want to try this toy example
https://pykeen.readthedocs.io/en/stable/tutorial/translational_toy_example.html

I need to define the tf by myself. I try two ways,

1. I create a csv file, data.csv

<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">

<head>

<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File
href="file:///C:/Users/mofan/AppData/Local/Temp/msohtmlclip1/01/clip.htm">
<link rel=File-List
href="file:///C:/Users/mofan/AppData/Local/Temp/msohtmlclip1/01/clip_filelist.xml">
<style>
<!--table
	{mso-displayed-decimal-separator:"\.";
	mso-displayed-thousand-separator:"\,";}
@page
	{margin:.75in .7in .75in .7in;
	mso-header-margin:.3in;
	mso-footer-margin:.3in;}
tr
	{mso-height-source:auto;}
col
	{mso-width-source:auto;}
br
	{mso-data-placement:same-cell;}
td
	{padding-top:1px;
	padding-right:1px;
	padding-left:1px;
	mso-ignore:padding;
	color:black;
	font-size:11.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:Calibri, sans-serif;
	mso-font-charset:0;
	mso-number-format:General;
	text-align:general;
	vertical-align:bottom;
	border:none;
	mso-background-source:auto;
	mso-pattern:auto;
	mso-protection:locked visible;
	white-space:nowrap;
	mso-rotate:0;}
-->
</style>
</head>

<body link="#0563C1" vlink="#954F72">



subject | relation | object
-- | -- | --
Brussels | LocatedIn | Belgium
Belgium | part of | EU
EU | hasCapital | Brussels



</body>

</html>

Then I use 
`tf = TriplesFactory.from_path('data.csv')`

I receive this error
"
  File "D:\Anaconda\envs\KnowledgeGraphEmbedding\lib\site-packages\pykeen\triples\triples_factory.py", line 906, in from_labeled_triples
    unique_relations, inverse = np.unique(triples[:, 1], return_inverse=True)
IndexError: index 1 is out of bounds for axis 1 with size 1"

2. I put the data in the sitepackage of the pykeen directly like the nation dataset, 
![image](https://user-images.githubusercontent.com/67538561/215910442-dc2aa845-6ff2-4176-a859-7e541bd3b74e.png)

Then  I use this, 

`from pykeen.datasets.Toy import Toy_TRAIN_PATH, Toy_TEST_PATH
     
tf = TriplesFactory.from_path(Toy_TRAIN_PATH)`


Still, it didn't work, can anybody help me?
Repository: pykeen/pykeen
Issue: No cuda devices were available. The model runs on CPU
Root cause: Environment Incompatibility

I installed the pytorch GPU version. However, when I run the following code
`transE_model = pipeline(
    training=training,
    testing=testing,
    validation=validation,
    model='TransE',
    stopper='early',
    epochs=5,
    dimensions=128,
    random_seed=420,
    result_tracker='csv',
    result_tracker_kwargs=dict(
        name='transE_model',
    ),
)`

I got this message.
`No cuda devices were available. The model runs on CPU
`

Any idea how to check this and solve this?

Dear @mf093087, have you seen the instructions on our issue tracker on how to report issues vs. asking for help (https://github.com/pykeen/pykeen/issues/new/choose)?This is more appropriate for a discussion topic and does not belong in the issue tracker. 

Further, I would suggest giving significantly more detail about your system and installation if you would like someone to help you debug. You can try running `pykeen version` in the terminal to get a nice summary that can be copy/pasted into GitHub.

Finally, what you wrote doesn't demonstrate that you spent a lot of time on this nor explain what you tried already. It's hard to be motivated to give feedback on an question that isn't more widely beneficial for the community.
Repository: pykeen/pykeen
Issue: Access background information on triples within a batch
Root cause: Unimplemented Feature Gap

I'm trying to implement a model that would learn with weighted triples. Weights are constant over epochs but they are provided by an external source. To inject these weights into loss calculation, the `Loss.forward(prediction, labels)` function should have a way to known what triples are part of the batch being processed but it only gets plausibility scores and cannot reconstruct the ID of the triples that lead to these plausibility scores.

My example is with weights, but various sorts of background information, indexed by triples, could be legitimately injected during training. Is there a way to do that in the current version of Pykeen? If you had to extend the framework to allow for that, how would you do it?

Hi @vcharpenay ,

your issue seems to be related to what @sbonner0 has started in this PR: https://github.com/pykeen/pykeen/pull/1142, and the discussion over there

Indeed, thank you.
Repository: pykeen/pykeen
Issue: Estimate diversity function of NodePieceRepresentation is broken
Root cause: Algorithmic Error

### Describe the bug

The `estimate_diversity` function of `NodePieceRepresentation` references `token_representations`, when it should reference `base`.

### How to reproduce

```python
>>> from pykeen.datasets import Nations
>>> from pykeen.models import NodePiece
>>> dataset = Nations(create_inverse_triples=True)
>>> model = NodePiece(
...     triples_factory=dataset.training,
...     tokenizers=["AnchorTokenizer", "RelationTokenizer"],
...     num_tokens=[20, 12],
...     embedding_dim=64,
...     interaction="rotate",
...     relation_initializer="init_phases",
...     relation_constrainer="complex_normalize",
...     entity_initializer="xavier_uniform_",
... )
>>> model.entity_representations[0].estimate_diversity()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dobraczka/Downloads/git/pykeen/src/pykeen/nn/node_piece/representation.py", line 352, in estimate_diversity
    tokens.assignment.unique(dim=0).shape[0] / self.max_id for tokens in self.token_representations
  File "/home/dobraczka/Downloads/git/pykeen/.direnv/python-3.10.6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1269, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'NodePieceRepresentation' object has no attribute 'token_representations'
```

### Environment

| Key             | Value                      |
|-----------------|----------------------------|
| OS              | posix                      |
| Platform        | Linux                      |
| Release         | 5.15.0-60-generic          |
| Time            | Tue Mar 21 11:44:03 2023   |
| Python          | 3.10.6                     |
| PyKEEN          | 1.10.2-dev                 |
| PyKEEN Hash     | 718ce463                   |
| PyKEEN Branch   | main |
| PyTorch         | 1.13.1+cu117               |
| CUDA Available? | false                      |
| CUDA Version    | 11.7                       |
| cuDNN Version   | 8500                       |

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed
Repository: pykeen/pykeen
Issue: Hello World notebook not updated to the current release
Root cause: Documentation Deficiency

### Describe the bug

Prediction portion code lines are not working.

### How to reproduce

Run the cells.

### Environment

1.10.2-dev

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Thanks for reporting the issue. It should be fixed with #1249 being merged.
Repository: pykeen/pykeen
Issue: inverse relation index
Root cause: Documentation Deficiency

### Describe the bug

when i set create_inverse_triples=True， the number of relations is n, the number of relation_representatoion is 2n, because the first half is a forward relationship and the second half is an inverse relationship, but  calling the get_inverse_id function, why do I get a relationship with an inverse relationship of id + 1 instead of id + relation_num

### How to reproduce

```python
origin_relation_id= torch.tensor([
        [ 51],
        [ 51],
        [128],
        [127],
        [ 10],
        [ 97]], device='cuda:0')
invert_id=transeModel.relation_inverter.get_inverse_id(origin_relation_id)
print(invert_id)
tensor([[ 52],
        [ 52],
        [129],
        [128],
        [ 11],
        [ 98]], device='cuda:0')
```

### Environment

pykeen 1.10.1

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Closed in favour of https://github.com/pykeen/pykeen/discussions/1248
Repository: pykeen/pykeen
Issue: Add PharMeBINet
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

PharMeBINet is a biomedical KG which attempts to update and expand Hetionet. 

**Dataset is pre-stratified into train/test/valid** [NO]

**Publication(s) or website describing benchmark KG dataset**: https://www.nature.com/articles/s41597-022-01510-3

**URL for downloading the benchmark KG dataset**:  https://zenodo.org/record/7009457