zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: Add NELL-995
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

A subset of http://rtw.ml.cmu.edu/rtw/

Besides normal triples, there are also explicit positive **and negative** labels for the triple classification task.


**Dataset is pre-stratified into train/test/valid**

There is a raw version without split, in TSV format.

In addition, there are triples which have been used to train Trans{D,E,H} embeddings.
These are split across multiple files in `tasks/{relation_name}/transX`.

Also, there are triples with explicit positive **and negative** label for the triple classification task.
Again, these are spread over multiple files in `tasks/{relation_name}/train.pairs`

**Publication(s) or website describing benchmark KG dataset**:
https://www.aclweb.org/anthology/D17-1060.pdf
https://github.com/xwhan/DeepPath

**URL for downloading the benchmark KG dataset**:
http://cs.ucsb.edu/~xwhan/datasets/NELL-995.zip

The zip file contains also pretrained embeddings for multiple models. If it is possible to only download the relevant parts of a ZIP file, it might be worth to do so.

https://github.com/ibalazevic/multirelational-poincare/tree/master/data has some additional derived datasets.

rock n rollllll i got u
Repository: pykeen/pykeen
Issue: Add Countries Dataset
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

This dataset connects each country its neighbors and the continent that it's in

**Dataset is pre-stratified into train/test/valid** [yes]

**Publication(s) or website describing benchmark KG dataset**:

https://github.com/ZhenfengLei/KGDatasets

**URL for downloading the benchmark KG dataset**:

https://github.com/ZhenfengLei/KGDatasets
Repository: pykeen/pykeen
Issue: PairRE
Root cause: Unimplemented Feature Gap

**On which paper(s) is your requested model based.**
https://arxiv.org/abs/2011.03798

**List the original and other existing implementations of this model**
No implementation known.

**Relevance of this model**
It is a straight-forward extension of RotatE. They report improved results compared to it.

**Ability to help and additional context**
I can implement it.
Repository: pykeen/pykeen
Issue: Add MuRE model
Root cause: Unimplemented Feature Gap

**On which paper(s) is your requested model based.**
Originally proposed as baseline to MuRP in https://arxiv.org/abs/1905.09791
More thoroughly investigated in https://openreview.net/forum?id=gLWj29369lW

**List the original and other existing implementations of this model**
Original Implemention: https://github.com/ibalazevic/multirelational-poincare

**Relevance of this model**
https://openreview.net/forum?id=gLWj29369lW argues that this model has some preferable properties among "linear" models, and a close connection to the distributional hypothesis for learning word embeddings. Moreover, it is a simple interaction function.

**Ability to help and additional context**
I will implement it.
Repository: pykeen/pykeen
Issue: The interaction_function of ComplEx is not what it should be
GitHub labels: bug

According to the documentation and the original paper for the ComplEx model, the last term is subtracted (ie. has a minus sign), but all the terms seem to have the same sign in the actual implementation. Am I missing something?


Hi @alkis-pap .

thanks for spotting this mistake. It seems like it was introduced by some refactoring done in https://github.com/pykeen/pykeen/commit/b208de2475865a008d608ab739a66b949912b0a3.

I will fix this error soon.
Repository: pykeen/pykeen
Issue: Add DB100K
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

From the paper:
> We create our third dataset from the mapping-based objects of core DBpedia. We eliminate relations not included within the DBpedia ontology such as HomePage and Logo, and discard entities appearing less than 20 times. The final dataset, referred to as DB100K, is composed of 470 relations and 99,604 entities.

**Dataset is pre-stratified into train/test/valid** [yes]

**Publication(s) or website describing benchmark KG dataset**:

- https://arxiv.org/pdf/1805.02408 (Ding *et al.*, 2018)
- https://github.com/iieir-km/ComplEx-NNE_AER

**URL for downloading the benchmark KG dataset**:

https://github.com/iieir-km/ComplEx-NNE_AER/tree/master/datasets/DB100K
Repository: pykeen/pykeen
Issue: OGB BioKG Incorrect Data Load
Root cause: Algorithmic Error

**Describe the bug**
It appears to me that the data loader code for the OGBBioKG dataset doesn't properly map the entities to their associated types, therefore mis-identifying entities in the triples sent in the TriplesFactory object. 

**To Reproduce**
Assuming the loading for OGBBioKG is handled by the [OGBLoader class](https://github.com/pykeen/pykeen/blob/master/src/pykeen/datasets/ogb.py#L25), then the line [`edge_split = dataset.get_edge_split()`](https://github.com/pykeen/pykeen/blob/master/src/pykeen/datasets/ogb.py#L50) returns a dictionary. There are two keys in this dictionary called `head_type` and `tail_type` which are being ignored by the rest of the loading procedure. This would normally not be an issue, but the entity mappings for this [ogbl-biokg data](https://ogb.stanford.edu/docs/linkprop/#ogbl-biokg) (which for me is found at the pystow default `~/.data/pykeen/datasets/ogbbiokg/ogbl_biokg/mapping`) indicate that _each entity type_ has its own 0-indexed identifier. Therefore, there are entities with different types (protein, drug, disease, function, sideeffect) that have the same `ent idx` as assigned by OGB. Because the OGBLoader doesn't use the `head_type` and `tail_type` information to disambiguate the entity types, the entity [identities passed to the TriplesFactory](https://github.com/pykeen/pykeen/blob/master/src/pykeen/datasets/ogb.py#L71) are therefore mis-aligned. 

**Expected behavior**
At least for the OGBBioKG dataset, we would expect the OGBLoader to take these entity types into account when assigning pykeen-specific identifiers.  
To do this, slightly more preprocessing is needed. First, we need to read in the mapping files provided by OGB (disease_entidx2name.csv.gz,  drug_entidx2name.csv.gz,  function_entidx2name.csv.gz,  protein_entidx2name.csv.gz, sideeffect_entidx2name.csv.gz) which can be found in the aforementioned mapping directory. We would then need to iteratively increment an identity counter, making sure that there are no collisions across entity types. We would also need to augment the `entity_to_id` values to match this new identification approach.

**Environment:**

| Key                  | Value              |
|----------------------|--------------------|
| `os.name`            | posix              |
| `platform.system()`  | Linux              |
| `platform.release()` | 4.15.0-122-generic |
| python               | 3.7.9              |
| pykeen               | 1.3.1-dev-UNHASHED |
| torch                | 1.7.0              |
| cuda available       | true               |
| cuda                 | 10.2               |
| cudnn                | 7605               |

OGB Version: 1.2.4

**Additional Comments**
I am happy to submit a PR for this once I figure out how to efficiently make this re-mapping. However, this may take me some time, and I wanted to flag this as a potential issue in case someone is running experiments on the OGBBioKG in the meantime and sees unexpected performance.
Repository: pykeen/pykeen
Issue: Domain Negative Sampling
GitHub labels: 🦞 Negative Sampler

💎 New Component

**On which paper(s) is your requested sampler based.**

Q. Xie, X. Ma, Z. Dai, and E. Hovy, “An interpretable knowledge transfer model for knowledge base completion,” in ACL, 2017, pp. 950–962.

https://arxiv.org/abs/1704.05908

**List the original and other existing implementations of this sampler**

<img width="673" alt="Screen Shot 2021-02-26 at 15 12 46" src="https://user-images.githubusercontent.com/5069736/109310659-1c6e4f80-7845-11eb-9266-ac01c79c9a90.png">

Could not find an implementation for ITransF (the model in this paper) or the domain negative sampling, though the paper has an appendix that describes how it works.

**Relevance of this sampler**

Could be interesting since it takes relationship into account, but also I want to have more negative samplers

**Ability to help and additional context**

I can motivate the team!


[In this paper](https://www.cl.uni-heidelberg.de/~nastase/publications/Papers/kbcom2018_analysis-impact-negative.pdf), they also discuss different negative sampling strategies.
Repository: pykeen/pykeen
Issue: Adversarial negative sampling
GitHub labels: 🦞 Negative Sampler

💎 New Component

**On which paper(s) is your requested sampler based.**

L. Cai and W. Y. Wang, “KBGAN: Adversarial learning for knowledge graph embeddings,” in NAACL, 2018, pp. 1470–1480.

https://arxiv.org/abs/1711.04071

**List the original and other existing implementations of this sampler**

https://github.com/cai-lw/KBGAN

**Relevance of this sampler**

This negative sampler is in contrast to the self-adversarial negative sampler from Sun et al., 2019

**Ability to help and additional context**

Motivation for the masses!
Repository: pykeen/pykeen
Issue: Update Optuna to 2.0.0
Root cause: External Dependency Breakage

You thought the 1.5 upgrade was confusing? Get ready for 2.0.0!

<blockquote class="twitter-tweet" data-partner="tweetdeck"><p lang="en" dir="ltr">The release of v2.0 is approaching and we are going through the final details 🎯 Let us know if you need any last minute fixes or changes through this release candidate.<br><br>pip install optuna==2.0.0rc0<a href="https://t.co/4K8Oyc5a2K">https://t.co/4K8Oyc5a2K</a> <a href="https://t.co/CeL5JQ9jbv">pic.twitter.com/CeL5JQ9jbv</a></p>&mdash; OptunaAutoML (@OptunaAutoML) <a href="https://twitter.com/OptunaAutoML/status/1280015129568927744?ref_src=twsrc%5Etfw">July 6, 2020</a></blockquote>
Repository: pykeen/pykeen
Issue: How to determine the checkpoint content?
Root cause: Documentation Deficiency

Dear Sir,

I want to ask if we could determine the output content in checkpoint? I have chekced the codes about pipeline(), it does not talk much about the checkpoint setting. I just notice it stored the content about the random seed, model, optimizer. Could I add more features in checkpoint? Eg. the training parameters or configuration.

Hi @ziwli, I'm not sure if you've seen the tutorial on checkpoints: https://pykeen.readthedocs.io/en/stable/tutorial/checkpoints.html. It's actually storing the whole model, optimizer, early stopper, random seed configuration, etc. You can see exactly what by checking the following code. All of the other configuration is defined by the call to the `pipeline()`, so you should have that already (though if you're using default values, this is a bit opaque).

If you want, you can directly unpickle that file that gets saved at each checkpoint and use the contents of the dictionary. It might also help to give a code example to go with your question, because I'm not sure if I answered it correctly.

https://github.com/pykeen/pykeen/blob/de817c54ba68f26fa14ce9333a81896e59481994/src/pykeen/training/training_loop.py#L873-L915



Dear Sir,

      ok, thank you! I think the last code information is very helpful.

On Fri, 5 Mar 2021 at 13:10, Charles Tapley Hoyt <notifications@github.com>
wrote:

> Hi @ziwli <https://github.com/ziwli>, I'm not sure if you've seen the
> tutorial on checkpoints:
> https://pykeen.readthedocs.io/en/stable/tutorial/checkpoints.html. It's
> actually storing the whole model, optimizer, early stopper, random seed
> configuration, etc. You can see exactly what by checking the following
> code. All of the other configuration is defined by the call to the
> pipeline(), so you should have that already (though if you're using
> default values, this is a bit opaque).
>
> If you want, you can directly unpickle that file that gets saved at each
> checkpoint and use the contents of the dictionary. It might also help to
> give a code example to go with your question, because I'm not sure if I
> answered it correctly.
>
>
> https://github.com/pykeen/pykeen/blob/de817c54ba68f26fa14ce9333a81896e59481994/src/pykeen/training/training_loop.py#L873-L915
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pykeen/pykeen/issues/331#issuecomment-791380012>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALQ6JMQYICBTMEIU7BPBNW3TCDC3LANCNFSM4YT2LP7Q>
> .
>
Repository: pykeen/pykeen
Issue: Batch-level loss and ResultTracker in console
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Similar to PyTorch, I'd like to be able to print out my loss on the batch-level instead of only at the end of every epoch. Reading the source, it looks like `result_tracker` only logs the epoch loss. 

**Describe the solution you'd like**
I'd like to be able to add a `result_tracker` for the batch-level loss, as well as potentially documentation on how to use ResultTrackers in Jupyter Notebooks by printing loss out to console (rather than with W&B/MLflow or File-based tracking).


Hi @vsocrates, that's a nice suggestion. We're going to have to think about how that might look, since it will be a pretty fundamental change to how the result tracker gets called (and potentially its data structure)

In the mean time, I will think about how we might be able to make tracking a bit more interactive in the Jupyter notebook, but to be honest, I think it wouldn't be a good use of time to try and develop a widget when MLFlow and W&B are already such excellent tools. Otherwise, a "print-to-console" tracker could be a very crude solution.

@vsocrates At least for the LCWA training loop, you might be able to get around by the following trick:

```python
# Add a wrapper around the training loop
class ModifiedTrainingLoop(LCWATrainingLoop):
  def _forward_pass(self, *args, **kwargs):
    loss = super()._forward_pass(*args, **kwargs)
    print(loss)  # or call anything you like
    return loss

# call pipeline
pipeline(
   ...
   training_loop=ModifiedTrainingLoop,
   ...
)
```

*Disclaimer: I have not tested this code, and it was written in browser*


@cthoyt for wrapping slcwa, the following code might lead to errors when you try to wrap as shown above:
https://github.com/pykeen/pykeen/blob/de817c54ba68f26fa14ce9333a81896e59481994/src/pykeen/pipeline/api.py#L883-L904

If we replace it by 
```python
not issubclass(training_loop_cls, SLCWATrainingLoop)
```
it should work again.

@mberr looks like this would be an excellent place to allow for callbacks. Should I get on twitter and start a war?

That makes sense, let me give it a shot! Thanks! 

EDIT: Subclassing works! I'll play around with a console-printing ResultTracker too! 

@vsocrates great! if you come up with something interesting, we'd be happy to walk you through making a PR.

P.S. I grew up in North Haven! I hope you enjoy living in the area around Yale.

@vsocrates we added a console tracker in #430, as well as making it easier to create training loop callbacks in #429. Thanks for your suggestions again!
Repository: pykeen/pykeen
Issue: Training loop callbacks
GitHub labels: enhancement

Callbacks should have a named tuple that describes what is available (and in some way can be extensible in the future). This would allow a pretty trivial implementation of #333 based on @mberr 's example

Linking [keras](https://keras.io/guides/writing_your_own_callbacks/#an-overview-of-callback-methods)' neat way of defining callback interaction points.
Repository: pykeen/pykeen
Issue: Running evaluation on the (subset of) training data
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Often we want to see how badly a model overfits. So far, the training loop explicitly calls an instance of an early stopper with the validation data
https://github.com/pykeen/pykeen/blob/00610012e9eb841dce56086e9284d4e6d4365f06/src/pykeen/training/training_loop.py#L595

So there is no out-of-the-box way to run the evaluation on a (subset of) training data which can be a parameter of course. 

**Describe the solution you'd like**
Allow the training loop to have an optional `train_stopper`argument (given that one would create another instance of an Early Stopper initialized with the required dataset)

**Describe alternatives you've considered**
So far I have to create a subclass of `TrainingLoop` with the new argument. And a rather ridiculous part is to create another subclass of an `EarlyStopper` to be able to put `prefix="training"` in the result tracker 😅 .
```python
self.result_tracker.log_metrics(
            metrics=metric_results.to_flat_dict(),
            step=epoch,
            prefix='training',
        )
```
The whole pipeline looks like this:
```python
dataset = ...  # any pykeen dataset
train_eval = copy.deepcopy(dataset.training)
train_eval.mapped_triples = train_eval.mapped_triples[torch.randperm(train_eval.shape[0])[: int(TR_RATIO * train_eval.shape[0])]]
train_stopper = TrainStopper(
    model=model,
    relative_delta=0.0005,
    evaluation_triples_factory=train_eval,
    frequency=eval_every,
    patience=1000,
    result_tracker=tracker,
    evaluation_batch_size=256,
    evaluator=valid_evaluator,
)
...
training_loop.train(..., stopper=<the one with validation data>, train_stopper=train_stopper)
```

And in the training loop it's two new lines:
```
            if train_stopper is not None:
                train_stopper.should_stop(epoch)

            if stopper is not None and stopper.should_evaluate(epoch) and stopper.should_stop(epoch):
                return self.losses_per_epochs
```
Of course, there is little sense to wrap the training eval into the early stopper but as of now it looks more elegant / in the style of the training loop



This may have been closed by #765 , right @migalkin ?

Right, can be closed now
Repository: pykeen/pykeen
Issue: Is it possible to stream triples on memory-constrained machines?
Root cause: Unimplemented Feature Gap

Hello,

I am looking at the best way at the moment to run PyKEEN on large data files that exceed my available RAM. Does PyKEEN have  support for steaming triples, rather than loading them all into a fixed-size array? At the moment I see PyKEEN is loading my entire TSV dataset into a fixed-size array, and I was not able to find anything in the docs about how to change that.

Sincerely,
--Jeffrey Sardina

Hi @Jeffrey-Sardina, the short answer is no - triple loading is pretty rigid. Since all models in PyKEEN need to at least know all of the entities and relations in the dataset, then train on multiple epochs over all of the triples, this was the naive solution.

I could imagine improvements where PyKEEN pre-processes a dataset in a streaming setting to calculate the entities and relations (as well as pre-map them, which could cut down on memory). @mberr had some great ideas about this and recently made some improvements to the TriplesFactory to prepare for this sort of thing.
Repository: pykeen/pykeen
Issue: Model of best epoch
Root cause: Documentation Deficiency

Dear Sir,

I want to ask that is the model trained of the best epoch or last epoch? I have read the tutorials that I could load the model from last time training, and it seems that this model is stored as last epoch result. I want to ask how could I get the trained model or checkpint of best epoch? Because I see the results.json, there writes in stopper best_epoch is XXX, I don't know if the trained model at last is the last epoch result or the best epoch result? If the last epoch result, could I get the best epoch result? (I also tried set the checkpoint_frequency to 0 which I could get the checkpoints after each epoch, but it failed maybe because my training time is too low....) 

Best wishes
Repository: pykeen/pykeen
Issue: Add QuatE
GitHub labels: 💃 Model

💎 New Component

**On which paper(s) is your requested model based.**

https://arxiv.org/abs/1904.10281

**List the original and other existing implementations of this model**

- Author's implementation: https://github.com/cheungdaven/QuatE
- https://github.com/Sujit-O/pykg2vec/blob/master/pykg2vec/models/pointwise.py
- https://github.com/Lion-ZS/DualE/blob/main/DualE-master/DualE-master/models/QuatE.py (verbatim copy from author's implementation)

**Relevance of this model**

The more higher-order geometric algebras, the better! Related: OctionE (#341)

**Ability to help and additional context**

I can cheerlead. 


It seems that [https://github.com/cheungdaven/QuatE](https://github.com/cheungdaven/QuatE) is the original implementation.

@mali-git none of the authors on that repo correspond to the names on the paper, and the paper does not reference this repo. How did you come to that conclusion?

> @mali-git none of the authors on that repo correspond to the names on the paper, and the paper does not reference this repo. How did you come to that conclusion?

The repository belongs to Shuai Zhang, who seems to be  the first author of the QuatE paper.

Oh, great. I must have misread. Thanks for the clarification
Repository: pykeen/pykeen
Issue: Add OctionE
GitHub labels: 💃 Model

💎 New Component

**On which paper(s) is your requested model based.**

https://arxiv.org/abs/1904.10281 (same as #340)

**List the original and other existing implementations of this model**

- Author's implementation: https://github.com/cheungdaven/QuatE/blob/master/models/OctonionE.py
- `pykg2vec` implementation: https://github.com/Sujit-O/pykg2vec/blob/aabbbfa5e859b99a14023f458e9cc989a5da0b1d/pykg2vec/models/pointwise.py#L844
- https://github.com/Lion-ZS/DualE/blob/main/DualE-master/DualE-master/models/OctonionE.py (verbatim copy from author's implementation)

**Relevance of this model**

Like QuatE (#340), more geometric algebra is fun!
Repository: pykeen/pykeen
Issue: Add DualE
Root cause: Unimplemented Feature Gap

**On which paper(s) is your requested model based.**

https://github.com/Lion-ZS/DualE/blob/main/DualE-master/DualE.pdf

**List the original and other existing implementations of this model**
Please provide links to the original implementation and an as comprehensive as possible list about other implementations
of this model.

- Author's implementation: https://github.com/Lion-ZS/DualE/blob/main/DualE-master/DualE-master/models/DualE.py

**Relevance of this model**

This model combines the coolness of QuatE (e.g., the ability to model as rotation) with the coolness of TransD (e.g., the ability to model translation)
Repository: pykeen/pykeen
Issue: Add MurP
Root cause: Unimplemented Feature Gap

**On which paper(s) is your requested model based.**

Originally proposed in https://arxiv.org/abs/1905.09791
More thoroughly investigated in https://openreview.net/forum?id=gLWj29369lW

**List the original and other existing implementations of this model**
Please provide links to the original implementation and an as comprehensive as possible list about other implementations
of this model.

Original Implementation: https://github.com/ibalazevic/multirelational-poincare

**Relevance of this model**

Another non-euclidean model, related to #310

**TODOs**

This issue will also require the implementation of the Reimann Optimizer and a new system for tagging different parameters for how they should be optimized