zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: How to use our own dataset?
Root cause: Documentation Deficiency

Hi, thanks for your great job.
I want to know how to use our own dataset? Are there any simple API ？

Hope for your reply!

Hi @quqxui ,

does the following tutorial help you?

https://pykeen.readthedocs.io/en/stable/byo/data.html

If you find that there is anything missing or unclear, please let us know so we can improve the documentation!

Hi @quqxui, I just updated the README to make sure that it's much more obvious how to find this tutorial! Please let us know if it's not sufficient, and I'll open this issue up again and we can find you a solution :)
Repository: pykeen/pykeen
Issue: New Setwise Loss(es)
GitHub labels: enhancement

💧 Loss

💎 New Component

Blocked by #18

Cross entropy loss and NSSA loss are the only two setwise loss formulations described in our paper. Its also a bit confusing becuase technically in the implementation, NSSA loss is inheriting from `PairwiseLoss`. Are there other setwise loss functions we could add, so there's more than 1?

NSSA loss now properly inherits from the Set-wise base class. Would be happy to add another set-wise loss to complement these, but it's time to close this issue.
Repository: pykeen/pykeen
Issue: Get embeddings from pretrained model
Root cause: Documentation Deficiency

Hello,
From your documentation I cannot understand how I can extract from a pre-trained model the final embeddings encoded for the entities and relations in my dataset. Moreover, I would like to know if it is possible to apply some filtering on the representations returned (i.e. returning only embeddings for certain entities).
Thanks in advance.

See also https://github.com/pykeen/pykeen/issues/465, https://github.com/pykeen/pykeen/issues/89

We've got a tutorial for just that in the documentation! It goes with the unreleased v1.5 code, but you can already check it out at https://pykeen.readthedocs.io/en/latest/tutorial/first_steps.html#using-learned-embeddings. In short you can do the following to get all of the representations:

```python
from typing import List

import pykeen.nn
from pykeen.pipeline import pipeline

result = pipeline(model='TransE', dataset='UMLS')
model = result.model

entity_representation_modules: List['pykeen.nn.RepresentationModule'] = model.entity_representations
entity_embeddings: pykeen.nn.Embedding = entity_representation_modules[0]
entity_embedding_tensor: torch.FloatTensor = entity_embeddings()

relation_representation_modules: List['pykeen.nn.RepresentationModule'] = model.relation_representations
relation_embeddings: pykeen.nn.Embedding = relation_representation_modules[0]
relation_embedding_tensor: torch.FloatTensor = relation_embeddings()
```

Some models have more than one representation, like TransD or NTN, so you can turn the 0 to another number in the index to access the other ones. If you want to only get a certain embeddding or list of embeddings, you can construct a `torch.LongTensor` that has the entity or relation indices to pass to the `__call__` on the `pykeen.nn.Embedding` like:

```python
entity_idx = torch.as_tensor([1,2,3], dtype=torch.float)
entity_embedding_tensor: torch.FloatTensor = entity_embeddings(indices=entity_idx)

To add to the answer: To get the ID for an entity label, you can use the `entity_to_id` mapping of the [Dataset](https://pykeen.readthedocs.io/en/stable/api/pykeen.datasets.base.Dataset.html#pykeen.datasets.base.Dataset.entity_to_id) or [TriplesFactory](https://pykeen.readthedocs.io/en/stable/reference/triples.html#pykeen.triples.TriplesFactory.entity_to_id).

The `TriplesFactory` also offers a convenience method to lookup multiple entities at once: https://pykeen.readthedocs.io/en/stable/reference/triples.html#pykeen.triples.TriplesFactory.entities_to_ids

We also had a similar question in #465 and #89. #465 has a bit more specific of an example

Great, thanks guys!
Repository: pykeen/pykeen
Issue: How do I update my v1.4.1 code to be compatible with v1.5.0+ code since `Model.triples_factory` no longer exists?
Root cause: Documentation Deficiency

Hi, 

As far as I understand from the documentation, the base KGE model object contains a TriplesFactory object. This can be called using `model.triples_factory` in pykeen 1.4.0. However, using the same command on a model trained in pykeen 1.4.1. raises the following error: 

`AttributeError: 'TransE' object has no attribute 'triples_factory'`

Is this intended? And is there another way to retrieve the TriplesFactory from the model? 

Thank you! 

Indeed, we cut the triples factory out of the model in #405 between 1.4.0 and 1.5.0-dev (whose release is imminent) in order to make pykeen faster and more memory efficient! We also did something similar to remove the triples factory from the negative sampler and the training loop in #409 and #406, respectively.

Sorry for the inconvenience - this means that you should load up an extra instance of the dataset or triples factory if you really need it. For most people, this would be because of label-based lookup.

If you're not sure how to update your code, please let me know what you're trying to do with the triples factory and I can try and give an example

Thanks so much for your prompt response. Indeed, I'm referring to the triples factory for a label-based lookup (specifically to get mapped triples from a dataframe with head, relation, and tail names). 

I'm also using my own dataset. Would the workaround be to:

1)  convert the text file to a triples factory 
2) point to this triples factory in the training pipeline? 

(And refer to it in whatever supplementary functions?)

Yup, that sounds about right. If you're using the `pipeline()` function, the resulting object should have a reference to the training factory as well. See https://pykeen.readthedocs.io/en/latest/_modules/pykeen/pipeline/api.html#PipelineResult

The point of this change was to make sure that the triples factory isn't a property on the model that gets brought to the GPU, so it's a slight workaround to leave it in the PipelineResult instance at the end

Ah, I hadn't realised I can retrieve it from the pipeline. That solves it, thank you!
Repository: pykeen/pykeen
Issue: How to get embeddings?
Root cause: Documentation Deficiency

Hi guys, thanks a lot for your works! I am a new starter, and bit of confusing about how to get the embeddings of the entities or the relations from the trained model?

See also:  #480, https://github.com/pykeen/pykeen/issues/465, https://github.com/pykeen/pykeen/issues/89.

Hello, 
If you are using the pipeline to learn the embeddings then you can use 
```
result.model.entity_embeddings()
```
to get the entity embeddings where _result_ is output from the pipeline method.

> Hello,
> If you are using the pipeline to learn the embeddings then you can use
> 
> ```
> result.model.entity_embeddings()
> ```
> 
> to get the entity embeddings where _result_ is output from the pipeline method.

Thanks a lot for this quick reply! 
And is there any function to get a specific entity embedding, like get_embedding('entity1) = [1,1,1,2,3,4]? Currently, I used the TriplesFactory.entity_to_id the get the mapping of entity and id, and using this id the get the embedding.


I have not tried that. I also did the mapping from entity to id and saved the corresponding embeddings. It seemed like an easier way for me.

Roger that, many thanks!

Note for everyone in the thread, there are a few other issues covering this question in detail: #480, #465, #89.

Keep in mind the PyKEEN 1.4.X series has a bit of inconsistency in how this works because of the transition from old-style models to new-style models (subclasses of the ERModel). The imminent PyKEEN 1.5.X series unifies this in a single interface.
Repository: pykeen/pykeen
Issue: Use torch.isin when torch 1.10 is released

Use torch's isin once 1.9 arrives to replace custom implementation here:
https://github.com/pykeen/pykeen/blob/2f00453e9a4be691523aef8f96d1f8b4c140f116/src/pykeen/utils.py#L437

cf. https://github.com/pytorch/pytorch/issues/3025

You're an oracle! 1.9 was released today ;)

Backpropogation in complex torch.nn.Embedding?!?1

Sorry guys but this is another case where torch closed a PR without actually merging it. The feature did not make it in to version 1.9

Maybe it was too late for inclusion in the release branch? Anyway, it should be available in 1.10 :rocket: 

They dropped pytorch 1.10 today which does indeed include `torch.isin`!

A few points from 1.10 which may be interesting:

## torch.conj
> Conjugate View: tensor.conj() now returns a view tensor that aliases the same memory and has conjugate bit set (pytorch/pytorch#54987, pytorch/pytorch#60522, pytorch/pytorch#66082, pytorch/pytorch#63602).

> Added fast path for conjugated tensors for torch.{dot, vdot, mm, addmm, bmm, baddbmm} (pytorch/pytorch#62915, pytorch/pytorch#59380)

cf. https://pytorch.org/docs/stable/generated/torch.conj.html#torch.conj

afaik, this is used by one of the ComplEx kernels.

## nn.CrossEntropyLoss
>nn.CrossEntropyLoss: Added support for class probability targets (pytorch/pytorch#61044)
>nn.CrossEntropyLoss: Added support for label smoothing (pytorch/pytorch#63122)

replace our custom label smoothing implementation?

## torch.scatter
> Added reduce variants for torch.scatter operation. (pytorch/pytorch#57015)

This may be useful for message aggregation.

@mberr even though this issue is going to be closed from #635, do you want to make new issues to look into using the other ones from the last post you made

> torch.scatter

This one seems to only support `add` and `mul` as aggregation, but not, e.g., mean. So currently, I do not directly see a place where we can apply this directly.

The other two may be interesting, though.

-> label smoothing: #636

`torch.conj` seems to be an improvement in speed where we do not have to adjust anything. We could investigate whether this actually brought us some speed improvement.
Repository: pykeen/pykeen
Issue: Accessing default hyperparameters for each model on each dataset
Root cause: Documentation Deficiency

Hi, 

It was explained in PyKEEN's documentation that each model's default hyperparameters for a given dataset are "chosen from the best-reported values in each model’s original paper unless otherwise stated on the model’s reference page. In case hyper-parameters for a model for a specific dataset were not available, we choose the hyper-parameters based on the findings in our large-scale benchmarking [ali2020a]."

I was wondering - is the best way to access these hyperparameters at https://github.com/pykeen/benchmarking?

For models whose parameters on a given dataset were established in the abovementioned benchmarking study, e.g. ComplEx on FB15k-237, would the final best pipeline be included somewhere in the ablation folder? I see 4 folders at https://github.com/pykeen/benchmarking/tree/master/ablation/results/complex/fb15k237/random/adam and am not sure which folder's best pipeline was implemented in the package.

Thank you very much for any help!




Hi @MagdalenaMl,

> I was wondering - is the best way to access these hyperparameters at https://github.com/pykeen/benchmarking?

For each model+dataset combination, we evaluated various compositions of model architecture (e.g., TransE), loss function (e.g., softplus loss), with/without inverse relations, and training approach (e.g., LCWA).
The sub-folders in `benchmarking/ablation/results/` contain the experimental artifacts of all evaluated compositions. The folder you linked contains the artifacts of all evalauted configurations  for ComplEx on FB15K-237.
For instance, [study.json](https://github.com/pykeen/benchmarking/blob/master/ablation/results/complex/fb15k237/random/adam/2020-03-01-12-36_18bc5f9e-d7d2-488a-9a6d-951b0176efec_hpo_complex_fb15k237_owa_adam/0000_fb15k237_complex/study.json) describes one specific configuration, and the corresponding results are contained in [results.json](https://github.com/pykeen/benchmarking/blob/master/ablation/results/complex/fb15k237/random/adam/2020-03-01-12-36_18bc5f9e-d7d2-488a-9a6d-951b0176efec_hpo_complex_fb15k237_owa_adam/0000_fb15k237_complex/best_pipeline/2020-08-26-04-03-17/replicates/replicate-00000/results.json).
In Tables 20-23 in our [paper](https://arxiv.org/pdf/2006.13365.pdf) (Appendix), we describe for each dataset+model combination the best performing configuration. For instance, the experimental artifacts for the best performing configuration for ComplEx on FB15K-237 are contained in https://github.com/pykeen/benchmarking/tree/master/ablation/results/complex/fb15k237/random/adam/2020-03-10-04-36_610f748e-afa6-4b5c-96b1-ce86f4fbeff6_hpo_complex_fb15k237_lcwa_crossentropy/0000_fb15k237_complex
We will update the repository soon and provide functions to query the experimental artifacts for each tested configuration.

Thank you, that clarifies all :)
Repository: pykeen/pykeen
Issue: device mismatch between triples and model when model is in inference mode
Root cause: Interface Contract Mismatch

**Describe the bug**
for a model trained on gpu, when I do `model.predict_tails(...) it gives an error of 

```
....
/usr/local/lib/python3.6/dist-packages/pykeen/models/base.py in _novel(self, h, r, t)
    399         """Return if the triple is novel with respect to the training triples."""
    400         triple = torch.tensor(data=[h, r, t], dtype=torch.long, device=self.device).view(1, 3)
--> 401         return (triple == self.triples_factory.mapped_triples).all(dim=1).any().item()
    402 
    403     def predict_scores_all_relations(

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in wrapped(*args, **kwargs)
     26     def wrapped(*args, **kwargs):
     27         try:
---> 28             return f(*args, **kwargs)
     29         except TypeError:
     30             return NotImplemented

RuntimeError: expected device cuda:0 but got device cpu
```
(not sure if the model should always be on the cpu? but if so would be nice to update the docs to point this out, since if I trained the model on gpu I should expect to be able to do inference on the gpu as well?)

**To Reproduce**
Steps to reproduce the behavior:
1. train a model on gpu
2. do model.predict_tail('entity', 'relation')
3. device mismatch error

**Expected behavior**
there shouldn't be a device mismatch

**Environment (please complete the following information):**
- OS: ubuntu 18.04
- Python version: 3.7
- Version of this software: 1.0.2dev
Versions of required Python packages: # run pip list | grep -Eiw 'torch|numpy|Click|click-default-group|tqdm'
- Torch: 1.4.0
- Numpy: 1.18.1
- Click: 7.1.2
- Click_default_group:1.2.2
- Tqdm: 4.46.1

**Additional information**
an example [on colab here](https://colab.research.google.com/drive/10b7sNKcupKGrLiSuOGRheQcrLTs2rvGU?usp=sharing)


Thanks @wwymak for pointing this out!
PR #51 will fix the issue.

A fix in the mean time is to do: 

```python
model = model.to_cpu_()
```

> A fix in the mean time is to do:
> 
> ```python
> model = model.to_cpu_()
> ```

If the finalization of PR #51 requires more time, we could consider merging PR #50.
Repository: pykeen/pykeen
Issue: Add PyKEEN 1.0 History
Root cause: Unimplemented Feature Gap

We need to merge from the private development repository. It will take an awful lot dark magic to get to there:

1. Make a new branch on pykeen/pykeen (optional - delete everything, and commit?)
2. Add the following to the .git/config file
```ini
[remote "poem"]
    url = git@github.com:mali-git/POEM_develop.git
    fetch = +refs/heads/*:refs/remotes/origin/*
```
3. `git pull poem master --allow-unrelated-histories --no-commit -S` and pray to rebased god. Figured this out with lots of googling and eventually [the perfect article](https://thoughts.t37.net/merging-2-different-git-repositories-without-losing-your-history-de7a06bba804), so thanks a bunch to the author @fdv
4. Commit
5. Profit
Repository: pykeen/pykeen
Issue: Add LiterallyWikidata datasets
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

> LiterallyWikidata is a collection of Knowledge Graph Completion benchmark datasets extracted from Wikidata and Wikipedia with a special focus on literals. 

This contains repo 3 datasets LitWD1K , LitWD19K, and LitWD48K.

**Dataset is pre-stratified into train/test/valid** 

Yes

**Publication(s) or website describing benchmark KG dataset**:

Download page on Zenodo: https://zenodo.org/record/4701190#.YNOx3OT3bDs

*not published yet. Can get more information from @GenetAsefa

**URL for downloading the benchmark KG dataset**:

https://zenodo.org/record/4701190/files/LiterallyWikidata.zip?download=1
Repository: pykeen/pykeen
Issue: Add WD50K
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

The dataset is extracted from Wikidata with the Wikidata-IDs still being available. This allows to retrieve further information about them via Wikidata, e.g., for interpretation. Although being of hyper-relational nature, i.e., single triples can have (an arbitary number of) additional qualifier pairs (relation-entity pairs) as "metadata" attached to them, we could, for now, restrict us to the triple part, which is also separately available.

Bonus: @migalkin knows all the details about it :wink: 

**Dataset is pre-stratified into train/test/valid** 
Yes, cf. https://github.com/migalkin/StarE/tree/master/data/clean/wd50k/triples (for triple-only split)

**Publication(s) or website describing benchmark KG dataset**:
https://www.aclweb.org/anthology/2020.emnlp-main.596/

**URL for downloading the benchmark KG dataset**:
https://github.com/migalkin/StarE/tree/master/data/clean

@cthoyt although being created via the new dataset template, the issues was not labeled with `Dataset` :thinking: 

Done in #511 , I think we can resolve the issue now
Repository: pykeen/pykeen
Issue: How to get the embedding of an entity and a triple from a trained KGE model?
GitHub labels: question

After I trained a KGE from my own data. I would like to get the embedding of a special entity and triple. Could you please tell me how to do it? Thanks.

Hi @GabrielLin .

the steps required to get an entity's embedding are described here: https://pykeen.readthedocs.io/en/stable/tutorial/first_steps.html#using-learned-embeddings

We do not have models with an explicit representation of a triple though. Instead the models have an interaction function, which takes head, relation and tail representations as input, and produces a scalar (plausability) score.

Hi @mberr . Thank you for your rapid response. I indeed checked the outline of the online document but did not expand the `First Steps` node. Sorry to increase your workload.

Hi @mberr . I have a further question. As shown in the document, the embedding is accessed by index. How can I access by the name of entity and relationship? Thanks.

@GabrielLin would you please open a new issue for this? I think it would be a good idea to add the answer to this question to the same section of the docs, since this is a pretty common thing to want to do

This issue is also related: https://github.com/pykeen/pykeen/issues/465

and this one, too: https://github.com/pykeen/pykeen/issues/480 :sweat_smile: 

Hi @cthoyt , I have created a new issue #526 .

@mberr , thank you for your help.
Repository: pykeen/pykeen
Issue: How do I know all the optional parameters？
GitHub labels: question

Hi, thanks for your job!
     I'm currently doing a parameter search on a dataset, But i am not sure how can I know all the optional parameters？ I had read the file : 

https://pykeen.readthedocs.io/en/stable/tutorial/running_hpo.html.      **Optimizing a Model’s Hyper-parameters**

for exmple below,  what can I put in the parameter `model_kwargs` ?
```
from pykeen.hpo import hpo_pipeline
hpo_pipeline_result = hpo_pipeline(
    dataset='Nations',
    model='TransE',
    model_kwargs=dict(embedding_dim=20, scoring_fct_norm=1),
    training_kwargs=dict(num_epochs=100, batch_size=128),
)
```

Each model has its own kwargs that correspond to the model class’s init function. You can check the reference section of the documentation on models (https://pykeen.readthedocs.io/en/stable/reference/models.html) for a listing of each 

thank you,  i found model parameters.  But how can i find other parameters such as :
```
loss_kwargs
regularizer_kwargs
optimizer_kwargs
negative_sampler_kwargs
training_kwargs
loss
...
```

Hi @quqxui ,

the parameters you mentioned seem to be the parameters of the `hpo_pipeline` function. There is a detailed documentation here: https://pykeen.readthedocs.io/en/stable/reference/hpo.html#pykeen.hpo.hpo_pipeline
Repository: pykeen/pykeen
Issue: Add Wikidata5M
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

One of the biggest available Wikidata-based graphs: 
~5M entities
~20M edges
822 relations

It comes in both transductive and inductive flavors, added are textual descriptions of every node (to be passed through some LM for featurizing for the inductive setup and/or joint pre-training with text).
We can start from the transductive-only split w/o additional features.

**Dataset is pre-stratified into train/test/valid** [ ]

Yes. 

**Publication(s) or website describing benchmark KG dataset**:

Website: https://deepgraphlearning.github.io/project/wikidata5m
Paper: https://arxiv.org/pdf/1911.06136.pdf - accepted to TACL

**URL for downloading the benchmark KG dataset**:

https://www.dropbox.com/s/6sbhm0rwo4l73jq/wikidata5m_transductive.tar.gz?dl=1



@migalkin I don't want to worry about this dropbox folder just disappearing. Can you petition the authors of this paper to post this data on an archive like Zenodo?

> @migalkin I don't want to worry about this dropbox folder just disappearing. Can you petition the authors of this paper to post this data on an archive like Zenodo?

Sure, I'll inquire about that. The dump is just 160 MB so it shouldn't be a tedious task

Otherwise this looks pretty straightforwards and I can take care of it. Thanks