zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: Use primitive data types in stopper's summary dicts
Root cause: Validation Gap

Hi guys. Minor thing ahead :)

**Is your feature request related to a problem? Please describe.**

The dictionary returned by `get_summary_dict()` on Stopper instances contains `numpy.float64` instead of the primitive python data types. This leads to confusing behavior for example when writing this dict to yaml.

```python
type(stopper.get_summary_dict()['best_metric'])
# <class 'numpy.float64'>
import yaml
print(yaml.dump(stopper.get_summary_dict()))
```

```
best_epoch: 2
best_metric: !!python/object/apply:numpy.core.multiarray.scalar
- &id001 !!python/object/apply:numpy.dtype
  args:
  - f8
  - false
  - true
  state: !!python/tuple
  - 3
  - <
  - null
  - null
  - null
  - -1
  - -1
  - 0
- !!binary |
  7Kdkss4BzT8=
frequency: 1
larger_is_better: true
metric: hits_at_k
patience: 50
relative_delta: 1.0e-05
results:
- !!python/object/apply:numpy.core.multiarray.scalar
  - *id001
  - !!binary |
    gDOcw/pKyD8=
- !!python/object/apply:numpy.core.multiarray.scalar
  - *id001
  - !!binary |
    7Kdkss4BzT8=
stopped: false
```
This concerns the fields `results` and `best_metric`

```python3
{k: type(v) if type(v) is not list else type(v[0]) for k, v in self.stopper.get_summary_dict().items()}
# {'frequency': <class 'int'>, 'patience': <class 'int'>, 'relative_delta': <class 'float'>, 'metric': <class 'str'>, 'larger_is_better': <class 'bool'>, 'results': <class 'numpy.float64'>, 'stopped': <class 'bool'>, 'best_epoch': <class 'int'>, 'best_metric': <class 'numpy.float64'>}
```

**Describe the solution you'd like**
Either explicitly convert the data to primitives or document this in the type annotation.

**Describe alternatives you've considered**

What I currently do after retrieving the dictionary:
```python
dic['best_metric'] = float(dic['best_metric'])
dic['results'] = list(map(float, dic['results']))
```

Happy to implement this and place a PR if you like. Should it be converted when setting/adding or afterwards, when the dictionary is created and returned?


Hi @kantholtz ,

thanks for finding this issue, and offering your help.

From my point of view, the best point of conversion would be here
https://github.com/pykeen/pykeen/blob/9bbb7c0c4f2054f3c920461aa70562e33267212c/src/pykeen/evaluation/rank_based_evaluator.py#L433-L438
when first computing the evaluation metrics.

Let me know when you have further questions or need assistance.
Repository: pykeen/pykeen
Issue: Closed World Training Loop
GitHub labels: good first issue

⚽ Training Loop

PyKEEN currently implements two training loops:

1. The local closed world assumption (LCWA). People often mistakenly call this the _closed world assumption_)
2. The _stochastic_ closed world assumption (sLCWA). People often mistakenly call this the _open world assumption_)

Training for the link prediction task under the open world assumption is mostly nonsensical (some negatives are necessary for meaningful results, even if they have to be randomly sampled).

Training for the link prediction task under the under the closed world assumption is usually not meaningful since all positive and negative links are already known. These graphs don't exist in reality in most domains or they aren't interesting for KGEMs and link prediction methodologies.

However, we do have some nice ways of generating toy graphs (see [`geometric-graphs`](https://github.com/cthoyt/geometric-graphs) and example training/evaluation of KGEMS on them at https://github.com/cthoyt/translational-toys) for which the positive and negative edges are known by definition. It would be useful to apply a CWA training loop in these kinds of situations where there doesn't need to be a training/testing stratification and evaluation can be done based on another metric (e.g., the closeness of the learned representations to the desired geometry).

One reason not to do this: if CWA is available, people might accidentally use it when it's not appropriate
Repository: pykeen/pykeen
Issue: reproducible experiments
Root cause: Configuration Error

**Describe the bug**
I want to use the experiment setting published in ''Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework.''. One setting is "training_loop": "OWA". But when I run, it shows that ''ValueError: Invalid TrainingLoop name: OWA. Valid choices are: ['lcwa', 'slcwa']''.

**To Reproduce**
Steps to reproduce the behavior:
1. Use the pipeline and correponding para setting.
2. Run the code 

**Expected behavior**
Run normally and could get the performance like you did after experiments.



Hi @lzw950905,
We have updated the nomenclature in the subsequent versions. `owa` has been renamed to `slcwa` (stochastic local closed world assumption).

@mali-git shouldn’t we update the config files to reflect the new name?

> @mali-git shouldn’t we update the config files to reflect the new name?

Maybe we can add a conversion script (in the benchmarking repository)? This would ensure that the configurations can be executed with the version used for the experiments and ensures that the configurations can be used with the current version.
Repository: pykeen/pykeen
Issue: Demonstrate re-implementation of R-GCN using PyTorch Geometric

There's an implementation of the R-GCN convolution in [`torch_geometric.nn.conv.RGCNConv`](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.RGCNConv). It would be interesting to see if it would be possible to re-implement the R-GCN model using this module, since it could give insight on how to choose arbitrary other PyTorch Geometric modules inside PyKEEN as _encoders_ (whereas interaction models like TransE, DistMult, etc. are _decoders_)

(cc @mali-git)

This was done in https://github.com/pykeen/pykeen/pull/620 but rejected.
Repository: pykeen/pykeen
Issue: ID Misalignment for FB15k237
Root cause: Validation Gap

**Describe the bug**
The entity-to-id mapping from the training part is not used for validation. Thus, their number of relations differ.

**To Reproduce**
```python-console
>>> dataset = get_dataset(dataset="fb15k237")
>>> dataset.train.num_relations, dataset.validation.num_relations
(237, 223)
```

**Expected behavior**
The mappings have to be consistent.

**Environment:**

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.11.0-16-generic        |
| Time            | Tue May 11 10:54:46 2021 |
| Python          | 3.9.4                    |
| PyKEEN          | 1.4.1-dev-c1ca656c       |
| PyTorch         | 1.8.1+cu102              |
| CUDA Available? | false                    |
| CUDA Version    | 10.2                     |
| cuDNN Version   | 7605                     |


**Additional information**
We should revisit all existing datasets to check for similar problems.
Repository: pykeen/pykeen
Issue: Micro vs. Macro Evaluation
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

The existing evaluation framework computes one rank value per evaluation triple and side, and then aggregates the ranks. For simplicity we discuss only the tail prediction here.

In the evaluation triples we can have multiple true tails `{t_1, ..., t_k}` for the same head-relation pair `(h, r)`. Depending on how the link prediction task is interpreted, these `k` triples actually belong to one link prediction task `(h, r, ?)`. Yet, we compute `k` ranks, and thus this one task influences the final aggregation more than the task `(h', r', ?)` with fewer correct answers. This situation is similar to class imbalance problems in classification settings, and parallel to them, we can also define micro and macro evaluation here:

The *micro* evaluation is what we have so far: We aggregate all individual ranks with uniform weight. The *macro* evaluation proceeds differently by first aggregating ranks for the same `(h, r)` prefix, and then aggregating the results globally.

_Acknowledgements: This is mostly a result of a discussion with @miselico. Similar ideas for a related task have been discussed earlier with @sharifza (cf. [here](https://arxiv.org/abs/1905.00966))._ 

**Describe the solution you'd like**
Add *macro* evaluation to at least the rank based evaluator. We can accomplish this easily by adding additional weights for each rank, and make sure to correctly treat the weights in aggregation. For some "non-linear" metrics, such as median rank, this may be non-obvious.

**Additional context**
* description of micro and macro F1 in [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html)


closed by https://github.com/pykeen/pykeen/pull/837
Repository: pykeen/pykeen
Issue: Implement square error loss
GitHub labels: enhancement

💧 Loss

💎 New Component

Blocked by #18

The  square  error  loss  function computes the squared difference between the predicted scores and the labels l_i ∈ {0,1}:

L(t_i,l_i) = (1/2)(f(t_i)−l_i)^2

The  squared  error  loss  strongly  penalizes  predictions  that deviate  considerably  from  the  labels,  and  is  usually  used  for regression problems. For simple models it often permits more efficient optimization algorithms involving analytical solutions of sub-problems, e.g. the Alternating Least Squares algorithm

Besides the 1/2 term this should be MSELoss

https://github.com/pykeen/pykeen/blob/ef83e7e45515e00e0cfcf0cbaee132923999d691/src/pykeen/losses.py#L132-L141

Oh whoops, I missed that. Why did we include the 1/2 in the paper but not in the implementation?

> Oh whoops, I missed that. Why did we include the 1/2 in the paper but not in the implementation?

Sorry for the belated answer; I somehow missed the notification.

The 1/2 is often added in front of squared loss in order to have a nicer derivative for analytical derivations (since it cancels the `2*...` of the `...^2` derivative). For the location of the minimum is does not play a role (since `argmin f(x) = argmin c*f(x)` for `c != 0`).
Repository: pykeen/pykeen
Issue: QuatE doesn't do anything with its default regularizer
Root cause: Validation Gap

https://github.com/pykeen/pykeen/blob/e762cae2dedacd86d3d89d6a5e277d60c45b89b2/src/pykeen/models/unimodal/quate.py#L79-L86

The default regularizer is neither applied to the entitity nor relation embeddings in the `__init__` of the QuatE class. @mali-git @mberr since you were responsible for the original QuatE PR, could you look into this? It's also causing bugs in the ablation pipeline (see #451)

Relevant PR: #367

This is from the paper (but for `p=2` normalization).

![Screenshot from 2021-05-19 11-42-30](https://user-images.githubusercontent.com/36984984/118791783-5759af80-b897-11eb-8331-9e31bb5b452e.png)

The N3 regularization is not explicitly described in the paper, so we need to check the code again.


In the code there is no trace of N3. I suggest we switch the default to L2.
Repository: pykeen/pykeen
Issue: Implement Pointwise Hinge Loss
GitHub labels: enhancement

💧 Loss

💎 New Component

Blocked by #18

The pointwise hinge loss sets the score of positive examples larger than a margin parameterλwhile  reducing  the  scores  of  negative  examples  to  values below−λ:

L(t_i,l_i) = max(0,λ−ˆl_i * f(ti))

where ˆl_i ∈ {−1,1}. The  loss  penalizes  scores  of  positive examples which are smaller than λ, but does not impose any restriction on value s> λ. Similarly, negative scores larger than−λcontribute to the loss, whereas all values smaller than−λ do  not  have  any  loss  contribution.  Thereby,  the  model  is not  encouraged  to  further  optimize  triples  which  are  already predicted well enough (according to the margin parameter λ).
Repository: pykeen/pykeen
Issue: Report Bug when I train the model using pipeline
Root cause: External Dependency Breakage

**Describe the bug**

When I train the model using pipeline, there is a bug about loss. 
loss_resolver.synonyms[_synonym] = _cls
TypeError: 'NoneType' object does not support item assignment

The loss I use is 'MarginRankingLoss', my code was:
from pykeen.pipeline import pipeline

pipeline_result = pipeline(
    dataset='FB15k',
    model='DistMult',
    model_kwargs={
        'embedding_dim': 100,
    },
    regularizer='Lp',
    regularizer_kwargs= {
      "apply_only_once": False,
      "weight": 0.0001,
      "p": 2.0,
      "normalize": False
    },
    optimizer='Adagrad',
    optimizer_kwargs=dict(
        lr=0.1,
    ),
    loss='MarginRankingLoss',
    loss_kwargs={
        "reduction": "mean",
        "margin": 1
                   },
    training_loop='slcwa',
    negative_sampler="basic",
    negative_sampler_kwargs= {
      "num_negs_per_pos": 2
    },
    training_kwargs=dict(
        num_epochs=1,  # more epochs than before
        batch_size=256,
    ),
    evaluator_kwargs={
        "filtered": True
    },
    filter_validation_when_testing=False,

)
pipeline_result.save_to_directory('fb15k_distmult')


**Expected behavior**
Train the model successfully.

**Environment:**

**Additional information**
I used to use pipeline and marginrankingloss, no bug reports. I don't know if you update the code?


Do `pip install -U class-resolver`

it works, thank you!
Repository: pykeen/pykeen
Issue: Ablation study error
Root cause: External Dependency Breakage

**Describe the bug**
When I run my ablation code, there is always bug reported:
'Trial 0 failed, because the value None could not be cast to float.'
I use GPU, and I have checked I did not use None.

**To Reproduce**
Steps to reproduce the behavior:


**Expected behavior**
[A clear and concise description of what you expected to happen.]
run the ablation study successfully.

**Environment:**


**Additional information**


Hi @lzw950905 I'm pretty sure this is happening because of the HPO code here:

https://github.com/pykeen/pykeen/blob/b33d8c3b13fae5ace983d7b52ea4b9afb72c2cba/src/pykeen/hpo/hpo.py#L242-L245

I have to double check Optuna's documentation - I think it used to be the case that returning `None` signified failure in older versions but there might be something to make it more explicit now, such as an exception. Perhaps you'd be interested to look into this and make a contribution to PyKEEN?

I am seeing the same error. Here is code to reproduce:
```python
from pykeen.hpo import hpo_pipeline
from pykeen.datasets import DRKG

drkg = DRKG()

evaluation_relation_whitelist = {
    i for i in drkg.relation_to_id.keys()
    if 'disease' in i.lower()
}

hpo_pipeline_result = hpo_pipeline(
    n_trials=20,
    dataset='DRKG',
    model='CompGCN',
    evaluation_relation_whitelist=evaluation_relation_whitelist,
)
```

Seems to have been fixed by #724, too.
Repository: pykeen/pykeen
Issue: Add InteractE
Root cause: Unimplemented Feature Gap

**On which paper(s) is your requested model based.**
* Conference Proceedings: https://ojs.aaai.org//index.php/AAAI/article/view/5694
* Preprint: https://arxiv.org/abs/1911.00219
* Supplementary: https://shikhar-vashishth.github.io/assets/pdf/interacte_supp.pdf

**List the original and other existing implementations of this model**
* official implementation: https://github.com/malllabiisc/InteractE

**Relevance of this model**
`InteractE` seems to be an extension of `ConvE` with a different reshaping / permutation operation, and using depth-wise convolution, and thus an interesting comparison partner to ConvE.
Repository: pykeen/pykeen
Issue: Implement Pairwise Logistic Loss
GitHub labels: enhancement

💧 Loss

💎 New Component

Blocked by #18

The pairwise logistic loss is defined as:

L(∆) = log(1 + exp(∆))

Thus, it can be seen as a soft-margin formulation of the pairwise hinge loss (MRL) with a margin of zero.

This is done, but in #540 we might make a subclass of MRL that makes it more explicit
Repository: pykeen/pykeen
Issue: AttributeError: 'PairRE' object has no attribute 'entity_embeddings'
Root cause: Documentation Deficiency

After completing the training, I get the trained_model.pkl file, but after I load the trained_model.pkl file, this prompt appears.
AttributeError: 'PairRE' object has no attribute 'entity_embeddings'
Isn't every model going to have the same properties? 
This bug prompt did not appear after my results from the TransE training.

This is my code:

```python
model = torch.load(f'./PairRE_200d/trained_model.pkl')
entity_embeddings = model.entity_embeddings
```

While old-style models have `model.entity_embeddings`, new style models (e.g., subclasses of [`ERModel`](https://pykeen.readthedocs.io/en/stable/api/pykeen.models.ERModel.html#pykeen.models.ERModel)) can have an arbitrary number of representations which are stored in `model.entity_representations`. PairRE is a new-style model, so you should try doing the following:

```python
model = torch.load(f'./PairRE_200d/trained_model.pkl')
entity_embeddings = model.entity_representations[0]
```

This is more explicitly written out in https://pykeen.readthedocs.io/en/latest/tutorial/first_steps.html#using-learned-embeddings

See also the comment from here: https://github.com/pykeen/pykeen/issues/465#issuecomment-848393810
Repository: pykeen/pykeen
Issue: How to set hyperparameters in ablation pipeline?
Root cause: Documentation Deficiency

When I run the ablation study, I have found two issues I want to ask:

1) I set the choices for training batch_size, but it never uses my choices. So it seems that some hyperparameters could not set the range.

2) I want to select different regularizers, like lp1,2,3, but it seems even though I set, the lp deault LP2, right?

3) I am not sure about the file '0000_nations_complex' like this kind of file, '0000' means one trial or what? Because the file produces automatically, but I don't know if each file represent one trial? It seems not, even though I set 30 trials, it produced 7 files. But how could I check each trial performance?
Repository: pykeen/pykeen
Issue: TriplesFactory not recognising all Triples
Root cause: Documentation Deficiency

I am trying to compute Knowledge Graph Embeddings using TransR and RotatE and have a pre-defined train, test and valid sets of data (I have three TSV files, each with 3 columns indicating heads, relations and tails). I have approximately 70,000 entities and 41 relations_types

Based on my understanding, prior to running the models through pipeline I have to generate a TriplesFactory. 

The issue I am having is that the final TriplesFactory Result has the incorrect num_relations and num_entities. 

I am not sure what I've been doing incorrectly, but I assume it's do with how I have structured my TSV file. 

The relations I used can be found at this [link](https://zenodo.org/record/1134693#.YLdt8JMzYc8) and are the strings (letters) in the brackets, while the entities are strings of identifiers (DB00001, MESH:D017299 etc).

I have added a screenshot of this code chunk with the incorrect values retrieved from TriplesFactory and the values I intend to have. 
<img width="936" alt="Screenshot 2021-06-02 at 1 45 54 PM" src="https://user-images.githubusercontent.com/29537935/120474750-ddeab280-c3a8-11eb-926d-89ce768fcfa6.png">

Thanks for your help! 

Link to Data to replicate code: 
https://zenodo.org/record/4884836/files/KG_FINAL.tsv?download=1

I think this is an issue with export of your data frame. You have the right number of triples + 1 because you should not export a header row. The number of unique entities is high because you didn't include `index=False` and the first column is a numerical index. The triples factory uses the first three columns - perhaps if you check out the TSV you'll see what I mean 

> I think this is an issue with export of your data frame. You have the right number of triples + 1 because you should not export a header row. The number of unique entities is high because you didn't include `index=False` and the first column is a numerical index. The triples factory uses the first three columns - perhaps if you check out the TSV you'll see what I mean

Oh my, that makes a lot of sense!
Thank you for your help and replying so quickyl!
Repository: pykeen/pykeen
Issue: How could we store the optimization result in HPO_pipeline?
Root cause: Documentation Deficiency

When I run the hpo_pipeline, I found there was no record for me to see the hpo results. I could just see some results in terminal. I think ablation_pipeline supports the perfect records but it seems a little unstable. 

So could you add the documentation about how could we store the intermediate and final results for hpo_pipeline?
Repository: pykeen/pykeen
Issue: Implement weighted geometric mean rank
Root cause: Unimplemented Feature Gap

The authors of *Revisiting the Evaluation Protocol of Knowledge Graph Completion Methods for Link Prediction* (@sudhanshu1994 and @crrivero) propose the *weighted geometric mean rank* metric, which has the nice properties that it takes into account the number of possible negative samples for each positive triple. 

<img width="648" alt="Screen Shot 2021-06-07 at 17 45 04" src="https://user-images.githubusercontent.com/5069736/121091516-1a0e8080-c7b8-11eb-9cb8-8e628b966326.png">

where C_S and C_O are acronyms for the sets of corrupted triples C_S((s, p, o), G_Y ) and C_O((s, p, o), G_Y ), respectively.

The source code is available at https://github.com/crrivero/www21. Unfortunately, it's based on OpenKE, which as far as I can tell has been defunkt for almost 2 years. Perhaps the authors would be interested in collaborating on re-implementing it in PyKEEN

The paper also mentions the geometric mean rank as well as adjusted variants of both the geometric mean rank and weighted geometric mean rank à la @mberr's framework for the adjusted arithmetic mean rank (and its associated index). However, I didn't see any derivation of these in the main text nor any related supplemental information attached on the ACM website. I haven't read the code yet, so we'll see if it's available there in a way we can understand it.

## Citation

- Web: https://doi.org/10.1145/3442381.3449856
- PDF: [3442381.3449856.pdf](https://github.com/pykeen/pykeen/files/6612170/3442381.3449856.pdf)

```bibtex
@inproceedings{10.1145/3442381.3449856,
  author = {Tiwari, Sudhanshu and Bansal, Iti and Rivero, Carlos R.},
  title = {Revisiting the Evaluation Protocol of Knowledge Graph Completion Methods for Link Prediction},
  year = {2021},
  url = {https://doi.org/10.1145/3442381.3449856},
  doi = {10.1145/3442381.3449856},
  booktitle = {Proceedings of the Web Conference 2021},
  pages = {809–820},
}
````

Hi @cthoyt,

We have the metrics implemented in our Evaluator (https://github.com/crrivero/www21/blob/main/Code/Evaluator.py#L230). It has a couple of tricks but I hope it is understandable.

We used OpenKE since it was easy for us to get started with. We have implemented several features already over OpenKE. In a previous project, we implemented fully in Python training and evaluation of different strategies to generate negative triples (https://github.com/crrivero/ImpactNegativeTriples). The WWW21 project (https://github.com/crrivero/www21/) includes all the features mentioned in the paper: metrics, validation step based on Wilcoxon, new splits, realistic ranks. The paper by @mberr et al. regarding ranks and metrics was really insightful and we decided to take a step further.

If you need any clarifications, please, just let us know.

Thanks,
Carlos