zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: Providing users with an option to initialize entities and relations with pertained embeddings 
Root cause: Documentation Deficiency

In addition to the options which are already there for entity/relation initializations (xavier_uniform, ...), it would also be great to give a configurable option for users to load pre-trained embeddings of entities/relations to use them for initialization. 

For example: 
loading a tsv file or a NumPy array with pre-trained weights of relations/entities.  



Hi. 

While not possible via CLI, from within Python you can pass entity and relation initializers to the models. 
The initializers can be any method which receives the weight tensor as input, and returns the initialized weights.

A simple example for initialization from a numpy array (without any error checking etc) is
```python
entity_initializer = lambda t: torch.as_tensor(np_array, dtype=torch.float32)
```

**caveat** if you do this, there could be issues with pickling the resulting model in the end. We might have to provide a class-based wrapper for doing this (which is a bit silly, but would solve the problem) or make the entity_initializer a bit more polymorphic

Right, that is one way to go. Thanks.  

@mberr would be cool to add this to the docs
Repository: pykeen/pykeen
Issue: saving results from the pipeline with MultiModals
Root cause: Validation Gap

**Describe the bug**
saving the result of the pipeline generates an error. 

**To Reproduce**
Steps to reproduce the behavior:
```python
from pykeen.pipeline import pipeline
from pykeen.datasets.nations import NationsLiteral

result = pipeline(
    dataset=NationsLiteral, 
    model='DistMultLiteral',
)
result.save_to_directory('./drive/MyDrive/')
```
returns the following error:

```
python-traceback
/usr/local/lib/python3.7/dist-packages/torch/serialization.py in _save(obj, zip_file, pickle_module, pickle_protocol)
    474     pickler = pickle_module.Pickler(data_buf, protocol=pickle_protocol)
    475     pickler.persistent_id = persistent_id
--> 476     pickler.dump(obj)
    477     data_value = data_buf.getvalue()
    478     zip_file.write_record('data.pkl', data_value, len(data_value))

AttributeError: Can't pickle local object 'DistMultLiteral.__init__.<locals>.<lambda>'
```


**Environment:**

pykeen==1.4.1.dev0

I haven't checked it but there could also be the same error with the ComplExLiteral model. 

@GenetAsefa should be all fixed!
Repository: pykeen/pykeen
Issue: NationsLiteral for HPO from config via CLI
Root cause: Unimplemented Feature Gap

While running the following command:
```
!pykeen experiments optimize -d /content/drive/MyDrive/NationsLiteral/0000_user_data_distmult/  /content/drive/MyDrive/HPO_Configurations/hpo_config.json 
```
where the content of the hpo_config.json file is :

```json
{
  "type": "hpo",
  "metadata": {
    "title": "HPO over NationsLiteral with LCWA"
  },
  "pipeline": {
    "stopper": "early",
    "stopper_kwargs": {
      "frequency": 10,
      "patience": 2,
      "metric": "mean_reciprocal_rank",
      "relative_delta": 0.002
    },
    "dataset": "NationsLiteral",
    "dataset_kwargs": {
      "create_inverse_triples": false
    },
    "model": "DistMultLiteral",
    "model_kwargs": {
      "entity_initializer": "xavier_uniform",
      "relation_initializer": "xavier_uniform",
      "entity_constrainer": "normalize"
    },
    "model_kwargs_ranges": {
      "embedding_dim": {
        "type": "int",
        "low": 6,
        "high": 8,
        "scale": "power_two"
      }
    },
    "loss": "CrossEntropyLoss",
    "regularizer": "LPRegularizer",
    "regularizer_kwargs_ranges": {
      "p": {
        "type": "int",
        "low": 1,
        "high": 2,
        "q": 1
      },
      "weight": {
        "type": "float",
        "low": 0.01,
        "high": 1.0,
        "scale": "log"
      }
    },
    "optimizer": "adam",
    "optimizer_kwargs": {
      "weight_decay": 0.0
    },
    "optimizer_kwargs_ranges": {
      "lr": {
        "type": "float",
        "low": 0.001,
        "high": 0.1,
        "sclae": "log"
      }
    },
    "training_loop": "LCWA",
    "training_kwargs": {
      "num_epochs": 100
    },
    "training_kwargs_ranges": {
      "batch_size": {
        "type": "int",
        "low": 7,
        "high": 9,
        "scale": "power_two"
      },
      "label_smoothing": {
        "type": "float",
        "low": 0.001,
       "high": 0.1,
        "sclae": "log"
      }
    },
    "evaluator": "RankBasedEvaluator",
    "evaluator_kwargs": {
      "filtered": true
    },
    "evaluation_kwargs": {
      "batch_size": null
    }
  },
  "optuna": {
    "n_trials": 3,
    "timeout": 1000,
    "metric": "mean_reciprocal_rank",
    "direction": "maximize",
    "sampler": "random",
    "pruner": "nop",
    "storage": "sqlite:////content/drive/MyDrive/NationsLiteral/0000_user_data_distmult/optuna_results.db",
    "save_model_directory": "/content/drive/MyDrive/NationsLiteral/0000_user_data_distmult/artifacts"
  }
}

```
I get the following error:
```python-traceback
File "/usr/local/lib/python3.7/dist-packages/pykeen/datasets/__init__.py", line 124, in get_dataset
    raise ValueError(f'dataset is neither a pre-defined dataset string nor a filepath: {dataset}')
ValueError: dataset is neither a pre-defined dataset string nor a filepath: NationsLiteral

```
I see that the 'NationsLiteral' dataset is not among the list of predefined datasets. Adding it there would fix the issue.
Repository: pykeen/pykeen
Issue: Bring your own pre-startified data for MultiModal KGEs for both pykeen.pipeline.pipeline() and Pykeen.hpo.hpo_pipeline() 
Root cause: Unimplemented Feature Gap

I am interested in using my own data for optimization experiments with multimodal KGEs.

Is it already possible to provide the path to the numeric literals in an "hpo_config.json" file?  like the way the training/validation/testing sets can be provided as in:

```
"training": "./drive/MyDrive/userdata/train.txt",
    "testing": "./drive/MyDrive/userdata/test.txt",
    "validation": "./drive/MyDrive/userdata/valid.txt", 
```
something that would look like this:

```
"numerical-triples": "./drive/MyDrive/userdata/literals.tsv",
```
that would be great.  Looking forward to your reply on this. 

Hi @GenetAsefa, we haven't forgotten about this, but there are a few high priority things we've been focused on. I do like this idea and we will do it eventually.
Repository: pykeen/pykeen
Issue: Generalize embedding initializers
Root cause: Unimplemented Feature Gap

A simple example for initialization from a numpy array (without any error checking etc) is
```python
entity_initializer = lambda t: torch.as_tensor(np_array, dtype=torch.float32)
```

**caveat** if you do this, there could be issues with pickling the resulting model in the end. We might have to provide a class-based wrapper for doing this (which is a bit silly, but would solve the problem) or make the entity_initializer a bit more polymorphic

_Originally posted by @mberr in https://github.com/pykeen/pykeen/issues/346#issuecomment-800246121_

This could generalize both the bring your own pre-training and the literals cases
Repository: pykeen/pykeen
Issue: Add ConEx
Root cause: Unimplemented Feature Gap

**On which paper(s) is your requested model based.**

https://openreview.net/forum?id=6T45-4TFqaX ([pdf](https://openreview.net/pdf?id=6T45-4TFqaX))

**List the original and other existing implementations of this model**

author’s implementation: 
- (repo) https://github.com/dice-group/Convolutional-Complex-Knowledge-Graph-Embeddings
- (specific code) https://github.com/dice-group/Convolutional-Complex-Knowledge-Graph-Embeddings/blob/ff40f0ac76bb4cc5642c9ce6eaabf31d4ff2c0d2/models/complex_models.py#L71-L223

**Relevance of this model**

It’s a complex number model that uses convolutions, so why not 

**Ability to help and additional context**

original codebase seems readable enough
Repository: pykeen/pykeen
Issue: Multimodal benchmark KGs
Root cause: Unimplemented Feature Gap

[kgbench: A Collection of Datasets for Multimodal and Relational Learning on Heterogeneous Knowledge](https://openreview.net/forum?id=yeK_9wxRDbA) ([pdf](https://openreview.net/pdf?id=yeK_9wxRDbA), [GitHub](
https://github.com/pbloem/kgbench)) presents several multimodal benchmark knowledge graphs, some with literals (and even others with images and audio! but that's work for another day). It would be good to add these directly in PyKEEN.

CC @GenetAsefa these might be interesting to you as well

Yes, it is interesting. Thank you for sharing!
Repository: pykeen/pykeen
Issue: Is it possible to load S,P,O triples as integers in PyKEEN?
Root cause: Documentation Deficiency

Hello,

I had a question about using my own data in PyKEEN. Suppose I have an n-triples .tsv file as so:
```
S P O
- - -
1 2 3
1 2 4
5 7 3
...
```

I noticed that PyKEEN currently loads these triples as a string, and then (as far as I can tell) associates an integer ID with each of them. I then see that there is some semantic / textual analysis (such as removing potential inverses), which of course requires that the data be strings.

However, is it possible to deactivate that analysis? If so, is it also possible to tell the PyKEEN pipeline to load the triples as integers, as in the example above? I am pretty sure the answer to the second of those is no at the moment (from looking through the code), but I am not sure.

In the case that a model does not use triple label names but rather only their inter-relationships (such as TransE), it I think it would save some RAM to represent the data as integers rather than strings of characters.

Thank you,
--Jeffrey Sardina


Hi @Jeffrey-Sardina, the short answer is yes, but it's not well documented because there are other improvements we'd like to make before publicizing it. You can use the [`pykeen.triples.CoreTriplesFactory`](https://pykeen.readthedocs.io/en/stable/reference/triples.html#pykeen.triples.CoreTriplesFactory) instead of the [`pykeen.triples.TriplesFactory`](https://pykeen.readthedocs.io/en/stable/reference/triples.html#pykeen.triples.TriplesFactory), which takes in numeric tensors instead of string ones. The TriplesFactory is actually a subclass of the CoreTriplesFactory that has some extra information about the labeling. Pretty much anywhere you use a TriplesFactory you can also use a CoreTriplesFactory, except in functions where labels are required (e.g., the [link prediction workflow](https://pykeen.readthedocs.io/en/stable/tutorial/making_predictions.html), but you could trivially re-roll that yourself)

Many libraries doing similar things to PyKEEN have  "preprocessing" scripts that take in datasets and prepare them in this numeric format to help deal with this RAM issue. Eventually, I think we will also do the same. Happy friday!

Wonderful, thank you very much. Happy Friday to you as well!
Repository: pykeen/pykeen
Issue: Add AceKG
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

Academia, bibliometrics

**Dataset is pre-stratified into train/test/valid** No

**Publication(s) or website describing benchmark KG dataset**:

https://arxiv.org/abs/1807.08484

**URL for downloading the benchmark KG dataset**:

https://archive.acemap.info/acekg/#download

the url for downloading is rejected
Repository: pykeen/pykeen
Issue: What does filtered OpenBioLink dataset mean?
GitHub labels: question

I have searched the document. The word "filter" is used in scenario about excluding false negative sample in training and excluding true positive tuple in rank-based evaluation.

I still don't get what does that mean for " First Filtered OpenBioLink 2020 Dataset" and "Second Filtered OpenBioLink 2020 Dataset". Does that refers to the dataset quality in "https://github.com/OpenBioLink/OpenBioLink"?

Hi @jerryIsHere ,

first a disclaimer: I am not very familiar with this dataset;  @mali-git  or @cthoyt should know more about it, and likely will answer later on.


> I have searched the document. The word "filter" is used in scenario about excluding false negative sample in training and excluding true positive tuple in rank-based evaluation.
> I still don't get what does that mean for " First Filtered OpenBioLink 2020 Dataset" and "Second Filtered OpenBioLink 2020 Dataset". Does that refers to the dataset quality in "https://github.com/OpenBioLink/OpenBioLink"?

From what I know, this is correct. The "filtered" does *not* refer to filtered evaluation, which is what you explained in your first paragraph.


@jerryIsHere OpenBioLink has a canonical dataset ("high quality") and a "low quality" dataset. We were doing some internal experiments to determine if the composition of the canonical OpenBioLink dataset was really meaningful (e.g., we wanted to see if some of the datasets that it includes are not informative/don't contribute to good performance). Unfortunately this project is currently on the back-burner, and before we made some improvements to the HPO pipeline to bring your own data, we had to register it this way to make sure those experiments were reproducible. For your use case, it's probably best to use 'OpenBiolink' or 'OpenBioLinkLQ'.

FYI, the F1 dataset did the following: 

 - remove all pathway, GO and anatomy sources from the original network
 - remove all undefined edges from original network
 - flip dis-drug interaction to drug-dis

The F2 did the following on top of F1:

- remove associations (gene-gene, gene-binding-gene)
Repository: pykeen/pykeen
Issue: Adding extra model training parameters
Root cause: Unimplemented Feature Gap

In some config files there are some parameters like `entity_initializer `and `relation_initializer.` When I specify these in the pipeline under `model_kwargs,` such as giving uniform as a value, I get an error saying these are not recognizable.

I also want to specify some details in the regularization such as entity `embedding weight` and `relation embedding weight` but I don't see this in the documentation or the source code, there is only one weight parameter. Any chance these can be changed?

Can you provide an example of the code you’re using that gives this error?

@cthoyt Thanks for the reply. 

I have tried just using [this](https://github.com/pykeen/pykeen/blob/master/src/pykeen/experiments/convkb/nguyen2018_transe_fb15k237.json) config of TransE from the experiments folder  (both by using `pipeline_from_config` and just by using `pipeline` and filling the parameters) 


I also wanted to use pykeen replicate experiments used in the [paper](https://www.semanticscholar.org/paper/You-CAN-Teach-an-Old-Dog-New-Tricks!-On-Training-Teach-Ruffinelli/829af9ef30d654c697c0309036a9087666b2f3e3)  titled You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings. In page 14, they change the values I mention above (example: `embedding weight `, `relation embedding weight`, `entity embedding dropout`, `relation embedding dropout`, Specifying the number of object and subject samples for the negative sampler, etc...). 

In the pykeen source code, some of these can be changed (in this case: hidden dropout) for some models like [ConvKB](https://pykeen.readthedocs.io/en/stable/_modules/pykeen/models/unimodal/conv_kb.html?highlight=dropout#) but it is not present for [ComplEx](https://pykeen.readthedocs.io/en/stable/_modules/pykeen/models/unimodal/complex.html#ComplEx) model and others.
Repository: pykeen/pykeen
Issue: Add mean average precision (MAP) metric
Root cause: Unimplemented Feature Gap

See section 2.3 of https://www.mdpi.com/2078-2489/12/4/147/htm

<img width="700" alt="Screen Shot 2021-04-07 at 11 53 56" src="https://user-images.githubusercontent.com/5069736/113847624-f4550300-9797-11eb-893f-2d0244d7cc7d.png">


Would [sklearn.metrics.average_precision_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html) + `SklearnEvaluator` suffice?

That might be the case. We could roll this up in our evaluation reckoning I want to combine the sklearn evaluator and rank-based evaluator and have the functions be more like plugins. I don't think it's ever the case that sklearn is so slow or memory intensive that we couldn't always do it, right?

I have not benchmarked them. However, the rank based evaluator can run on GPU (for rank computation), and only the rank aggregation happens on CPU.
Repository: pykeen/pykeen
Issue: Add RotL
Root cause: Unimplemented Feature Gap

**On which paper(s) is your requested model based.**

"High-efficiency Euclidean-based Models for Low-dimensional Knowledge Graph Embeddings"
https://arxiv.org/abs/2103.14930

**List the original and other existing implementations of this model**

Paper claims the code is available on GitHub but there's no link inside and I wasn't able to find it

**Relevance of this model**

- Companion to #370
- cool
- not hyperbolic but like hyperbolic
Repository: pykeen/pykeen
Issue: Add Rot2L
Root cause: Unimplemented Feature Gap

**On which paper(s) is your requested model based.**

"High-efficiency Euclidean-based Models for Low-dimensional Knowledge Graph Embeddings"
https://arxiv.org/abs/2103.14930

**List the original and other existing implementations of this model**

Paper claims the code is available on GitHub but there's no link inside and I wasn't able to find it

**Relevance of this model**

- Companion to #369
- cool
- not hyperbolic but like hyperbolic
Repository: pykeen/pykeen
Issue: Bug in word cloud
GitHub labels: bug

```python-traceback
RuntimeError                              Traceback (most recent call last)
<ipython-input-11-3e30a69c8791> in <module>
----> 1 tf.entity_word_cloud()

~/dev/pykeen/src/pykeen/triples/triples_factory.py in entity_word_cloud(self, top)
    905             ``pip install git+https://github.com/kavgan/word_cloud.git``.
    906         """
--> 907         return self._word_cloud(
    908             ids=self.mapped_triples[:, [0, 2]],
    909             id_to_label=self.entity_labeling.id_to_label,

~/dev/pykeen/src/pykeen/triples/triples_factory.py in _word_cloud(self, ids, id_to_label, top)
    940         # pre-filter to keep only topk
    941         uniq, counts = ids.view(-1).unique(return_counts=True)
--> 942         top_counts, top_ids = counts.topk(k=top, largest=True)
    943 
    944         # generate text

RuntimeError: selected index k out of range
```

@cthoyt I suppose this happens if `top` is larger than the number of available options (entities in this case). A simple fix should be to set `top = min(top, uniq.numel())`

Yes @mberr that did the trick! I will include it in my super random PR
Repository: pykeen/pykeen
Issue: Add inverse geometric mean rank metric
Root cause: Unimplemented Feature Gap

@mberr do you think inverse geometric mean rank (IGMR) or reciprocal geometric mean rank (RGMR) sounds better?

Why not directly include all of them?

# Mean Rank
* arithmetic_mean_rank - MR (for backwards-compatiblity, but also since sadly AMR collides with adjusted mean rank :crying_cat_face: )
* geometric_mean_rank - GMR
* harmonic_mean_rank - HMR

# Mean Reciprocal Rank
* arithmetic_mean_reciprocal_rank - MRR (for backwards-compatibility, alternative AMRR)
* geometric_mean_reciprocal_rank - GMRR
* harmonic_mean_reciprocal_rank - HMRR

Median also seems to be interesting and straight-forward to implement (however, the only [holistic aggregation](https://www.sciencedirect.com/topics/computer-science/aggregate-function) so far)

Also, let's add `std`, so we can do tests with mean + std (I think you already talked about this some time ago).

You asked for it. Now you're going to be sorry 👻  here come 11 new ones :)))
Repository: pykeen/pykeen
Issue: Add Tensorboard Tracker
GitHub labels: 🐺 Tracker

💎 New Component

**Is your feature request related to a problem? Please describe.**
Tensorboard is another popular tracker. Integration seems easy, cf. https://pytorch.org/tutorials/recipes/recipes/tensorboard_with_pytorch.html

**Describe the solution you'd like**
Add a tracker for Tensorboard.
Repository: pykeen/pykeen
Issue: setup.cfg update request
Root cause: Configuration Error

Given the last updates to the rank_based_evaluator.py file (https://github.com/pykeen/pykeen/commit/a3a463a03a02efb24f1cec5ae583be765612a6b9), the setup.cfg should be updated in order to force the **scipy** library to be at least on version 1.5.0, since the function "median_abs_deviation" only exists from that version onwards.

Best regards,

Rodrigo Pereira


Hi @Rodrigo-A-Pereira ,

thanks for reporting! I'll take care of it now.

For reference, here is also the link to the function in scipy: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.median_abs_deviation.html
Repository: pykeen/pykeen
Issue: Connect pykeen to torch.hub
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
[torch.hub](https://pytorch.org/docs/stable/hub.html) allows to share pre-trained models. Providing pre-trained models enables re-use for down-stream applications without the need to train your own model.

**Describe the solution you'd like**
torch hub requires to create a `hubconf.py` file at the repository root, which contains entry-points for loading pre-trained models. The model weights do not need to be stored in this repository but can be stored externally.

**Describe alternatives you've considered**
There is already a way to serialize trained models to pickle files. A simple alternative is thus to share such plain files. Connecting to torch.hub allows to use a simple syntax built into torch itself. Other libraries, e.g., [pytorch/fairseq](https://github.com/pytorch/fairseq) or [huggingface/pytorch-transformers](https://pytorch.org/hub/huggingface_pytorch-transformers/) use this service and hence users might already be used to the syntax.
Repository: pykeen/pykeen
Issue: Update minimum version of pystow
Root cause: External Dependency Breakage

> @migalkin do `pip install --upgrade pystow`

Another candidate for updated minimum version requirement in `setup.cfg`, @cthoyt ?

_Originally posted by @mberr in https://github.com/pykeen/pykeen/issues/393#issuecomment-825066884_

Closed by #245
Repository: pykeen/pykeen
Issue: Update dataset resolution to class-resolver
Root cause: Unimplemented Feature Gap

> @cthoyt if I see it correctly, there is currently no way to select non-default graph_pair / side via the CLI. We might be able to fix it by using class-resolver to lookup the datasets.

> hmm I had considered switching this over, but there are a few tricky bits to the dataset handling we'd have to consider

_From the discussion [here](https://github.com/pykeen/pykeen/pull/403#discussion_r620021422)_
Repository: pykeen/pykeen
Issue: Update deprecated use of numpy.long
Root cause: External Dependency Breakage

```
.../src/pykeen/triples/triples_factory.py:112: DeprecationWarning: `np.long` is a deprecated alias for
 `np.compat.long`. To silence this warning, use `np.compat.long` by itself. In the likely event your
 code does not need to work on Python 2 you can use the builtin `int` for which `np.compat.long` 
is itself an alias. Doing this will not modify any behaviour and is safe. When replacing `np.long`, 
you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your
 current use, check the release note link for additional information.
  Deprecated in NumPy 1.20; for more details and guidance: 
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    triples_of_ids = np.array(triples_of_ids, dtype=np.long)
```
Repository: pykeen/pykeen
Issue: Port ConvKB to ERModel
Root cause: Architectural Constraint

The ConvKB model is the only one that takes care of the current usage of the new-style model's `ERModel.append_weight_regularizer` function. This happens **after** the call to `super().__init__()` and might be antithetical to #411, so it would be good to see how this will affect that PR.