zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: Add CoDEx Dataset
Root cause: Unimplemented Feature Gap

Hi all, I'm the maintainer of the CoDEx benchmark ([GitHub repo](https://github.com/tsafavi/codex), [EMNLP 2020 paper](https://arxiv.org/pdf/2009.07810.pdf)), a new set of datasets for knowledge graph completion extracted from Wikidata and Wikipedia. The goal of the CoDEx project is to address the shortcomings of existing Freebase-based benchmarks like FB15k and FB15k-237 in several areas:

- **Comprehensiveness**: CoDEx combines knowledge from several sources, including positive/negative relational triples, multilingual text, and entity types;
- **Interpretability and coverage**: Wikidata covers a broader set of knowledge domains than Freebase, is multilingual, and has easier-to-interpret relations; and 
- **Difficulty**: We showed in our paper that a relatively large proportion of link prediction queries on FB15k-237 can be covered with frequency rules. 

I chatted with @migalkin yesterday at EMNLP and he expressed interest in including the CoDEx datasets in PyKEEN. If this is a direction that you'd like to pursue, let me know which part(s) of CoDEx you are interested in including (i.e., just the relational parts, or the text as well), and I can make a PR. 


 Hi Tara, thanks for the chat yesterday! Indeed, it would be very cool to have CODEX S/M/L right in the library!

So far it looks to me that adding a new dataset (at least with triples for link prediction) as easy as, e.g., with [freebase](https://github.com/pykeen/pykeen/blob/master/src/pykeen/datasets/freebase.py), the only things needed:
* persistent URLs
* relative filenames

The rest should be automatically processed by data loaders. Tagging @mali-git and @cthoyt in case I missed some nitty-gritty detail

Hi @tsafavi, this sounds great! I'm probably the best one on the team to take care of this.

@migalkin is right - we need a persistent link to the data, and we can get creative with pre-processing or re-use one of the several harnesses for automatic acquisition of data. Is the data in https://github.com/tsafavi/codex/tree/master/data/triples/codex-s appropriate? I see there are three sizes so we can add 3 different datasets to PyKEEN.

**Caveat**: right now, PyKEEN doesn't support user-supplied negative triples, but this is definitely a good idea to include in the near future.

Hi @cthoyt, I'm glad to hear that! 

I see that PyKEEN supports remote tar files, in which case it might be easier to use the zipped versions of the relational triples in each dataset hosted on my website: 
```
https://tsafavi.github.io/assets/data/codex-s.tar.gz
https://tsafavi.github.io/assets/data/codex-m.tar.gz
https://tsafavi.github.io/assets/data/codex-l.tar.gz
```
For each, the filenames are `{train,valid,test}.txt`. In CoDEx-S and CoDEx-L, there are also files of negatives: `{valid_negatives,test_negatives}.txt`. I assume these would just be ignored in the data-loading process, but if they cause a problem for PyKEEN I can remove them.
Repository: pykeen/pykeen
Issue: The early stop mechanism is slow
Root cause: Configuration Error

### Describe the bug

I found that when I enabled early stopping, generating checkpoints was very slow each time, and I found that the maximum batch size that memory fits is not valid in early stopping. We need to set batch_size in evaluation_kwargs, and this is only useful for the first early stopping. The second early stop mechanism is again full of gpu memory which makes the verification process very slow and I don't know what to do about it.

### How to reproduce

```python
import pickle
import torch
import pandas as pd
from pykeen.models import ERModel
from pykeen.pipeline import pipeline
from pykeen.triples import TriplesFactory
from src.utils.utils import create_evaluation_table
from pykeen.nn.representation import Embedding

# 初始化随机种子
random_seed = 42

# 加载原始数据
triples_factory = TriplesFactory.from_path(
    path='data/KGE/pico.tsv',
    create_inverse_triples=False,
    header=0,
    column_names=['Head', 'Relation', 'Tail'],
    sep='\t'
)
training, validation, testing = triples_factory.split(
    ratios=[0.9, 0.05, 0.05],
    random_state=random_seed,
)

# 训练Pipeline
pipeline_result = pipeline(
    training=training,
    validation=validation,
    testing=testing,
    # 指定模型类型
    model='TransE',
    # 可以使用以字典的形式给出模型的参数model_kwargs，模型的默认超参数均基于最初发布该模型的论文中报告的最佳值进行选择。
    model_kwargs={
        "embedding_dim": 256,  # 嵌入维度
    },
    evaluation_kwargs={
        "batch_size": 8,
    },
    epochs=30,
    stopper='early',
    stopper_kwargs={
        "frequency": 2,  # 每2个epoch评估一次
        "patience": 5,  # 最多等待5次无提升
    },
    random_seed=random_seed
)
# 绘制损失曲线并保存
ax = pipeline_result.plot_losses()
ax.get_figure().savefig('checkpoint/KGE/loss_curve.png')
```

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | nt                       |
| Platform        | Windows                  |
| Release         | 10                       |
| Time            | Thu May 15 18:07:19 2025 |
| Python          | 3.10.16                  |
| PyKEEN          | 1.11.1                   |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 2.4.0                    |
| CUDA Available? | true                     |
| CUDA Version    | 11.8                     |
| cuDNN Version   | 90100                    |


### Additional information

_No response_

### Issue Template Checks

- [ ] This is not a feature request (use a different issue template if it is)
- [ ] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [ ] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed
Repository: pykeen/pykeen
Issue: sLCWA Evaluation Loop
Root cause: Unimplemented Feature Gap

### Problem Statement

Right now, the only evaluation loop implemented is for LCWA. This nomenclature implies the existence of other evaluation protocols, using the same ideas as the training loop.

### Describe the solution you'd like

Implementation of sLWCA evaluation loop

### Describe alternatives you've considered

It might make sense to make a different abstraction besides the LCWA/sLCWA/CWA axis

### Additional information

_No response_

### Issue Template Checks

- [x] This is not a bug report (use a different issue template if it is)
- [x] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)

I think the current `SampledRankBasedEvaluator` is actually doing an sLCWA style evalation in its `evaluate`!

great insight! related to the place you already commented in https://github.com/pykeen/pykeen/pull/1543 to decide how to do sampled evaluation
Repository: pykeen/pykeen
Issue: RGCN HPO pipeline error
Root cause: Validation Gap

On a default settings everytime the base interaction is ermlp i get an error that TypeError: ERMLPInteraction.init() missing 1 required positional argument: 'embedding_dim', is there a way to fix this in the hpo_pipeline, without just removing the ermlp interaction

pipeline_results = hpo_pipeline(
study_name="RGCN_hpo",
device='cpu',
n_trials=1,
dataset='nations',
model='RGCN'
)

hi @s160677, thanks for the comment but this is insufficient for a response. Please resubmit following the bug issue form: https://github.com/pykeen/pykeen/issues/new?template=bug.yml
Repository: pykeen/pykeen
Issue: Add Open Graph Benchmark (OGB) Loader and Datasets
Root cause: Unimplemented Feature Gap

Information: https://ogb.stanford.edu/docs/linkprop
Download: https://snap.stanford.edu/ogb/data/linkproppred/

```sh
pip install ogb
```

```python
from ogb.linkproppred import LinkPropPredDataset

dataset = LinkPropPredDataset(name='ogbl-biokg')

split_edge = dataset.get_edge_split()
train_edge, valid_edge, test_edge = split_edge["train"], split_edge["valid"], split_edge["test"]
graph = dataset[0]  # graph: library-agnostic graph object
```
Repository: pykeen/pykeen
Issue: Functions created by pykeen.utils.compose can't be pickled
Root cause: Architectural Constraint

This creates a problem for SE, DistMult, TransE, and TransR which all define their initializers using the compose function. The following code is the offender:

https://github.com/pykeen/pykeen/blob/2ef88ac25fc9097d52f0031e5e1c629a71382256/src/pykeen/utils.py#L234-L244

Perhaps the compose function could be replaced with a compose class

[this](https://stackoverflow.com/questions/1253528/is-there-an-easy-way-to-pickle-a-python-function-or-otherwise-serialize-its-cod) may be relevant; one suggestion: use [dill](https://pypi.org/project/dill/)

@mberr the issue is that `torch.save` is only naively wrapping `pickle.dump`. The solution in #170 gets around this much more elegantly anyway

> @mberr the issue is that `torch.save` is only naively wrapping `pickle.dump`. The solution in #170 gets around this much more elegantly anyway

Agreed. It also avoids additional dependencies :+1:
Repository: pykeen/pykeen
Issue: Python 3.9 Support
Root cause: Environment Incompatibility

Blocked by:

- [x] Optuna (https://github.com/optuna/optuna/issues/2034 / https://github.com/optuna/optuna/pull/1908)
- [ ] PyTorch (https://github.com/pytorch/pytorch/issues/45731)

TODO:
- [ ] Switch all CI stuff to also test on 3.9 (while still testing on 3.7 and 3.8)
- [ ] Try to recover from the following sorrow

Hi @cthoyt

Optuna supports Python3.9 (https://github.com/optuna/optuna/pull/1908) 🆕 

p.s. Thank you for using Optuna!

@crcrpar excellent! Thank you so much for the update and the wonderful software :)

@cthoyt PyTorch 1.7.1 supports Python 3.9, cf. https://github.com/pytorch/pytorch/releases/tag/v1.7.1

@mberr yeah, it said that, but it didn't work :/ The problem is there are no wheels available

I'll give it another try just to check though

@mberr just checked, still failing on both linux and windows
Repository: pykeen/pykeen
Issue: RTD build fails on main branch
Root cause: Configuration Error

https://github.com/pykeen/pykeen/runs/1475961189

I'm not sure if I like having github actions take care of this instead of readthedocs' integration... either way we need to look into what's going on here

@cthoyt Thanks for pointing this out.
It was simply a missing "https://" in the RTD web hook URL.
I have fixed it and it just passed - closing this issue now.
Repository: pykeen/pykeen
Issue: Extend Documentation For Negative Sampling
GitHub labels: documentation

We currently have comparably less documentation for negative sampling than the documentation of the other components (e.g., interaction models). We could add more information about negative sampling using the information in our benchmarking paper.

@mali-git can you give some more context please? (in the description of the issue)

> @mali-git can you give some more context please? (in the description of the issue)

Done.
Repository: pykeen/pykeen
Issue: Add Fractional Hits@K Documentation
GitHub labels: documentation

Since #17 was merged before I got a chance to make a fuss about there not being documentation, it needs to be revisited.

1. Since you touched it last, you're responsible for writing the documentation for the `RankBasedEvaluator.__init__()` which should include explanations of what all parameters do (`ks` and `filtered`) as well as an explanation of integer hits@k versus fractional hits@k. References required.
2. What happens when somebody uses a fractional hits@k downstream? right now all code interprets whatever happens after the @ as an integer. This will break all usages of `pipeline()` and `hpo_pipeline`. Please provide unittests showing fractional hits@k in use
Repository: pykeen/pykeen
Issue: Clinical Knowledge Graph (CKG) Dataset
Root cause: Algorithmic Error

Entities: ~16 million
Triples: ~220 million
GitHub: https://github.com/MannLabs/CKG
Download page: https://data.mendeley.com/datasets/mrcf7f4tc2/2 (version 2)
Download: https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/mrcf7f4tc2-1.zip or https://data.mendeley.com/public-files/datasets/mrcf7f4tc2/files/c0d058a2-adfa-4b96-97d9-c9ec7fc5adb9/file_downloaded

#### Dependencies:
* [x] Blocked by #196 because splitting algorithm is currently too slow for big datasets (with more than ~5 million triples)
Repository: pykeen/pykeen
Issue: Example literal dataset and tests for LiteralE models
Root cause: Unimplemented Feature Gap

Right now, there's no test dataset that has literals. The LiteralE paper gives versions of FB15k, YAGO3, and WN18 that have literals, but these are way too big for testing purposes.

- Create a "fake" dataset that at least has the right shape
- Update TriplesNumericLiteralsFactory to load it properly
- Create a Dataset base class that will allow numeric triple loading
- Add a pipeline test for ComplExLiteral and DistMultLiteral
Repository: pykeen/pykeen
Issue: New hyperbolic models: RotH, RefH, AttH
Root cause: Unimplemented Feature Gap

**On which paper(s) is your requested model based.**

Low-Dimensional Hyperbolic Knowledge Graph Embeddings
Chami *et al.* (2021)
https://arxiv.org/abs/2005.00545

**List the original and other existing implementations of this model**
Official pytorch implementation: https://github.com/HazyResearch/KGEmb
Tensorflow implementation: https://github.com/tensorflow/neural-structured-learning/tree/master/research/kg_hyp_emb

**Relevance of this model**
1. Hyperbolic thing blow my mind
2. More variety
3. Performing pretty well

**Ability to help and additional context**
Your background and ability to help with the implementation of the model.


Related: https://arxiv.org/abs/1905.09791 (#343) (and https://arxiv.org/abs/1705.08039, which is only uni-relational)

Yup we had issues in the development repo for those too. Since these guys did it in pytorch, I figured it would be relatively straightforwards to include them
Repository: pykeen/pykeen
Issue: Improve Sealant duplicate/inverse relation detection speed
GitHub labels: enhancement

Once #193 is solved, we can accelerate the inverse/duplicate relation by converting each relation to a [boolean sparse adjacency matrix](https://docs.scipy.org/doc/scipy/reference/sparse.html), and computing the jaccard similarity as

```python
j[r1, r2] = r1.multiply(r2).sum() / (r1 + r2).sum()
```

The inverse relation is given by `r2.T`

#### Dependencies
* [ ] #193 


This was done in #187
Repository: pykeen/pykeen
Issue: Calling .plot() on a pipline results in error - ModuleAttributeError: 'Embedding' object has no attribute 'weight'
Root cause: Interface Contract Mismatch

Hi,

Called .plot() on any pipline will result in the error: ModuleAttributeError: 'Embedding' object has no attribute 'weight'
This seems to be caused by the .plot_er() method as I am able to call plot_losses() ok. 

Thanks for your work on pykeen - it's a great package!

**Environment**
 - OS: Arch Linux
 - Python version: 3.8.5
 - pykeen 1.0.6.dev0
 - click 7.1.2
 - click-default-group 1.2.2
 - numpy 1.19.2
 - torch 1.7.0
 - tqdm  4.54.1

@sbonner0 thank you for point this out. Luckily, this was quite easy to fix. The accompanying pull request also includes some other updates I noticed while checking through the demo for the plotting.

In case you're interested - the error was due to our recent generalization of the concept of embeddings from `torch.nn.Embedding` to a more general representation module `pykeen.nn.RepresentationModule` (done in #132 and #136). Typically, people access the `weight` attribute from the `torch.nn.Embedding` class to represent all entities' embeddings in a 2-tensor. However, because we believe that there are more exotic possible representations that could be generated from other modes like images, audio, text, that the usage should be through a `forward()` method where dynamic calculation/lookup are possible, so we switched to using our own API rather than PyTorch's.

Hey @cthoyt - glad it was an easy fix for you and thanks for tackling it so promptly! 

I had a quick look at pykeen.nn.RepresentationModule, it looks very interesting. I had a quick through the PRs you referenced, I saw that it is now possible to chose initializations through HPO now - does that mean it can be chosen with Optuna? If so, is that documented anywhere?

@sbonner0 no, we haven't done that, but I can imagine a way we could set it up as a "categorical" variable. If you'd like that feature, it would be great to post a new issue!

Great thanks! I'll be diving into the possibilities of PyKeen and Optuna over the next few days - I'll open some new issues if I encounter anything. I'd also love to contribute directly if at all possible. 

Should I close this issue as you have already fixed it?

@sbonner0 this issue will be automatically closed upon merge of the PR, so please leave it for now.

Issues are welcome! We hope we can help (and the rest of the team will chime in if your issues are about one of their expertises too)

If you'd like to make some meaningful contributions you can either just dive in, or we can set up a short call to onboard you. We haven't had this yet outside of the core team, so it would be a learning process all around :)


Thanks @cthoyt - I'll dive using it for a week or so and see if there is anything I could contribute to meaningfully (I'd hate to slow you down or cause more issues than I would solve ;))
Repository: pykeen/pykeen
Issue: Fix issue with GPU and plots
Root cause: Validation Gap

two comments:
* directly use `__call__` instead of explicitly calling `forward()`
* Make sure to use `.cpu()` before calling `numpy()`, in case the tensor resides on a GPU. (Otherwise leads to `TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.`)

In total: `embeddings(None).detach().cpu().numpy()`

_Originally posted by @mberr in https://github.com/pykeen/pykeen/pull/205#discussion_r540093622_
Repository: pykeen/pykeen
Issue: R-GCN raises Optuna error
Root cause: Interface Contract Mismatch

If I select the RGCN model, the trials return None:
Optuna __init__() got an unexpected keyword argument 'base_model_cls'  ...
Choices for a categorical distribution should be a tuple of None, bool, int, float and str for persistent storage but contains <function symmetric_edge_weights at 0x2aac1328bb90> which is of type function.
Is there something I didn't realize? With other models it works fine.

Thank you for your feedback @LFGMUW. Our current RGCN model has some issues. We have a new implementation of RGCN under way which will solve this issue together with some others. Stay tuned :)

It's probably gucci now since we've made lots of nice updates
Repository: pykeen/pykeen
Issue: Strange jump in batch size choice during a HPO run causing excessive run times
Root cause: Algorithmic Error

Hi,

I've been having issues where the training batch size during a hpo_pipeline run will suddenly decrease, thus increasing the run time quite dramatically. For example I had a run this morning where the batch size sequence for trails went: 2832, 3932, 131. 

I gather there is some kind of optimization process automatically determining the best batch size. I wonder if this went wrong in this instance? Or is the GPU memory not being freed correctly perhaps? 

Do you think I would be best disabling the automated batch size tunning and setting it manually? 

Closing as batch size was one of the HPO params
Repository: pykeen/pykeen
Issue: Add CSKG dataset
Root cause: Unimplemented Feature Gap

Link: https://github.com/usc-isi-i2/cskg
Data: https://zenodo.org/record/4331372/files/cskg.tsv.gz columns 1,2,3 ("node1", "relation", and "node2")
Arxiv: https://arxiv.org/abs/2012.11490
Repository: pykeen/pykeen
Issue: Slow trail run times using a HPO pipeline but not using a normal pipeline
Root cause: Resource Mishandling

Hi,

Happy new year!

I have had some exceptionally long run times when using ConvE as part of a HPO pipeline. What is confusing me is that when I start a normal pipeline using the same set of parameters, I see run times nearly an order of magnitude less!

For example I had a single ConvE trial take over 4 days, reporting that it took 311.55s/epoch. I loaded the same set of hyper-parameters into a pipeline run and have had an average runtime of 47.83s/epoch. This is using the same python env and running on the same machine with a Tesla V100 GPU. 

So my questions is - is there something explicit in the HPO process that would cause such a discrepancy in run times? Optuna was running a random search if that help. 

This may well apply to other models as well - ConvE just happened to have the longest runtime of all the approaches I have tried thus far. 

Below I have included a graph I have made of the average trail duration - all seem quite high with ConvE, ConvKG and RESCAL being clear outliers. All runs were using Hetionet as the input graph and were using the same parameter search space. 

<img width="517" alt="image" src="https://user-images.githubusercontent.com/10208489/103785239-e07ff080-5032-11eb-80ff-35556037ddb2.png">

My env is:

- Python: 3.8.5
- pykeen: 1.0.6.dev0
- torch: 1.7.0


Hi @sbonner0,

could you please provide the `pipeline_config.json` file returned by the HPO-pipeline and the configuration that you used to configure the pipeline yourself?

Hi @mali-git thanks so much for your reply - sadly I do not have the `pipeline_config.json` yet as the number of trials I had asked for with ConvE has not yet completed. What I had done is load from the Optuna SQLite file and select enter the parameters from the long running trial manually into the pipeline object.

Closing this issue as I have not seen the issue again.
Repository: pykeen/pykeen
Issue: RuntimeError: CUDA out of memory.
GitHub labels: bug

I got a memory problem when training my data on RGCN.

Here are my codes.

```
from pykeen.models import RGCN
model = RGCN(triples_factory=training,automatic_memory_optimization=True)

from torch.optim import Adam
optimizer = Adam(params=model.get_grad_params())

from pykeen.training import LCWATrainingLoop
training_loop = LCWATrainingLoop(model=model, optimizer=optimizer)

training_loop.train(num_epochs=10, batch_size=1,slice_size=1,sub_batch_size=1)
```

I got the error 
`RuntimeError: CUDA out of memory. Tried to allocate 50.00 MiB (GPU 0; 15.75 GiB total capacity; 14.28 GiB already allocated; 19.62 MiB free; 14.74 GiB reserved in total by PyTorch) (malloc at /opt/conda/conda-bld/pytorch_1591914895884/work/c10/cuda/CUDACachingAllocator.cpp:289)`

Is there any solution to fix this problem?

Since R-GCN has a relatively large default embedding dimension (`embedding_dim=500`, cf. [here](https://pykeen.readthedocs.io/en/latest/api/pykeen.models.RGCN.html#pykeen.models.RGCN)), you can try to reduce this as follows

```python
from pykeen.models import RGCN
model = RGCN(triples_factory=training, embedding_dim=128, automatic_memory_optimization=True)
```

Thanks! It works. 
Another question is that I got a warning `"Calculations will fall back to using the score_hrt method, since this model does not have a specific score_h function. This might cause the calculations to take longer than necessary"` when I evaluated the model using the code `results = evaluator.evaluate(model, mapped_triples)` .
Could I close this warning?
@mberr 

TL;DR: Regarding correctness you can ignore this warning. It only informs about sub-optimal evaluation efficiency.

This message is caused since the R-GCN model's implementation does not provide an optimized `score_h` function for fast 1:n scoring (as described [here](https://pykeen.readthedocs.io/en/latest/tutorial/performance.html#tuple-broadcasting)). There is a relatively simple solution to it by falling back to the decoder's implementation of `score_h`. However, since we are generally not really satisfied with the current implementation of R-GCN, we are working on a [larger restructuring](https://github.com/pykeen/pykeen/pull/110), also addressing this issue.