zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Tests inside Scenarios not Filtered by --filter-pattern
Root cause: Validation Gap

config.yaml:
```
providers:
  - id: 'python:./holmes_runner.py'
    config:
      holmesFlags: "--verbose"

scenarios:
  - config:
    - vars:
        ... 

    tests:
      - description: SomeTest
        vars:
          ...
```

When I run the following, I would expected SomeTest to not run, but it does run:
```console
$ promptfoo eval -c config.yaml --filter-pattern nomatch
```


Cleaned up the example a little to remove irrelevant bits

To elaborate a little more on the motivation here, I have tests that should run on the same scenario data, but which can't run at the same time as the scenarios are not re-entrant. I would like to solve this by invoking promptfoo twice in a row, each time with a separate test. (I also considered having 2 promptfoo configs and re-using the scenario data as external var files - hence the other GitHub issue I opened recently.)

Fix in https://github.com/promptfoo/promptfoo/pull/996 Thank you @aantn! 

Wonderful, thanks. Any ideas when the next release will be?
Repository: promptfoo/promptfoo
Issue: Errors from custom providers are rendered in cli but not web UI
Root cause: Interface Contract Mismatch

When using a custom provider, errors show up in the cli:

<img width="1387" alt="Screenshot 2024-06-22 at 19 23 24" src="https://github.com/promptfoo/promptfoo/assets/494087/8f411b07-691f-4b4d-9201-3a0c1a2c1fd9">

But in the UI the test is marked as failure with no clear reason:

<img width="1165" alt="Screenshot 2024-06-22 at 19 21 50" src="https://github.com/promptfoo/promptfoo/assets/494087/f2f97034-4bcc-4f09-b95b-1e8952307a60">

It would be great to see the same error in the UI.

Here is a minimal reproducible example:

promptfoo config:
```
prompts:
  - "foo"

providers:
  - id: 'python:./test.py'

tests:
  - vars:
      x: y
    description: abc
```

python provider (test.py):
```
def call_api(prompt, options, context):
    return {
        "error": "something went wrong"
    }
```

Fix in #990.  Thanks @aantn 

Thank you very much as always.

Released in 0.66.0
Repository: promptfoo/promptfoo
Issue: Enhance UI to group `--repeat` tests into collapsible sections
Root cause: Unimplemented Feature Gap

I often test with `--repeat 20` to verify that output is reliable across multiple LLM runs. The resulting table output in the web viewer is hard to navigate due to all the duplicates.

It would be wonderful if I could get an overview of the tests and see at a glance how many times each test PASSED/FAILED out of the 20 runs.

Here is an idea for a simple UI improvement that could work for me: 

<details>
<summary>Description of Test1 - 18 PASS, 2 FAIL</summary>
Normal table output goes here for all 20 instances of this test
</details>
<details>
<summary>Description of Test2 - 20 PASS</summary>
Normal table output goes here for all 20 instances of this test
</details>

One more comment on this, as I've considered parsing the json output and building our own UI to show results.

In the json output, there is currently nothing that you can use to group together different --repeats of the same test. So a good first step would be to add to the json output an identifier that you can use to group all the repeats of the same test.
Repository: pykeen/pykeen
Issue: get_relation_prediction_df   not having ok results
Root cause: Documentation Deficiency

### What is your question

Hi, 

Thanks a lot for the pykeen package, it is great. I have followed the pykeen published paper, and tried extensively training on FB15k-237, the test score can match the best one on the paper. But when I tried to predict the relation,  get_relation_prediction_df, with different models, I can not get the "True" score on top 10 ranking even with training data. It works much better with tail prediction. Can you give me some suggestions? Like what kind of setup will work for the relation prediction, or this is expected, since the training is on tail/head predition?

Thanks a lot

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.4.0-1080-aws           |
| Time            | Wed Jun 29 22:10:32 2022 |
| Python          | 3.9.12                   |
| PyKEEN          | 1.8.2                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.11.0+cu102             |
| CUDA Available? | true                     |
| CUDA Version    | 10.2                     |
| cuDNN Version   | 7605                     |

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed
Repository: pykeen/pykeen
Issue: BioKG - Benchmark Datasets
GitHub labels: question

### What is your question

Hello team PyKEEN,

I have a question regarding the [BioKG](https://pykeen.readthedocs.io/en/stable/api/pykeen.datasets.BioKG.html) dataset (Welsh et al), that is included in `pykeen.datasets`.

Specifically, I wanted to know whether the benchmark datasets (FDA-DPI, DDI-efficacy etc) were included in your `train`, `validation` and `test` splits or whether you performed some preprocessing prior to the split generation.

Thank you for any hints and keep up the great work with pykeen!!

Best, 
Dimitrios

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Darwin                   |
| Release         | 21.5.0                   |
| Time            | Fri Jul  1 14:57:56 2022 |
| Python          | 3.8.5                    |
| PyKEEN          | 1.8.2                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.12.0                   |
| CUDA Available? | false                    |
| CUDA Version    | N/A                      |
| cuDNN Version   | N/A                      |

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

The PyKEEN dataset uses https://github.com/dsi-bdi/biokg/releases/download/v1.0.0/biokg.zip and randomly splits it using the same procedure that's applied to all datasets that aren't pre-split, which is detailed in https://pykeen.readthedocs.io/en/stable/api/pykeen.datasets.base.CompressedSingleDataset.html and relies on the [`TriplesFactory.split()`](https://pykeen.readthedocs.io/en/stable/reference/triples.html#pykeen.triples.CoreTriplesFactory.split) functionality

Alright! Thank you for the pointers :)
Repository: pykeen/pykeen
Issue: exporting triplets
GitHub labels: question

### What is your question

Hi I was wondering if I can export some of the pykeen datasets as triples/edge list format to be used in [CLEP](https://github.com/hybrid-kg/clep) as well?



Source | Relation | Target
-- | -- | --
HGNC_ID_1 | association | HGNC_ID_2
HGNC_ID_2 | decreases | HGNC_ID_3
HGNC_ID_3 | increases | HGNC_ID_1

Thank you.

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Darwin                   |
| Release         | 21.5.0                   |
| Time            | Sun Jul  3 12:46:27 2022 |
| Python          | 3.8.0                    |
| PyKEEN          | 1.8.3-dev                |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.12.0                   |
| CUDA Available? | false                    |
| CUDA Version    | N/A                      |
| cuDNN Version   | N/A                      |


### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

@cthoyt might know better than me about CLEP 😉 

For converting ID-based triples back to the label-based format, please take a look at https://pykeen.readthedocs.io/en/stable/reference/triples.html#pykeen.triples.TriplesFactory.label_triples
Repository: pykeen/pykeen
Issue: TypeError in  Pipline in Pykeen1.8.1 (TypeError: gmean() got an unexpected keyword argument 'weights)
Root cause: External Dependency Breakage

### Describe the bug

I have been using Pykeen 1.8.1with pipeline. It worked well without any problem, but today I am trying to rerun my code with BoxE or PairRE models, but I got this error 

`TypeError: gmean() got an unexpected keyword argument 'weights'`

Error: 

```
INFO:pykeen.pipeline.api:Using device: gpu
/usr/local/lib/python3.7/dist-packages/pykeen/nn/representation.py:372: UserWarning: Directly use Embedding.max_id instead of num_embeddings.
  warnings.warn(f"Directly use {self.__class__.__name__}.max_id instead of num_embeddings.")
INFO:pykeen.evaluation.evaluator:Starting batch_size search for evaluation now...
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-39-da37e65b885d>](https://localhost:8080/#) in <module>()
     10     model_kwargs =dict(embedding_dim=200),
     11     random_seed=1235,
---> 12     use_tqdm = False, device='gpu')


9 frames
[/usr/local/lib/python3.7/dist-packages/pykeen/metrics/ranking.py](https://localhost:8080/#) in __call__(self, ranks, num_candidates, weights)
   1052         self, ranks: np.ndarray, num_candidates: Optional[np.ndarray] = None, weights: Optional[np.ndarray] = None
   1053     ) -> float:  # noqa: D102
-> 1054         return np.reciprocal(stats.gmean(ranks, weights=weights)).item()
   1055 
   1056 

TypeError: gmean() got an unexpected keyword argument 'weights'
```

### How to reproduce

```
def_paire_model = pipeline(
    training=training,
    testing=testing,
    validation= validation,
    ## model argument 
    model='PairRE',   
    negative_sampler_kwargs = dict(
    corruption_scheme = ('head', 'tail')),
    model_kwargs =dict(embedding_dim=200),
    random_seed=1235,
    use_tqdm = False, device='cpu')
```

### Environment

<html>
<body>
<!--StartFragment-->

Key | Value
-- | --
OS | posix
Platform | Linux
Release | 5.4.188+
Time | Wed Jul 13 09:41:52 2022
Python | 3.7.13
PyKEEN | 1.8.1
PyKEEN Hash | UNHASHED
PyKEEN Branch |  
PyTorch | 1.12.0+cu113
CUDA Available? | true
CUDA Version | 11.3
cuDNN Version | 8302

<!--EndFragment-->
</body>
</html>

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

This is related to an too old version of `scipy`, cf. https://github.com/scipy/scipy/issues/13065. Please upgrade it. With the latest version of PyKEEN, this should be done automatically, cf. https://github.com/pykeen/pykeen/pull/891
Repository: pykeen/pykeen
Issue: Run "Getting Started with NodePiece"-Using NodePiece with pipeline
Root cause: Validation Gap

### Describe the bug

I get a bug: Base representations comprise only 475 representations, but at least 948 are required.

### How to reproduce

```python
import torch.nn


from pykeen.models import NodePiece
from pykeen.pipeline import pipeline

result = pipeline(
    dataset="fb15k237",
    dataset_kwargs=dict(
        create_inverse_triples=True,
    ),
    model=NodePiece,
    model_kwargs=dict(
        tokenizers=["AnchorTokenizer", "RelationTokenizer"],
        num_tokens=[20, 12],
        tokenizers_kwargs=[
            dict(
                selection="MixtureAnchorSelection",
                selection_kwargs=dict(
                    selections=["degree", "pagerank", "random"],
                    ratios=[0.4, 0.4, 0.2],
                    num_anchors=500,
                ),
                searcher="ScipySparse",
            ),
            dict(),  # empty dict for the RelationTokenizer - it doesn't need any kwargs
        ],
        embedding_dim=64,
        interaction="rotate",
    ),
)
```

### Environment

python -m pykeen 1.8.2

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Hi @jhrsya ,

could you please provide the full traceback and environment information?

closed in favour of #1033
Repository: pykeen/pykeen
Issue: `_safe_evaluate` still calls `model.to_cpu_()` and thus crashes when falling back to CPU if GPU OOMs
Root cause: Validation Gap

### Describe the bug

It seems the function `_safe_evaluate` still calls `model.to_cpu_()` to fallback to CPU when required. However this function no longer seems to be there in any of the base model classes and thus will fail if an out of memory error occurs on a GPU and evaluation fallback is set. I guess it should probably just be `.cpu()`?

### How to reproduce

``` python
from pykeen.pipeline import pipeline

pipeline_result = pipeline(
    dataset='Nations',
    model='TransE',
)
pipeline_result.model.to_cpu_()
```

### Environment

| Key             | Value                       |
|-----------------|-----------------------------|
| OS              | posix                       |
| Platform        | Linux                       |
| Release         | 3.10.0-1160.11.1.el7.x86_64 |
| Time            | Mon Jul 18 17:43:03 2022    |
| Python          | 3.9.5                       |
| PyKEEN          | 1.8.3-dev                   |
| PyKEEN Hash     | UNHASHED                    |
| PyKEEN Branch   |                             |
| PyTorch         | 1.11.0+cu113                |
| CUDA Available? | false                       |
| CUDA Version    | 11.3                        |
| cuDNN Version   | 8200                        |

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Hi @sbonner0 ,

thanks for catching the issue - we still have no automated way to test on GPUs 😕 

> I guess it should probably just be `.cpu()`?

That should already fix it - would you mind to open a PR for it, and maybe give the fix a try with the code you encountered the error in? 🙂
Repository: pykeen/pykeen
Issue: Removing validation set with (remove_known = True) in predict 
Root cause: Validation Gap

### Problem Statement

In prediction tails or heads  such as `get_head_prediction_df`, the option of (`remove_known=True`), it removes just the well known triples from training only and not validation set 

### Describe the solution you'd like

I think it should Remove the known validation triples available with `remove_known` in `get_head_prediction_df` or `get_tail_prediction_df`

### Describe alternatives you've considered

I removed validation sets after getting the results of predictions  

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)

Can you please provide an example? It's not so easy to understand what you want from what you wrote.

In the `get_tail_prediction_df ` documentation 

`remove_known: – Should non-novel triples (those appearing in the training set) be shown with the results?` 

My argument why you do not also include validation triples which they are also non-novel to model. 

In `remove_known= True`: it excludes the training triples from the prediction, is this right?  
If yes, I think it should exclude also the validation sets from results of prediction, because the model has seen before during training. 

Example: 
if this triple in training   set
x | increases | y
-- | -- | --

This triple in validation set

x | increases | i
-- | -- | --

I try to predict tail for this (x, increases,?) by this code: 

`get_tail_prediction_df(transe.model , "x", 'increases', triples_factory = model.training,  remove_known=True)
`
so I got  answers 

x | increases | u
-- | -- | --
x | increases | i
-- | -- | --

so i from validation also be in the results and ` remove_known=True`  just remove sets from training but not sets from validation.

Hi @Ahmed-fub ,

with #1048 being merged to `master` branch, you can now filter an arbitrary number of triples. The following is a minimal example using the built-in `nations` dataset (and skipping training by `training_kwargs=dict(num_epochs=0)`):

```python
from pykeen.datasets import get_dataset
from pykeen.predict import predict_target
from pykeen.pipeline import pipeline

dataset = get_dataset(dataset="nations")
# simulate training
result = pipeline(model="distmult", dataset=dataset, training_kwargs=dict(num_epochs=0))
# get predictions
pred = predict_target(
    model=result.model,
    head="uk",
    relation="conferences",
    triples_factory=result.training,
)
# filter prediction; you can use any number of filter triples here
pred_filtered = pred.filter_triples(dataset.training, dataset.validation)
# get the result as dataframe
pred_df = pred_filtered.df
```

Until our next release, you need to install from source to get this feature:
```console
pip install git+https://github.com/pykeen/pykeen.git
```
Repository: pykeen/pykeen
Issue: Add Freebase86M
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

The dataset is shipped as part of [DGL-KE](https://github.com/awslabs/dgl-ke/tree/master/examples).
It is a subset of the Freebase KG.
It comprises around 86M entities, 15k relations and 338M triples.

* [x]  **Dataset is pre-stratified into train/test/valid** 

**Publication(s) or website describing benchmark KG dataset**:
https://arxiv.org/abs/2004.08532

**URL for downloading the benchmark KG dataset**:
https://data.dgl.ai/dataset/Freebase.zip (2.5GB compressed)

Inner structure of the folder:
- `test.txt`
- `train.txt`
- `valid.txt`
These contain contiguous identifiers and no labels - use a core triples factory. There are also `entity2id.txt` and `relation2id` that map back to freebase identifiers, but these aren't themselves labels either.
Repository: pykeen/pykeen
Issue: How to use the model to get the embedding of new triples?
Root cause: Documentation Deficiency

How to use the model to get the embedding of new triples?


Please check out the documentation https://pykeen.readthedocs.io (the answer to your question is in the getting started tutorial). If you have follow-up questions, the discussion board (https://github.com/pykeen/pykeen/discussions) is the appropriate place to go.

Thanks for your quick answer. But I think you may misunderstand my question. 
For example, I train my model with triples <a, b, c>, <c, d, e>. After I trained this model and I want to use this model to predict the embedding of new triples like <m, o, n>. How can I get the embedding of <m, o, n> ? Does pykeen support this?

It sounds like you're looking for inductive models. There's also a whole article on that in the tutorial section at https://pykeen.readthedocs.io/en/stable/tutorial/inductive_lp.html. Again, please carefully read through all of the documentation to try and address your problem before opening an issue, or better, you can go to to the discussion board and see if someone in the community wants to help
Repository: pykeen/pykeen
Issue: Bug in tutorial: 'TriplesFactory' object has no attribute 'relation_to_ids'
Root cause: Documentation Deficiency

Hi,

In Mapping Entity and Relation Identifiers to their Names of this tutorial: https://pykeen.readthedocs.io/en/stable/tutorial/first_steps.html 
when I use, triples_factory.relation_to_ids, AttributeError is reported. After I change it to triples_factory.relations_to_ids, problem is solved. I am wondering if there is a typo in this tutorial?

Yes, you are right. Thanks for opening the issue. I pushed a fix under #1067 .

You can already see the fix by changing from "stable" to "latest", i.e., https://pykeen.readthedocs.io/en/latest/tutorial/first_steps.html

ok, thx!
Repository: pykeen/pykeen
Issue: Add 1-1 restrictions to prediction pipeline
Root cause: Unimplemented Feature Gap

### Problem Statement

The prediction pipeline currently allows you to filter out known triples from the training, testing, and validation sets. This allows a user to look at *novel* relations. However, in the entity alignment scenario, we can add a nice additional constraint that any node appearing as either the head or tail in any of the training, testing, and validation sets can be considered as aligned (i.e., we don't have to consider that there are 1-n or n-1 relations) and can therefore be omitted from generating new predictions. This can significantly chip away at the expense of doing a top k prediction.

### Describe the solution you'd like

Add a flag to [`pykeen.models.predict.get_all_prediction_df()`](https://pykeen.readthedocs.io/en/stable/api/pykeen.models.predict.get_all_prediction_df.html#pykeen.models.predict.get_all_prediction_df) and related functions called something like `filter_known_nodes` that applies the pre-filter described above

1. Given training, testing, and validation triples, get all subjects and object appearing in relations that constitute mappings between the two KGs
2. Get the complement of entities from those (i.e., take all entities from training set that aren't appearing in this set)
3. Predict relations constrained that the subject and object are only in these entities and relations are constrained to only be mappings. Could also cut down further knowing that these are symmetric by using `itt.combinations` to generate the possible triples to score. Would benefit from #1090 

### Describe alternatives you've considered

You could run the prediction function then do post-hoc filtering, but this wastes a lot of computation

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)