Repository: pykeen/pykeen
Issue: How to set evaluation sidedness ?
Root cause: Unimplemented Feature Gap

### What is your question

I have a dataset with only single type of relationship between head entities and tail entities. And what I want to do is simply a link prediction task that given a head, predict the possible tails, which is the [right-side evaluation ](https://pykeen.readthedocs.io/en/stable/tutorial/understanding_evaluation.html)

However, pykeen by default uses both the right- and left-side evaluation, which gives 3 metrics (both, head, tail). How can I set the evaluation to only right-side evaluation (the tail one) ?

I tried to add a parameter "target" for the evaluation_kwargs in the pipeline setting as follows
```python
evaluation_kwargs=dict(
    targets=('tail'),
)
```

But it gives the a TypeError message that:
```python-traceback
TypeError: evaluate() got an unexpected keyword argument 'targets'
```

I quickly checked the implementation, and it seems that there are two `evaluate` functions
- one defined in the Evaluator class, which does not allow `targets`
- one defined as an independent function, which does allow `targets`.

Any idea ?










































































### Environment

| Key             | Value                        |
|-----------------|------------------------------|
| OS              | posix                        |
| Platform        | Linux                        |
| Release         | 4.18.0-305.17.1.el8_4.x86_64 |
| Time            | Mon Apr 11 05:14:50 2022     |
| Python          | 3.9.11                       |
| PyKEEN          | 1.8.0                        |
| PyKEEN Hash     | UNHASHED                     |
| PyKEEN Branch   |                              |
| PyTorch         | 1.11.0                       |
| CUDA Available? | true                         |
| CUDA Version    | 11.3                         |
| cuDNN Version   | 8200                         |

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Hi @lzhang12 ,

with #938 being merged this should be fixed now. For instance, the following snippet works for me
```python
from pykeen.pipeline import pipeline

pipeline(
    dataset="nations",
    model="ntn",
    evaluation_kwargs=dict(
        targets=("tail",),
    ),
)
```

Notice that `targets` needs to be a *collection* of target names, e.g., a tuple or list, rather than a single entry.

Hi @mberr , 
I am trying to evaluate Mrr to relations also. 
 
but still same error for me `evaluate() got an unexpected keyword argument 'targets'`

my code : 
```python
results = evaluator.evaluate(
    model= model,
    mapped_triples=testing.mapped_triples,
    targets=['head', 'relation', 'tail']
    additional_filter_triples=[
        training.mapped_triples,
        validation.mapped_triples,])
```
Note may be important, 
I am using pykeen by installation through pip  and not Github since install through Github gave error message 

`import pykeen`
( No module named 'pykeen' )

Sounds like you need pip install —upgrade

> Sounds like you need pip install —upgrade

Thanks for your reply, Still same problem after upgrading` TypeError: evaluate() got an unexpected keyword argument 'targets'`

<html>
<body>
<!--StartFragment-->

Key | Value
-- | --
OS | posix
Platform | Linux
Release | 5.10.107+
Time | Wed Jun 15 07:59:28 2022
Python | 3.7.12
PyKEEN | 1.8.1
PyKEEN Hash | UNHASHED
PyKEEN Branch |  
PyTorch | 1.11.0
CUDA Available? | true
CUDA Version | 11.0
cuDNN Version | 8005

<!--EndFragment-->
</body>
</html>

Still looks like you’re on an old version of pykeen - 1.8.2 is the newest and you need python 3.8

Updating Pykeen 1.8.3-dev and same error 

`TypeError: __init__() got an unexpected keyword argument 'targets` 

My code: 

```
def_paire_model = pipeline(

    training=training,
    testing=testing,
    validation= validation,
    ## model argument 
    model='PairRE',   
    negative_sampler_kwargs = dict(
    corruption_scheme = ('head', 'relation','tail')),
    model_kwargs =dict(embedding_dim=200),
    evaluator_kwargs=dict(filtered=True , targets=("head","relation","tail"))
    random_seed=1235,
    use_tqdm = False, device='cpu')
```
`pykeen.env()`
Key	Value
OS	nt
Platform	Windows
Release	10
Time	Wed Jul 13 11:45:14 2022
Python	3.8.5
PyKEEN	1.8.3-dev
PyKEEN Hash	UNHASHED
PyKEEN Branch	
PyTorch	1.12.0+cpu
CUDA Available?	false
CUDA Version	N/A
cuDNN Version	N/A


Notice the difference between `evaluation_kwargs` and `evaluator_kwargs`. The latter is used to instantiate the evaluator, while the former is passed to the `evaluate` method.

Thanks, I did not notice that. 

I am trying and still same error 

```python-traceback
while True:
   1422     try:
-> 1423         metric_results: MetricResults = evaluator.evaluate(
   1424             model=model,
   1425             mapped_triples=mapped_triples,
   1426             **evaluation_kwargs,
   1427         )
   1428     except (MemoryError, RuntimeError) as e:
   1429         # If the evaluation still fail using the CPU, the error is raised
   1430         if model.device.type != "cuda" or not evaluation_fallback:

TypeError: evaluate() got an unexpected keyword argument 'targets'
```
code: 
```python
Pairre_model =  pipeline(
        training=training,
        testing=testing,
        validation = validation,
        model='Pairre',
        model_kwargs =dict(embedding_dim=200, p= 1),
        training_loop='sLCWA',
        optimizer='adam',
        negative_sampler='basic',  
        negative_sampler_kwargs=dict(
        corruption_scheme = ('head', 'relation', 'tail') ,
        filtered=True,
        filterer='python-set' , 
        num_negs_per_pos = 30), 
        
        #stopp training
        stopper='early',
        stopper_kwargs=dict(frequency=5, patience=2, relative_delta=0.002),
        # evaluator
         evaluation_kwargs=dict(
        targets=("head","relation","tail")),
 
        filter_validation_when_testing = True ,
            
        training_kwargs=dict(num_epochs=100, batch_size= 256),
        random_seed=1235,evaluation_fallback = True,
        use_tqdm = False, device='gpu') 
```
env: 
<html>
<body>
<!--StartFragment-->

Key | Value
-- | --
OS | posix
Platform | Linux
Release | 5.4.0-89-generic
Time | Wed Jul 13 11:14:39 2022
Python | 3.9.13
PyKEEN | 1.8.2
PyKEEN Hash | UNHASHED
PyKEEN Branch |  
PyTorch | 1.12.0+cu116
CUDA Available? | true
CUDA Version | 11.6
cuDNN Version | 8302

<!--EndFragment-->
</body>
</html>

I reduced your example to a minimum working example, which is executable without your custom data:
```python
from pykeen.pipeline import pipeline

pipeline(
    dataset="nations",
    model="PairRE",
    negative_sampler_kwargs=dict(corruption_scheme=("head", "relation", "tail")),
    evaluation_kwargs=dict(targets=("head", "relation", "tail")),
    training_kwargs=dict(num_epochs=1, batch_size=32),
    random_seed=1235,
)
```

I get the following error on the current `master` (63c40ba188c332d47c0beca02233165ea1513c4d)
```python-traceback
Traceback (most recent call last):
  File ".../scratch/i867.py", line 3, in <module>
    Pairre_model = pipeline(
  File ".../src/pykeen/pipeline/api.py", line 1368, in pipeline
    metric_results: MetricResults = _safe_evaluate(
  File ".../src/pykeen/pipeline/api.py", line 1432, in _safe_evaluate
    raise e
  File ".../src/pykeen/pipeline/api.py", line 1424, in _safe_evaluate
    metric_results: MetricResults = evaluator.evaluate(
  File ".../src/pykeen/evaluation/evaluator.py", line 208, in evaluate
    rv = evaluate(
  File ".../src/pykeen/evaluation/evaluator.py", line 600, in evaluate
    raise NotImplementedError("cf. https://github.com/pykeen/pykeen/pull/728")
NotImplementedError: cf. https://github.com/pykeen/pykeen/pull/728
```

So, it seems that while we support training with relation corruption, evaluation is not yet implemented. The error message refers to #728 .

iirc, it is mostly blocked by the https://github.com/pykeen/pykeen/pull/752, which needs clarification about how to handle inverse relations when evaluating or training.