Repository: pykeen/pykeen
Issue: Possible issue with model evaluation when using datasets with inverse triples
Root cause: Documentation Deficiency

### Describe the bug

Using the build-in dataset ```Nations```, we show that there might be a problem with the model evaluation process when using datasets with inverse triples, i.e., when ```dataset = Nations(create_inverse_triples=True)```. The evaluation process in that case seems to be different from the process when using a model without inverse triples, i.e., when ```dataset = Nations(create_inverse_triples=False)```. In the following we provide a minimal example to show the difference between the evaluation process  for these two scenarios: 

- scenario 1: no inverse triples are present, i.e., ```dataset = Nations(create_inverse_triples=False)```
- scenario 2: inverse triples are present, i.e., ```dataset = Nations(create_inverse_triples=True)```

and show results for metrics using identical setups apart from the fact that in scenario 1 we are not using inverse triples and in scenario 2 we do use inverse triples.

### How to reproduce

### 1. The setup

Required imports.

```
from pykeen.nn.modules import Interaction
from pykeen.pipeline import pipeline
from pykeen.datasets import Nations
from pykeen.models import make_model_cls
```

Use the standard DistMult interaction function.

```
class DistMultInteraction(Interaction):
    def forward(self, h, r, t):
        return (h * r * t).sum(dim=-1)
```

Take the DistMult model with embedding dimension d=4.

```
model = make_model_cls(
    interaction=DistMultInteraction,
    dimensions={'d': 4},)
```

We create a regular pipeline object with a basic setup and set ```num_epochs=0``` to only evaluate the model on the initialized embeddings such that there is no training happening. We set ```seed=126``` to guarantee that the initial embeddings in both scenarios are identical. In each run we use the same parameters so that we can properly compare the model evaluation for scenario 1 and scenario 2. Since inverse triples are not being used in the evaluation process, the resulting scores for all metrics should be identical in both scenarios because we set  ```num_epochs=0``` and we use the same seed ```random_seed=126```.

```
result = pipeline(
    random_seed=126,
    dataset=dataset,
    model=model,
    loss="BCEWithLogitsLoss",
    training_loop="sLCWA",
    training_kwargs=dict(
        num_epochs=0,
        batch_size= 32,),
    negative_sampler = "basic",
    negative_sampler_kwargs=dict(
        num_negs_per_pos = 1,),
)
```

### 2. Model results for both scenarios 

From this, we get the following results for the ```hits_at_3``` metric.

1. Scenario: There are no inverse triples present, i.e., ```dataset = Nations(create_inverse_triples=False)```

| Side | Rank_type | Metric | Value
-- | -- | -- | -- 
head | optimistic | hits_at_3 | 0.437811
tail | optimistic | hits_at_3 | 0.398010
both | optimistic | hits_at_3 | 0.417910
head | realistic | hits_at_3 | 0.437811
tail | realistic | hits_at_3 | 0.398010
both | realistic | hits_at_3 | 0.417910
head | pessimistic | hits_at_3 | 0.437811
tail | pessimistic | hits_at_3 | 0.398010
both | pessimistic | hits_at_3 | 0.417910


2. Scenario: There are inverse triples present, i.e., ```dataset = Nations(create_inverse_triples=True)```

| Side | Rank_type | Metric | Value
-- | -- | -- | -- 
head | optimistic | hits_at_3 | 0.507463
tail | optimistic | hits_at_3 | 0.417910
both | optimistic | hits_at_3 | 0.462687
head | realistic | hits_at_3 | 0.507463
tail | realistic | hits_at_3 | 0.417910
both | realistic | hits_at_3 | 0.462687
head | pessimistic | hits_at_3 | 0.507463
tail | pessimistic | hits_at_3 | 0.417910
both | pessimistic | hits_at_3 | 0.462687

As can be seen, the results are different despite the fact that the embeddings are identical. This suggests that model evaluation is different when using a dataset with ```inverse_triples=False``` compared to the same dataset with ```inverse_triples=True``` even though inverse triples are not being used in model evaluation (such that the evaluation part of the model in scenario 1 and scenario 2 should be identical and thus give the same scores.

### 3. Details about the ```Nations``` dataset.

The dataset has ```num_entites=14``` and ```num_triples=1992```. The size of the test set is: ``` dataset.testing.mapped_triples.shape = torch.Size([201, 3])```. So there are 201 (mapped) triples in the test set.  The first 10 triples of the test set are:

```
dataset.testing.mapped_triples[:10] = 
tensor([[ 0,  7,  5],
        [ 0, 14, 11],
        [ 0, 18, 12],
        [ 0, 20,  3],
        [ 0, 20, 10],
        [ 0, 21, 12],
        [ 1,  3, 12],
        [ 1,  7,  0],
        [ 1, 10, 11],
        [ 1, 18,  5]])
```

We checked that the test triples are identical for the scenario with and without inverse triples, i.e., scenario 1 (```Nations``` without inverse triples) and scenario 2 (```Nations``` with inverse triples). 

### 4. The evaluation process in detail

As far as I understand, the model evaluation on the test set happens in two steps. 

- Step 1: Corrupt heads: For every triple in the test set, fix the tail and replace the head by any entity that is present in the graph.
- Step 2: Corrupt tails: For every triple in the test set, fix the head and replace the tail by any entity that is present in the graph.

In the following we will go through the 2-step evaluation process for both described scenarios (scenario 1 is ```Nations``` without inverse triples, scenario 2 is ```Nations``` with inverse triples)

We print heads ```h```, tails ```,t``` and relations ```r``` that are passed to the interaction function during the evaluation loop by printing their shapes within the interaction function:

```
class DistMultInteraction(Interaction):
    def forward(self, h, r, t):
        print("Interaction Function: head, tail and relations")
        print("h", h.shape)
        print("t", t.shape)
        print("r", r.shape)
        return (h * r * t).sum(dim=-1)
```

In the following we report the corresponding shapes for both steps (step 1: corrupt heads, step 2: corrupt tails) and the two described scenarios (scenario 1: without inverse relations, scenario 2: with inverse relations).

**Scenario 1: ```Nations``` without inverse relations**

Step 1: head corruption:

Dimensions of head, tail and relation tensors during head corruption evaluation

```
h torch.Size([1, 14, 4])
t torch.Size([201, 1, 4])
r torch.Size([201, 1, 4])
```

Tails correspond to the size of the test set, ```t.shape = torch.Size([201,1,4])``` and head has size 14, ```h.shape=torch.Size([1,14,4])``` which corresponds to the number of entities in ```Nations```. So the model is evaluated by replacing for each of the 201 triples the head while the tail is fixed. This will generate a ```[201,14]``` dimensional scores tensor that contains the score for each of the 201 test triples all 14 possible triples that can be generated by corrupting the head (i.e., fixing the tail and replacing the head with any entity that is in the graph. Since there are 14, the second dimension of the head tensor is 14).

Step 2: Tail corruption:

Dimensions of head, tail and relation tensors during tail corruption evaluation

```
h torch.Size([201, 1, 4])
t torch.Size([1, 14, 4])
r torch.Size([201, 1, 4])
```

Heads correspond to the size of the test set, ```h.shape = torch.Size([201,1,4])``` and tail has size 14, ```t.shape=torch.Size([1,14,4])``` which corresponds to the number of entities in ```Nations```. So the model is evaluated by replacing for each of the 201 triples the tail while the head is fixed. Again, this generates a ```[201,14]``` dimensional scores tensor only this time with fixed heads and corrupted tails for each of the 201 test triples.

### 5. The possible issue with inverse triples

If we repeat this procedure on ```Nations``` with inverse triples we get the following:

**Scenario 2: ```Nations``` with inverse relations**

Step 1: head corruption:

Dimensions of head, tail and relation tensors during head corruption evaluation

```
h torch.Size([201, 1, 4])
t torch.Size([1, 14, 4])
r torch.Size([201, 1, 4])
```

Even though we are in head corruption mode, we are presented with a head tensor that has the size of the test set. By looking at the actually embeddings in the head set it turns out that they correspond to the tails of the test set. So what the interaction function is given as heads is actually the tails of the test set. We checked this by using ```print(h)``` in the interaction function and what we get is the following:

The 10 first head embeddings:

```
print(h[:10]) = 
tensor([[[ 0.4079, -1.2848,  0.5473, -1.4430]],
        [[-0.6264, -1.1625,  0.6276, -0.2435]],
        [[-0.4292,  2.2524,  0.1795,  1.8248]],
        [[-0.0718,  1.0695, -0.0087, -0.6665]],
        [[ 1.2585, -1.7765, -1.7085,  0.8416]],
        [[-0.4292,  2.2524,  0.1795,  1.8248]],
        [[-0.4292,  2.2524,  0.1795,  1.8248]],
        [[-0.2661, -0.5869, -1.0888,  1.5446]],
        [[-0.6264, -1.1625,  0.6276, -0.2435]],
        [[ 0.4079, -1.2848,  0.5473, -1.4430]]])
```

We see that the embedding in position 2 (if we start counting rows at 0) and position 5 and 6 are identical. The same is true for position 0 and 9 and for position 1 and 8. If we have again a look at the first 10 test triples and if we focus on the tail indices

```
dataset.testing.mapped_triples[:10]=
tensor([[ 0,  7,  5],
        [ 0, 14, 11],
        [ 0, 18, 12],
        [ 0, 20,  3],
        [ 0, 20, 10],
        [ 0, 21, 12],
        [ 1,  3, 12],
        [ 1,  7,  0],
        [ 1, 10, 11],
        [ 1, 18,  5]])
```

we see that in position 2,6,7 we have the same entity (with index 12). The same is true for position 0 and 9 (entity 5) and position 1 and 8 (entity 11). So this means that we are presented with 201 head entities that are actually the tails of the test set. This is the first observation.

The second observation is the fact that we are given 201 heads and 14 tails even though we are in the head corruption step. However, instead of corrupting for each of the 201 test triples the head position we are given 201 heads that actually correspond to the tails of the test set and for each of them we create 14 triples where the tail is corrupted. So it seems that the interaction function is not receiving the right tensors for head and tail.

Lets now have a look at step 2, the tail corruption part of the evaluation process.

Step 2: tail corruption:

Dimensions of head, tail and relation tensors during tail corruption evaluation

```
h torch.Size([201, 1, 4])
t torch.Size([1, 14, 4])
r torch.Size([201, 1, 4])
```

As can be seen, the presented dimensions are identical to step 1 (the head corruption mode). We are now presented with the expected dimensions for head and tail. We want to corrupt the the tail by any entity for each of the 201 test triples and we fix the head.

Looking at the first 10 head embeddings gives us:

```
print(h[:10]) = 
tensor([[[-0.2661, -0.5869, -1.0888,  1.5446]],
        [[-0.2661, -0.5869, -1.0888,  1.5446]],
        [[-0.2661, -0.5869, -1.0888,  1.5446]],
        [[-0.2661, -0.5869, -1.0888,  1.5446]],
        [[-0.2661, -0.5869, -1.0888,  1.5446]],
        [[-0.2661, -0.5869, -1.0888,  1.5446]],
        [[ 2.1995,  0.4334,  1.2047,  0.2162]],
        [[ 2.1995,  0.4334,  1.2047,  0.2162]],
        [[ 2.1995,  0.4334,  1.2047,  0.2162]],
        [[ 2.1995,  0.4334,  1.2047,  0.2162]]])
```

We see that the first six embeddings (from row 0 to 5) are identical. From row 6 to 9 we see another embedding. Having again a look at the test set and focussing this time at the head position

```
dataset.testing.mapped_triples[:10]=
tensor([[ 0,  7,  5],
        [ 0, 14, 11],
        [ 0, 18, 12],
        [ 0, 20,  3],
        [ 0, 20, 10],
        [ 0, 21, 12],
        [ 1,  3, 12],
        [ 1,  7,  0],
        [ 1, 10, 11],
        [ 1, 18,  5]])
```

we see that the first 6 heads are identical This is in agreement with the first 10 embeddings of the heads. So in step 2 (the tail corruption mode) we are now given the correct heads and we generate 14 version where the tail is replaced by any entity for each of the 201 test triples. 

**So in summary. For datasets where inverse triples are present, i.e., ```inverse_triples=True```, we find that in both steps of the evaluation process (step 1: head corruption, step2: tail corruption) the dimensions of the head and tail tensor are identical (whereas they are different when using datasets with ```inverse_triples=False```). In fact, in step 1 the heads correspond to the tails and in step 2 the heads are the actual steps.**

### 6. Is this a problem that might skew metrics?

Is it possible that what we have just reported might modify the evaluation process when datasets with inverse triples are being used (as it is for example required when NodePiece is being used).

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Darwin                   |
| Release         | 22.6.0                   |
| Time            | Fri Nov 17 17:58:24 2023 |
| Python          | 3.9.5                    |
| PyKEEN          | 1.10.2-dev               |
| PyKEEN Hash     | c94213ca                 |
| PyKEEN Branch   | master                   |
| PyTorch         | 2.1.0                    |
| CUDA Available? | false                    |
| CUDA Version    | N/A                      |
| cuDNN Version   | N/A                      |

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Hi @martytom ,

if you train a model on a dataset that uses inverse relations that has an effect on the model:

- the model will have twice as many relation representations
- the model will convert any head prediction $(?, r, t)$ to inverse tail prediction, i.e., $(t, r^{-1}, ?)$

Thus, the evaluation results are expected to be different since you are evaluating a *different* model on the same evaluation dataset.

You can verify that the two models are different by, e.g., looking at their string-representation.