Repository: pykeen/pykeen
Issue: Bug in filtered negative triples
Root cause: Algorithmic Error

The filtering method of the negative sampler gives often wrongly flagged negative triples (negative triples that are true negative, but get flagged as true positive and the other way)

Steps to reproduce the behavior:

``` python

import torch
import numpy as np

seed = 8

np.random.seed(seed)
torch.manual_seed(seed)

num_rel = 100
num_ent = 100

n_positive = 60
n_negative = 15

positive = torch.zeros((n_positive,3))
positive[:,0] = torch.randint(0, num_ent, (n_positive,))
positive[:,1] = torch.randint(0, num_rel, (n_positive,))
positive[:,2] = torch.randint(0, num_ent, (n_positive,))

negative = torch.randint(0,num_rel,(n_negative,3))
negative[:,0] = torch.randint(0, num_ent, (n_negative,))
negative[:,1] = torch.randint(0, num_rel, (n_negative,))
negative[:,2] = torch.randint(0, num_ent, (n_negative,))

from pykeen.sampling.basic_negative_sampler import BasicNegativeSampler
from pykeen.triples.triples_factory import TriplesFactory

tf = TriplesFactory(
        mapped_triples = positive,
    entity_to_id = {},
    relation_to_id = {}
    )

bns = BasicNegativeSampler(tf, 200)
print(bns.filter_negative_triples(negative))
```
``` bash
(tensor([[65, 86, 20],
        [ 3,  4, 32],
        [28, 78, 10],
        [ 0, 73, 16],
        [86, 22, 64],
        [31, 24, 85],
        [94, 70, 49],
        [89, 28, 90],
        [54, 22, 88],
        [73, 87, 35],
        [66, 73, 58],
        [ 3, 17, 11],
        [49, 11, 45],
        [54, 67, 37]]), tensor([ True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True, False,  True,  True]))
```

``` bash
Positive triples:
tensor([[23., 28., 32.],
        [48., 92., 14.],
        [81., 63., 11.],
        [29., 63., 72.],
        [17., 62., 41.],
        [66.,  8., 66.],
        [16., 59., 43.],
        [15., 45., 15.],
        [52., 98., 87.],
        [11., 90., 63.],
        [88., 23., 46.],
        [49., 87.,  0.],
        [93., 88., 80.],
        [28., 43., 39.],
        [37., 28., 33.],
        [19., 49., 70.],
        [ 2., 60., 27.],
        [63., 82., 15.],
        [ 0., 11., 27.],
        [ 5., 28., 38.],
        [97., 61., 48.],
        [17., 47., 36.],
        [67., 82.,  5.],
        [53., 78., 63.],
        [ 2., 94., 88.],
        [39., 98., 11.],
        [90.,  6., 95.],
        [14., 67., 44.],
        [42., 98., 87.],
        [89., 82., 20.],
        [32., 25., 41.],
        [77., 86., 24.],
        [95., 27., 79.],
        [40., 52., 46.],
        [ 5., 17., 49.],
        [53., 54., 41.],
        [15., 47., 82.],
        [79., 26., 22.],
        [ 2., 85., 10.],
        [17., 82., 75.],
        [90., 90., 87.],
        [38., 82., 32.],
        [38.,  7.,  6.],
        [53., 31., 45.],
        [ 1., 96., 21.],
        [73., 29., 46.],
        [30., 64.,  0.],
        [45., 20., 88.],
        [95., 54., 89.],
        [ 7., 30., 17.],
        [48.,  9., 91.],
        [15., 75.,  9.],
        [67., 56., 33.],
        [ 3., 99., 12.],
        [94., 73.,  8.],
        [95.,  0., 56.],
        [24., 60., 68.],
        [73., 54., 72.],
        [45., 23.,  2.],
        [84., 22., 75.]])
Negative triples:
tensor([[65, 86, 20],
        [ 3,  4, 32],
        [28, 78, 10],
        [ 0, 73, 16],
        [86, 22, 64],
        [31, 24, 85],
        [94, 70, 49],
        [89, 28, 90],
        [54, 22, 88],
        [73, 87, 35],
        [66, 73, 58],
        [ 3, 17, 11],
        [30, 41,  8],
        [49, 11, 45],
        [54, 67, 37]])
```

The third last negative triple [30, 41,  8] gets falsely flagged as true positive. As I tried to give a minimal reproducible example, only 1 of 15 negative triples were falsely filtered. Experiments with bigger samples showed, that this bug can affect a lot more triples.

**Expected behavior**
None of the negative triples should get filtered in this example.

**Environment:**

| Key                  | Value          |
|----------------------|----------------|
| `os.name`            | nt             |
| `platform.system()`  | Windows        |
| `platform.release()` | 10             |
| python               | 3.8.5          |
| pykeen               | 1.1.0-UNHASHED |
| torch                | 1.7.1+cpu      |
| cuda available       | false          |
| cuda                 |                |
| cudnn                |                |

**Additional information**
[Add any other information about the problem here.]


Hi @nomisto ,

first of all thanks for looking into this functionality!

The default dtype of torch is usually `float32`, hence `positive = torch.zeros((n_positive,3))` creates a zero tensor of floating point values. 

Could you please re-run your test with `positive = torch.zeros((n_positive,3), dtype=torch.long)`, and verify that the error persists?



Hi @mberr ,

Thanks for your answer. The bug persists with long as datatype.
I tried looking into the bug, and this is what I discovered: When you reduce the search space you are not checking head-relation combinations etc, but only head, relation and tail individually.

Consider the positive and negative triples from above. 

The first head reduction gives the following triples
```
sub_mapped_triples = tensor([
        [66,  8, 66],
        [49, 87,  0],
        [28, 43, 39],
        [ 0, 11, 27],
        [89, 82, 20],
        [73, 29, 46],
        [30, 64,  0],
        [ 3, 99, 12],
        [94, 73,  8],
        [73, 54, 72]])
```
which contains only positive triples, that have a head in the negative triples.

The second relation reduction gives:
```
sub_mapped_triples = tensor([
        [49, 87,  0],
        [ 0, 11, 27],
        [94, 73,  8]])
```
which contains only triples with relations that are present in the previous sub_mapped_triples and the negative triples, regardless of their actual head/rel combination. (As you can see none of the head/rel combinations: 49-87-?, 0-11-? are actually in the negative triples, however f.e. 49-11-? is).

The last step, which generates the actual filtered boolean mask for the negative edges compares now only the tails with the sub_mapped_triples. As `[30, 41, 8]` has 8 as tail, and 8 is a tail in sub_mapped_triples it is considered as true positive.