Repository: pykeen/pykeen
Issue: Question about the use of `create_inverse_triples`

Hi, 

I have a question regarding the `create_inverse_triple` argument when loading a TripleFactory, 
such as:
```python
train = TriplesFactory.from_path(data_path / 'train.txt',  create_inverse_triples=True)
valid = TriplesFactory.from_path(data_path / 'valid.txt',  create_inverse_triples=True)
test = TriplesFactory.from_path(data_path / 'test.txt', ,  create_inverse_triples=True)
```

I understood that this should be about avoiding leakage through inverse relations $r_1, r_2$ if $(h, r_1, t)$ is in train and $(h, r_2, t)$ is in test. 
It is described [in this blogpost](https://cthoyt.com/2020/12/11/inverse-triple-scoring.html) that some models are more sensitive to the problem than others. 

I have not yet fully understood what happens when I use `create_inverse_triples=True` in comparison to setting it to `False`. Looking at the TripleFactories I only recognize that the value of  `num_relations` doubles. 

**My questions now are:** 
* What happens during training and testing if I set `create_inverse_triples = True` when training and testing on the loaded data with a pykeen pipeline? 
* does it depend on the number of inverse relations in my dataset how I should use `create_inverse_triples`? 
* Is there a function in pykeen that can automatically examine my data in advance to see if some relations are inverse ? 

It would be amazing to have some additional documentation regarding these questions. Thanks a lot in advance :-) 

Hi @LuisaWerner ,

I will try to give a broader overview before answering your specific questions below.

## Overview

First, `create_inverse_triple=True` is a flag of a triples factory (or more precisely, one of its base classes, [`KGInfo`](https://github.com/pykeen/pykeen/blob/c94213cafc5f9cebad5d23ff0a89c869ae1d5ee0/src/pykeen/triples/triples_factory.py#L274)), and means that we will add inverse triples when creating *training* instances: for each training triples $(h, r, t)$, we will add a triple $(t, r^{-1}, t)$, where $r^{-1}$ is a new relation symbol.

Since this creates new relation symbols, we must also learn representations for them, i.e., the model itself will train twice as many relation representations. Thus, the `num_relations` attribute of a triples factory will be twice the number of original relations, cf. [here](https://github.com/pykeen/pykeen/blob/c94213cafc5f9cebad5d23ff0a89c869ae1d5ee0/src/pykeen/triples/triples_factory.py#L306-L309)

For evaluation, we do not need to add inverse relations. Rather, a _model which has been trained with inverse relations_, will (by default) use them to make head predictions, i.e., transform a head prediction $(?, r, t)$ to a tail prediction with the inverse relations $(t, r^{-1}, ?)$. The relevant code for this is [here](https://github.com/pykeen/pykeen/blob/c94213cafc5f9cebad5d23ff0a89c869ae1d5ee0/src/pykeen/models/base.py#L344-L377).


Besides these "artificial" inverse relations, datasets may also contain "natural" inverse relations, the presence of which may lead to unexpected results, cf., e.g., [Dettmers et al., 2017](https://arxiv.org/abs/1707.01476). Examples of such "contaminated" datasets are `FB15k` or `WN18`, and thus "cleaned" dataset versions have been created, called `FB15k-237` / `WN18-RR`. PyKEEN contains code for this deduplication process in the [`pykeen.triples.leakage`](https://github.com/pykeen/pykeen/blob/c94213cafc5f9cebad5d23ff0a89c869ae1d5ee0/src/pykeen/triples/leakage.py) module; this file also shows sample code to reproduce `FB15k-237` from `FB15k`. The [`Sealant`](https://github.com/pykeen/pykeen/blob/c94213cafc5f9cebad5d23ff0a89c869ae1d5ee0/src/pykeen/triples/leakage.py#L182) object used in this process can also be used to further investigate intermediate results about duplicate / inverse relation pair candidates. 

## Answers
> What happens during training and testing if I set create_inverse_triples = True when training and testing on the loaded data with a pykeen pipeline?

When it is set for the *training* factory, this will 
- create additional "artificial" relations and train representations for them (i.e., the model gets more parameters)
- the training loop will create additional training instances (~triples), cf, e.g., [here](https://github.com/pykeen/pykeen/blob/c94213cafc5f9cebad5d23ff0a89c869ae1d5ee0/src/pykeen/triples/triples_factory.py#L492-L499); this means that in each epoch, we essentially see the triples twice - once in their natural form $(h, r, t)$, and once as the inverse $(t, r^{-1}, h)$.

For an evaluation factory, setting the flag will have no effect; I am not sure if we would see an error raised if the flag is set; if not, it would be better to change this. 🙂 

> does it depend on the number of inverse relations in my dataset how I should use create_inverse_triples?

No; the `create_inverse_triples` flag does only controls the creation of *additional* (artificial) inverse relations. It does not affect existing "natural" ones.

> Is there a function in pykeen that can automatically examine my data in advance to see if some relations are inverse ?

Yes, the methods in the [`pykeen.triples.leakage`](https://github.com/pykeen/pykeen/blob/c94213cafc5f9cebad5d23ff0a89c869ae1d5ee0/src/pykeen/triples/leakage.py) module, in particular, [`unleak`](https://github.com/pykeen/pykeen/blob/c94213cafc5f9cebad5d23ff0a89c869ae1d5ee0/src/pykeen/triples/leakage.py#L239) for a high-level API for removing inverse and duplicate relations from a dataset, and [`Sealant`](https://github.com/pykeen/pykeen/blob/c94213cafc5f9cebad5d23ff0a89c869ae1d5ee0/src/pykeen/triples/leakage.py#L182C14-L182C15) for a more fine-grained control / introspection of candidate relation pairs.

Thanks a lot for your fast and detailed answer. This helps me a lot understanding the creation of inverse triples. 

_For an evaluation factory, setting the flag will have no effect; I am not sure if we would see an error raised if the flag is set; if not, it would be better to change this. 🙂_
Regarding your note, I tested this and importing the test triples with `create_inverse_tripes=True` didn't throw an error or warning.
The code for reproducing this looks as follows: 

```python
train = TriplesFactory.from_path(data_path / 'train.txt', , create_inverse_triples=True)
valid = TriplesFactory.from_path(data_path / 'valid.txt',  create_inverse_triples=True)
test = TriplesFactory.from_path(data_path / 'test.txt',, create_inverse_triples=True)

result = pipeline(
        training=train,
        validation=valid,
        testing=test,
        model="ConvE",  
        model_kwargs=dict(predict_with_sigmoid=True),
        loss=BCEWithLogitsLoss,  
        training_loop='sLCWA', 
        negative_sampler='bernoulli',
        epochs=10,
        stopper='early'
    )
    result.save_to_directory(model_path)
```