Repository: pykeen/pykeen
Issue: Unexpected behavior in generation of transductive splits motivates inductive splitting
Root cause: Documentation Deficiency

### Describe the bug

Hi, I am experiencing a strange output when performing link prediction after embedding a model with my own dataset.

First of all, I can't make my train-test-split working. I create a Triples Factory with
```python
data = pd.read_csv(...)
tf = TriplesFactory.from_labeled_triples( data.values )
```

and then  I split it with
```python
train, test = tf.split([0.7, 0.3], random_state=2)
```

My dataset represents a graph with 5 different relation types. I would like to perform link prediction on only one of these, therefore I must be sure that these relationship are equally split between train and test set.  The problem is that, for any random state I set, all the "ORIG" relationships (the ones I'm targeting) appear in the training set and none in the test set! I also tried to change the split fractions, but I always get this behaviour. Maybe I am missing something on the theory of train-test splits on graphs?

To overcome this, I split the DataFrame before creating the triples factories, and then I create two different triples factories `tf_test` and `tf_train`. During the pipeline I passed them to the model with

```python
result = pipeline.pipeline( training=tf_train, testing=tf_test, ...)
```

and the when I try to call

```python
scores = get_head_prediction_df(
    model=result.model, 
    relation_label="ORIG",
    tail_label='000',
    triples_factory=result.training, 
    testing=tf_test.mapped_triples,
    remove_known=False, 
    add_novelties=True,
)
```

I get a dataframe where the `in_testing` column is false in every single row. 
I don't understand if there is a bug in the `in_testing` check, if I am passing the wrong input in the "testing" argument, if there is some problem in the train-test split or if I simply got everything really wrong, and none of the model predictions belong to the test set (weird though).




### How to reproduce

I'm sorry but I cannot provide my full triples dataset. I just wanted to ask if anyone has encountered a similar behaviour in the "in_testing" boolean column. 

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | nt                       |
| Platform        | Windows                  |
| Release         | 10                       |
| Time            | Thu Mar 31 11:49:19 2022 |
| Python          | 3.9.7                    |
| PyKEEN          | 1.7.0                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.10.2+cpu               |
| CUDA Available? | false                    |
| CUDA Version    | N/A                      |
| cuDNN Version   | N/A                      |

### Additional information

_No response_

> My dataset represents a graph with 5 different relation types. I would like to perform link prediction on only one of these, therefore I must be sure that these relationship are equally split between train and test set. The problem is that, for any random state I set, all the "ORIG" relationships (the ones I'm targeting) appear in the training set and none in the test set! I also tried to change the split fractions, but I always get this behaviour. Maybe I am missing something on the theory of train-test splits on graphs?

The split procedures provided in PyKEEN make sure that all entities and relations occur in at least one training triple, which is a requirement for applying transductive KGEM. Maybe that is a problem in your case?

>To overcome this, I split the DataFrame before creating the triples factories, and then I create two different triples factories tf_test and tf_train.

When creating these factories, did you make sure that they share the *same* entity/relation-to-id mapping?

> The split procedures provided in PyKEEN make sure that all entities and relations occur in at least one training triple, which is a requirement for applying transductive KGEM. Maybe that is a problem in your case?
Well, I have approximately 350 instances of this particular relation and they all share the same two entities. I understand that at least 1 of them should be in the training set, but _all_ of them? 

Here's what I get to give you an example:
```
Total number of interesting rels:  381
With sklearn split: train 282, test 99
With pykeen split: train 381, test 0
```


> When creating these factories, did you make sure that they share the same entity/relation-to-id mapping?

I didn't pass anything to the entity2id mapping parameter, but the two dataframes are obtained by splitting in two the same original df,  so they look exactly the same:

```
data_test = data[::3]
data_train = pd.merge(data, data_test, indicator=True, how='outer'
```

Tbh I didn't properly understand how the whole entity/relation to id mapping works, the docs look a bit scarce to me for that matter.
Sorry, I'll try to be more clear:
My dataset consists of a list of triples such as:

| source   |    type      |         target  |
|---|---|---|
| A123     |  suffers_from      |   fever |
|A852    |   has_patient        |   37 |
| fever |        cured_by         |  pp42 |         


How can I specify the entities in `from_labeled_triples`? I mean, I would like to specify that A123 and A852 are instances of the `Doctor` entity, fever is `Illness`,  37 is `Patient` and so on...

Edit: I think there is a misunderstanding on what we identify as "entity". For me entity means the node type, while for you it represents the single node, so my dataset would have 20k+ entities... Now I see why all my "ORIG" relations all go into the test set: because they all start from different Patient nodes (so, for you, different entities...). 
My graph is constructed in a way such that every "entity" (intended as you do) appears only one or two times, but they can be clustered into 5 main node classes... Is there a way to achieve this in PyKeen?

It somewhat depends on the type of pattern you would like to learn, and whether you have additional features available to distinguish different patients.

The simplest solution is for the case where you are only interested in patterns on the class-level:
You can achieve this by mapping the entities (or only a subset you cannot / do not want to distinguish, e.g. all patients) to the classes, i.e.,

| source | type  | target  |
| -- | -- | -- |
|A123 | suffers_from |         fever |
|A852| has_patient|           37|
|fever| cured_by|           pp42         |

becomes

| source | type  | target  |
| -- | -- | -- |
|Doctor | suffers_from |         fever |
|Doctor| has_patient|           Patient|
|fever| cured_by|           pp42         |

In this case, you can formulate your task as a transductive learning task, i.e., all entities / relations are seen during training, and use the split method.


If you want to keep the instance-level annotations, differ between instances from a class, and you only a few a triples per instance, you may want to consider an inductive setting instead, where not all entities need to be seen during training. There is a documentation page about this here: https://pykeen.readthedocs.io/en/stable/tutorial/inductive_lp.html

We do not yet(?) have a split method for this case, although this might change in the (near) future.

The objective of my project is to understand which patients are connected to a certain disease... so unfortunately I need to keep in the model both the fact of the node being a patient, and the patient ID. Thanks though! I will now have a look into inductive learning.

Since this isn't really a bug but more of an issue with making transductive splits, I think the right thing to do forwards is to rename this issue for future reference and convert it to a discussion. As far as I can tell, this isn't a bug in PyKEEN but rather a mismatch between data and methods