Repository: pykeen/pykeen
Issue: Splitting a triples factory could not find a coverage of all entities and relations
Root cause: Validation Gap

### Discussed in https://github.com/pykeen/pykeen/discussions/1326

<div type='discussions-op-text'>

<sup>Originally posted by **dinani65** April 12, 2023</sup>
I am getting the error "ValueError: Could not find a coverage of all entities and relation with only 160 triples." once the tf is split!

```python
data = run_query("""
    MATCH (s)-[r]->(t)
    RETURN toString(id(s)) as source, toString(id(t)) AS target, type(r) as type
""")

tf = TriplesFactory.from_labeled_triples(
  data[["source", "type", "target"]].values
)

training, testing, validation = tf.split([.8, .1, .1])`
```

Additionally, tf is **TriplesFactory(num_entities=205, num_relations=6, create_inverse_triples=False, num_triples=200)**
can you please tell me how to fix it?
thanks in advance!</div>

I'm hitting this same issue that @cthoyt was helping with in the discussion above and figured I'd open a new thread here.

I'm unable to share the specific work data I'm using, but for this small set there are 120 entities that are connected across 4 different kinds of relationships. All entity ids are string values that were passed into the `TriplesFactory.from_labeled_triples` function as so:
```python
triple_factory = TriplesFactory.from_labeled_triples(
    hrt_df.values,
    create_inverse_triples=True,
    entity_to_id=None,
    relation_to_id=None,
    compact_id=True,
    filter_out_candidate_inverse_relations=True,
    metadata=None,
)
```
to create a triple factory looking like `TriplesFactory(num_entities=120, num_relations=4, create_inverse_triples=True, num_triples=110)`

Trying to split these data with `.split()` is what throws the ValueError below:
```python
transductive_training, inductive_inference, inductive_testing, inductive_validation = (
    triple_factory.split(ratios=[0.7, 0.1, 0.1, 0.1], random_state=42)
)
```
ValueError: Could not find a coverage of all entities and relation with only 77 triples.

Not sure if I'm able to use `.split()` for creating an _inductive_ dataset from my own data (docs didn't say anything how creating your own inductive dataset), but I'll throw out there that I get the same error even if I try it with just 3 ratios at [0.8, 0.1, 0.1].

Any advice? Many thanks in advance.

Update – this goes away when I use it on a larger graph:
```
TriplesFactory(num_entities=11774, num_relations=14, create_inverse_triples=True, num_triples=77190)

TriplesFactory(num_entities=11774, num_relations=14, create_inverse_triples=True, num_triples=54033)

TriplesFactory(num_entities=11774, num_relations=7, create_inverse_triples=False, num_triples=7718)

TriplesFactory(num_entities=11774, num_relations=7, create_inverse_triples=False, num_triples=7720)

TriplesFactory(num_entities=11774, num_relations=7, create_inverse_triples=False, num_triples=7719)
```

However, I think the issue is that what `coverage` means here and what settings are available for `SPLIT_METHODS` are not readily found, which prevents users from triaging this error.

Hi @AlexMRuch ,

I think the description over at https://github.com/pykeen/pykeen/issues/641#issuecomment-978078528 might still be up-to-date. I'll take a look on to best include that into the documention.

*tl;dr*: I think you are observing the issue because there is no selection of 80% of all triples which contains all of the entities/relations. That is likely a symptom from having a very sparse graph.

Ah, thanks so much, @mberr!

I was expecting that was the case. I've encountered that issue with https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html as well, and the fact that it resolved with the larger graph sample improved my confidence in that hypothesis. FWIW, I think the sklearn error that is thrown in equivalent situations is more informative than what PyKEEN reported as it explicitly stated something along the lines of the test set including entities that aren't in the training set, etc.

I wasn't sure if coverage would be an issue or not with the splitter given that I am developing an inductive LP model. Since the pipeline only requires a transductive training triple factory and then the inductive inference, validation, and testing sets, in my mind I didn't think coverage would be an issue. Please correct me if my assumption is misguided or if I am correct and the error that `triple_factory.split()` threw should only be thrown in _transductive_ cases (in which case maybe raising a warning rather than an error would be more appropriate?). Based on the other issue you shared, seems like the "only transductive" case is true at the moment.

Also, thanks for your work in https://github.com/pykeen/pykeen/pull/1411/files <- very informative!

> Please correct me if my assumption is misguided or if I am correct and the error that triple_factory.split() threw should only be thrown in transductive cases (in which case maybe raising a warning rather than an error would be more appropriate?)

The current implementation for coverage-based splits throws an error if the minimum training set size to ensure coverage exceeds the desired training ratio. I could see changing this to a warning and instead returning a result with the training proportion increased, which would behave more like the randomized splitting algorithms.

> Based on the other issue you shared, seems like the "only transductive" case is true at the moment.

Yes, exactly. I started adding a few ways for inductive splits in https://github.com/pykeen/pykeen/pull/1416, but the range of possible inductive split methods is larger than for transductive splits so it might not cover your desired use case.

Thanks so much for these additional details! Very helpful for growing my understanding of how PyKEEN works with inductive modeling!