Repository: pykeen/pykeen
Issue: Memory Efficient Data Processing/Training
Root cause: Resource Mishandling

(Similar to https://github.com/pykeen/pykeen/issues/338)

I've been trying to use the ConceptNet dataset, but the python process either gets killed or the jupyter notebook dies due to low memory. As the ConceptNet dataset (and any other large knowledge base) takes up a lot of memory, perhaps it would be beneficial to have an option to not load the whole dataset into memory at once. 

For the use case of building a triples factory from your own data, it would be trivial to build the mappings by iterating through the dataset and not loading it all at once. But, it would get more complicated for building instances for training or splitting the data, or any operation that access the 'mapped_triples' attribute.

**Describe the solution you'd like**
A TriplesFactory class where the mapped_triples attribute doesn't exist or is an iterable, so you don't have to keep it in memory, you just access it from a file when needed. Instance classes that inherit from torch.utils.data.IterableDataset (instead of data.Dataset) to not have to load it all into memory. TrainingLoop classes that work with IterableDataset Instances.

**Describe alternatives you've considered**
I will probably end up subclassing TriplesFactory, Instance, and TrainingLoop as described. The alternatives are: get a large machine with more RAM or use a smaller dataset.

I hope this issue makes sense, I have probably made some errors in my assumptions regarding the solution.
Thank you

Hi @joerenner .

the main reason for the memory issue for this dataset is currently, that we do store the labels in memory by default. We already worked on separating the storage of metadata (including labelling) into a separate component and to have a core component of index-based triples, the [CoreTriplesFactory](https://pykeen.readthedocs.io/en/stable/reference/triples.html?highlight=triples#pykeen.triples.CoreTriplesFactory). For ConceptNet, [we have approx. 34M triples](https://github.com/pykeen/pykeen/blob/4e0bd827f0f82ac70504c2711acd8718cd99fec7/src/pykeen/datasets/conceptnet.py#L33-L36), meaning that we can store all index-based triples in 34M * 3 * 8 byte < 1GiB of memory.

Until we solve the problem, as a kind of hotfix, I have loaded the dataset on a machine with larger amount of main memory, and dumped the index-based triples, as well as the entity and relation to ID mappings into separate files. Here is the corresponding code

```python
import pickle
import torch
import pandas
from pykeen.datasets import get_dataset
ds = get_dataset(dataset="conceptnet")
torch.save(dict(
    num_entities=ds.num_entities,
    num_relations=ds.num_relations,
    mapped_triples=dict(
        train=ds.training.mapped_triples,
        testing=ds.testing.mapped_triples,
        validation=ds.validation.mapped_triples,
    ),
), "/tmp/conceptnet.triples.pt", pickle_protocol=pickle.HIGHEST_PROTOCOL)
pandas.DataFrame(data=list(ds.entity_to_id.items()), columns=["entity_label", "entity_id"]).to_csv("/tmp/conceptnet.entities.tsv.xz", sep="\t", index=False)
pandas.DataFrame(data=list(ds.relation_to_id.items()), columns=["relation_label", "relation_id"]).to_csv("/tmp/conceptnet.relations.tsv.xz", sep="\t", index=False)
```


I make the files available [here](https://www.dbs.ifi.lmu.de/~berrendorf/share/conceptnet/), although I cannot guarantee they will remain accessible there forever :wink:. From these, you should be able to load the index-based triples on your machine, and create `CoreTriplesFactory`s out of them. 

```python
from pykeen.triples import CoreTriplesFactory
data = torch.load("conceptnet.triples.pt")
factories = {
    key: CoreTriplesFactory.create(
        mapped_triples=mapped_triples, 
        num_entities=data["num_entities"], 
        num_relations=data["num_relations"],
    )
    for key, mapped_triples in data["mapped_triples"].items()
}
```

They can be used in most places where you would normally provide the "full" TriplesFactory, including the training pipeline and evaluation.
```python
from pykeen.pipeline import pipeline
pipeline(
    training=factories["train"], 
    testing=factories["testing"], 
    model="transe", 
    model_kwargs=dict(embedding_dim=2),
)
```

Since you cannot fit all labels into memory, you would need to lookup entity/relation Ids on your own from the (compressed) TSV files.

Hi @mberr,

Thank you for the detailed reply! I appreciate you making the files available. I'll use this solution.