Repository: pykeen/pykeen
Issue: Joint training of KGE and GloVE
Root cause: Unimplemented Feature Gap

I want to jointly train TuckER on the entities as well as the GloVe from the co-occurrence matrix of raw text. Basically, for every iteration, I want to do alternate updates after tying their model weights together.

Can this be possible by extending PyKeen?

Yes, it should be possible. In our library you will only find the link prediction part, so you would need to add the Word2Vec parts yourself. You can start from the following example code, adjust some settings to your use case, and implement what is written as TODO. The key insight is to share the entity embedding module between both models.

```python
from pykeen.datasets import Nations
from pykeen.models import ERModel
from pykeen.nn import EmbeddingSpecification
from pykeen.nn.modules import TuckerInteraction

embedding_dim = 300
relation_dim = 300  # can also be different for TuckER

# Get a training dataset
dataset = Nations()
training_triples_factory = dataset.training

# create entity representation outside, to share them with word2vec
entity_representations = EmbeddingSpecification(
    embedding_dim=embedding_dim,
    # there are more parameters you can set, e.g., for initialization
).make(num_embeddings=training_triples_factory.num_entities)

# create link prediction model:
model = ERModel(
    triples_factory=training_triples_factory,
    interaction=TuckerInteraction,
    interaction_kwargs=dict(
        embedding_dim=embedding_dim,
        relation_dim=relation_dim,
    ),
    entity_representations=entity_representations,
    relation_representations=EmbeddingSpecification(embedding_dim=relation_dim),
)

# TODO: Create word2vec model. It is very important that the IDs match with those from the training TriplesFactory
# TODO: create your own training loop, which alternates Word2Vec & TuckER updates
```


Thanks, this is helpful. 

Can I use PyKeen to create the training pipeline with alternate updates or should I use PyTorch for that? Do you have a sample training code that does this optimization? Also, how do we ensure that the IDs for the GloVe match with those of the original triples used to train the TuckER? Should I convert all the raw words in the corpus to their Freebase ID equivalents, for instance?

> Also, how do we ensure that the IDs for the GloVe match with those of the original triples used to train the TuckER? Should I convert all the raw words in the corpus to their Freebase ID equivalents, for instance?

Within pykeen, this is ensured when creating the triples factories (for a dataset), where the label-based triples are converted to ID-based triples. The mapping is stored in dictionaries `entity_to_id` / `relation_to_id`. If you start from raw text, you likely need a more involved pipeline where you identify entities from the text, and align them with the KG you want to use. I am no expert in this field, but maybe you can find something helpful in [ntlk](https://github.com/nltk/nltk) or [spacy](https://spacy.io/).

> Can I use PyKeen to create the training pipeline with alternate updates or should I use PyTorch for that? Do you have a sample training code that does this optimization?

You can look at our training pipelines: The basic structure is shown [here](https://github.com/pykeen/pykeen/blob/faf9a3d365432da2840d6d40efa6a1ca11d49ad5/src/pykeen/training/training_loop.py#L97). We have two specific variants implemented, which differ in their training assumptions: [LCWA](https://github.com/pykeen/pykeen/blob/faf9a3d365432da2840d6d40efa6a1ca11d49ad5/src/pykeen/training/lcwa.py#L22) and [sLCWA](https://github.com/pykeen/pykeen/blob/faf9a3d365432da2840d6d40efa6a1ca11d49ad5/src/pykeen/training/slcwa.py#L28).

As of https://github.com/pykeen/pykeen/pull/964, it's also much more straightforward to include a pre-trained embedding.