Repository: pykeen/pykeen
Issue: Why do embeddings look different for R-GCN?
GitHub labels: question

☠️ R-GCN ☠️

I am looking at using the embeddings created by the RGCN model. 

I assumed that it is the enriched embeddings. I was just confused because it is slightly different from the other models where I use entity embeddings. I just want to double check. I can test this myself so if you don't feel like answering this issue and just want to close it feel free.

**RGCN**
```python
{'training': False,
 '_parameters': OrderedDict([('base_embeddings',
               Parameter containing:
               tensor([[-8.7584e-03, -2.0186e-02,  1.7592e-02,  ..., -1.9370e-02,
                        -4.8222e-03, -3.6431e-03],
                       [ 7.2705e-04, -6.6818e-03, -1.0354e-02,  ..., -2.1720e-02,
                        -1.7344e-02, -3.5045e-03],
                       [ 2.1779e-05, -1.3025e-02,  6.5314e-03,  ..., -1.5803e-04,
                        -2.3960e-03,  8.4298e-03],
                       ...,
                       [-9.0894e-03,  7.5927e-04,  1.7926e-03,  ...,  5.7311e-03,
                         1.9608e-02,  3.4487e-03],
                       [ 4.5891e-03, -1.4208e-02,  9.3362e-03,  ...,  5.2397e-03,
                        -1.8979e-02,  9.1112e-03],
                       [ 5.4939e-03, -4.2473e-03, -3.6072e-03,  ...,  1.5401e-02,
                         1.8973e-02,  8.7341e-03]], device='cuda:0', requires_grad=True))]),
 '_buffers': OrderedDict([('sources',
               tensor([    0,     0,     0,  ..., 12486, 12486, 12486], device='cuda:0')),
              ('targets',
               tensor([  279,   376,   673,  ..., 11257, 11403, 12132], device='cuda:0')),
              ('edge_types',
               tensor([2, 2, 2,  ..., 2, 2, 2], device='cuda:0'))]),
 '_non_persistent_buffers_set': set(),
 '_backward_hooks': OrderedDict(),
 '_forward_hooks': OrderedDict(),
 '_forward_pre_hooks': OrderedDict(),
 '_state_dict_hooks': OrderedDict(),
 '_load_state_dict_pre_hooks': OrderedDict(),
 '_modules': OrderedDict([('loss',
               SoftplusLoss(
                 (softplus): Softplus(beta=1, threshold=20)
               )),
              ('regularizer', NoRegularizer()),
              ('base_model',
               DistMult(
                 (loss): SoftplusLoss(
                   (softplus): Softplus(beta=1, threshold=20)
                 )
                 (regularizer): LpRegularizer()
                 (entity_embeddings): Embedding(12487, 100)
                 (relation_embeddings): Embedding(4, 100)
               )),
              ('activations',
               ModuleList(
                 (0): ReLU()
                 (1): ReLU()
               )),
              ('bases',
               ParameterList(
                   (0): Parameter containing: [torch.cuda.FloatTensor of size 4x100x100 (GPU 0)]
                   (1): Parameter containing: [torch.cuda.FloatTensor of size 4x100x100 (GPU 0)]
               )),
              ('att',
               ParameterList(
                   (0): Parameter containing: [torch.cuda.FloatTensor of size 5x4 (GPU 0)]
                   (1): Parameter containing: [torch.cuda.FloatTensor of size 5x4 (GPU 0)]
               )),
              ('biases',
               ParameterList(
                   (0): Parameter containing: [torch.cuda.FloatTensor of size 100 (GPU 0)]
                   (1): Parameter containing: [torch.cuda.FloatTensor of size 100 (GPU 0)]
               ))]),
 'device': device(type='cuda'),
 'is_mr_loss': False,
 'is_nssa_loss': False,
 'triples_factory': TriplesFactory(path="<None>"),
 'predict_with_sigmoid': False,
 'automatic_memory_optimization': True,
 'embedding_dim': 100,
 'decomposition': 'basis',
 'num_bases': 4,
 'buffer_messages': True,
 'enriched_embeddings': tensor([[1.9732e-04, 0.0000e+00, 1.1546e-04,  ..., 0.0000e+00, 0.0000e+00,
          4.3017e-05],
         [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00, 0.0000e+00,
          3.2970e-04],
         [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00, 0.0000e+00,
          1.8651e-04],
         ...,
         [2.5343e-04, 0.0000e+00, 4.7290e-05,  ..., 0.0000e+00, 0.0000e+00,
          2.9352e-04],
         [6.5879e-05, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00, 0.0000e+00,
          1.3707e-04],
         [1.8369e-04, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00, 0.0000e+00,
          1.4848e-04]], device='cuda:0'),
 'edge_weighting': <function pykeen.models.unimodal.rgcn.inverse_indegree_edge_weights(source: torch.LongTensor, target: torch.LongTensor) -> torch.FloatTensor>,
 'edge_dropout': 0.4,
 'self_loop_dropout': 0.2,
 'use_batch_norm': False,
 'activation_cls': torch.nn.modules.activation.ReLU,
 'activation_kwargs': None,
 'use_bias': True,
 'num_layers': 2,
 'sparse_messages_slcwa': True,
 'batch_norms': None}
```

**TransE**
```python
{'training': False,
 '_parameters': OrderedDict(),
 '_buffers': OrderedDict(),
 '_non_persistent_buffers_set': set(),
 '_backward_hooks': OrderedDict(),
 '_forward_hooks': OrderedDict(),
 '_forward_pre_hooks': OrderedDict(),
 '_state_dict_hooks': OrderedDict(),
 '_load_state_dict_pre_hooks': OrderedDict(),
 '_modules': OrderedDict([('loss',
               SoftplusLoss(
                 (softplus): Softplus(beta=1, threshold=20)
               )),
              ('regularizer', NoRegularizer()),
              ('entity_embeddings', Embedding(12487, 100)),
              ('relation_embeddings', Embedding(4, 100))]),
 'device': device(type='cuda'),
 'is_mr_loss': False,
 'is_nssa_loss': False,
 'triples_factory': TriplesFactory(path="<None>"),
 'predict_with_sigmoid': False,
 'automatic_memory_optimization': True,
 'embedding_dim': 100,
 'relation_dim': 100,
 'scoring_fct_norm': 1}
```


Hi @ChristopherMarais ,

as an initial remark: We are not very happy with the modularization of the RGCN model yet, and will change that in the future.

The RGCN model is indeed different to the other models, since its comprises two components:
* a GCN-like encoder, which takes entity embeddings, and enriches them by combining it with embeddings of their neighboring entities. This combination step also involves a relation-specific transformation, a fact in which vanilla GCN and RGCN differ.
* a decoder, which takes a head entity representation, a relation representation and a tail entity representation and computes a scalar score out of it

In the current implementation in the `master` this is reflected as follows:

The attribute `base_embeddings` contains the embeddings for entities
https://github.com/pykeen/pykeen/blob/e8894ec4c787f12eb16464e98a64ceea505f0168/src/pykeen/models/unimodal/rgcn.py#L215-L222

The attribute `base_model` contains the decoder model
https://github.com/pykeen/pykeen/blob/e8894ec4c787f12eb16464e98a64ceea505f0168/src/pykeen/models/unimodal/rgcn.py#L204-L214

In the `score_hrt` where we compute the score for a triple (h, r, t), we first enrich the embeddings by message passing / relation-specific GCN, and then evaluate the base_models interaction function on it.
https://github.com/pykeen/pykeen/blob/e8894ec4c787f12eb16464e98a64ceea505f0168/src/pykeen/models/unimodal/rgcn.py#L526-L529

This part is quite hacky, since we overwrite the `base_models` own entity representations (which are never used) with the enriched ones.

`enriched_embeddings` stores the entity representations *after* message passing and is used as a cache to accelerate computing scores when no parameters change, e.g. during evaluation
https://github.com/pykeen/pykeen/blob/e8894ec4c787f12eb16464e98a64ceea505f0168/src/pykeen/models/unimodal/rgcn.py#L493-L494


Thus, assuming that you have finished training the model, and do not change the model's parameters, you can use `enriched_embeddings` as "enriched" entity representations.

For computing the scores as in the `DistMult` model in which I'll need the final embeddings of 
entities and relations. How to access such as data and what do you refer to them?
Seems the `enriched_embeddings` are the final embeddings for entities? how about relations matrices ?

> For computing the scores as in the `DistMult` model in which I'll need the final embeddings of
> entities and relations. How to access such as data and what do you refer to them?
> Seems the `enriched_embeddings` are the final embeddings for entities? how about relations matrices ?

You can directly use the `RGCN.score_*` methods to get triple scores from the decoder model on the enriched embeddings. 

The enriched_embeddings are used as entity representations, and there is an additional set of relation-specific weights stored in the decoder model, which you can access using `rgcn_model.base_model.relation_embeddings`.

On the topic of extracting post processing information, whenever I run a TransE and TransD model I do not get a hits@1 score. It is always 0. I get hits@3,5 and 10 just not 1. 

Is that a bug or is that something to do with the Trans style models? I haven't gone into fully understanding how they work yet.

Here are some graphs to show the outputs that I get for different embedding sizes
[report_embedding_size.zip](https://github.com/pykeen/pykeen/files/5393803/report_embedding_size.zip)


> On the topic of extracting post processing information, whenever I run a TransE and TransD model I do not get a hits@1 score. It is always 0. I get hits@3,5 and 10 just not 1.
> 
> Is that a bug or is that something to do with the Trans style models? I haven't gone into fully understanding how they work yet.
> 
> Here are some graphs to show the outputs that I get for different embedding sizes
> [report_embedding_size.zip](https://github.com/pykeen/pykeen/files/5393803/report_embedding_size.zip)

Hi @ChristopherMarais ,

I think is is neither a bug, nor something very specific to the Trans* models. Hits@1 is a very strict metric: The correct entity has to occur at the first place in the ranking; If it only occurs at second place, this triple is scored with a zero. Thus, we often observe that it takes a longer time until the model is capable of doing the correct predictions at first place, and hence improve in H@1, while other metrics, e.g. mean rank often show a consistent improvement throughout the training. 

Here is a configuration for TransE and Nations which achieved non-zero Hits@1 for me:
```python
from pykeen.pipeline import pipeline
result = pipeline(
	model="TransE",
	dataset="Nations",
	training_kwargs=dict(num_epochs=100),
	optimizer_kwargs=dict(lr=1.0e-02),
	random_seed=42,
)
```
```python-console
>>> print(result.metric_results.get_metric(name="hits_at_1"))
0.004975124378109453
```