Repository: pykeen/pykeen
Issue: Only update non-zero values in projection matrix in TransR?
Root cause: Validation Gap

Hi, I am doing some modifications for transR. I want to implement #579 finally but I am starting with some simple modifications first. I have some questions about updating the projection matrix.

For example, there are three relations` rel1, rel2, rel3`. `rel1` connects 20% entity pairs, `rel2` connects 30% entity pairs, and `rel3` connects 50% entity pairs. Accordingly, I want to realize that in the projection matrixes of` rel1, rel2, `and `rel3`, 80%, 70%, and 50% of values are zeros, respectively. I modified the `_projection_initializer` in [pykeen.models.unimodal.trans_r](https://pykeen.readthedocs.io/en/stable/_modules/pykeen/models/unimodal/trans_r.html#TransR) as follows:
```
def _projection_initializer(
    x: torch.FloatTensor,
    num_relations: int,
    embedding_dim: int,
    relation_dim: int,
) -> torch.FloatTensor:
    """Initialize by Glorot."""
    m_t = torch.nn.init.xavier_uniform_(x.view(num_relations, embedding_dim, relation_dim)).view(x.shape)
    ratio = [0.8, 0.7, 0.5]
    for i in range(3):
        for j in range(1500): # 1500: entity_dim * relation_dim 
            if random.random() <= ratio[i]:
                m_t[i][j] = 0
    return m_t
```
But when I run the transR again as follows:
```
results_transR = pipeline(
    training=training,
    testing=testing,
    validation=validation,
    model='transR',
    training_kwargs=dict(num_epochs=10,batch_size=512),
    random_seed=42
)
```
there are no zeros values in the projection matrix, as follows:

```
model = torch.load(MODEL)  # load the trained transR model
projection_representation_modules: List['pykeen.nn.RepresentationModule'] = model.relation_projections
projection_embeddings: pykeen.nn.Embedding = projection_representation_modules
projection_embedding_tensor: torch.FloatTensor = projection_embeddings()
print(torch.count_nonzero(projection_embedding_tensor[0]))
```
Output:
`tensor(1500)`

So I think the zeros in the initialzied projection matrix were updated in the training process. How to only update the non-zero values in the projection matrix? Thanks!


Hi @Hao-666 ,

the projection matrices are still dense, and just happen to have some zeros in the beginning. Thus, the optimizer will also update these zero entries resulting in the observed behaviour.

I think there are (at least) two ways how you could proceed:
* use sparse tensors instead, cf. https://pytorch.org/docs/stable/sparse.html
* use a dense tensor, and element-wise multiply it by a "sparsity" mask (a non-trainable one-hot tensor of the same shape with 1 everywhere where you want to keep the values and 0 otherwise). This has to be done whenever the projection is used. You might want to look into the [property decorator](https://docs.python.org/3.8/library/functions.html#property) for implementation.

Thanks @mberr . For the first way, do you mean that once the sparse tensors are constructed in the initialization as the projection matrixes, only the non-zero values will be updated afterwards?

> Thanks @mberr . For the first way, do you mean that once the sparse tensors are constructed in the initialization as the projection matrixes, only the non-zero values will be updated afterwards?

Yes, exactly. You can try it yourself:
```python
import torch
from torch import optim, nn

n = 5
nnz = 7

indices = torch.randint(n, size=(2, nnz))
values = nn.Parameter(torch.rand(nnz))

max_iteration = 100
optimizer = optim.SGD(lr=0.01, params=[values])
for i in range(max_iteration):
	m = torch.sparse_coo_tensor(indices, values, size=(n, n))
	print(m.to_dense())
	optimizer.zero_grad()
        # we optimize some mock loss, which encourages entries to become close to 1
	loss = (1 - m.to_dense()).pow(2).mean()
	loss.backward()
	optimizer.step()
```

Notice how only the non-zero entries change.