Repository: pykeen/pykeen
Issue: Unexpected Results with Toy Example
Root cause: Documentation Deficiency

I have created a toy example as follows:
```
Brussels	locatedIn	Belgium
Belgium	partOf	EU
EU	hasCapital	Brussels
```

As far as I understand, there should be a trivial embedding solution in two dimensions (the three entities form a triangle, with the three relations being their connecting vectors). The code I use for this is as follows:

```python
results = pipeline(
    training_triples_factory=tf,
    testing_triples_factory=tf,
    model = 'TransE',
    model_kwargs=dict(embedding_dim=2),
    random_seed=1,
    device='cpu',
)
```

However, the result looks kind of unexpected:
![result](https://user-images.githubusercontent.com/3635132/94903232-c6fe6400-0499-11eb-9dba-df55633ec2ab.png)

I also played with different random seeds and training parameters, but I never came to a sensible solution. DistMult and HolE, among others, come up with a similar solution.

Am I doing something wrong? Or is my conceptual understanding of the expected outcome incorrect?

Hi @HeikoPaulheim,

first of all thank you for your interest in knowledge graph embeddings and our library in particular!

The result looks indeed unexpected, but I can see a few possible reasons for it:

1. The model is not converged. To check this, could you please take a look at the losses history? You can do so e.g. by

```python
from matplotlib import pyplot as plt
plt.plot(result.losses)

# Or, if you want to use our builtin plot method for this
result.plot_losses() 
```

*addition from @cthoyt, there's actually a [convenience function](https://pykeen.readthedocs.io/en/latest/reference/pipeline.html#pykeen.pipeline.PipelineResult.plot_losses) for this already :) I added it in the above code block under the `Or, ...`. 



Does the loss decrease? If not, you may need to tune some parameters, e.g. the optimizer's learning rate, or the number of epochs you train the model. You can do so by passing custom `optimizer_kwargs` and `training_kwargs` to the pipeline, e.g.

```python
result = pipeline(
    training_triples_factory=tf,
    testing_triples_factory=tf,
    model='TransE',
    model_kwargs=dict(embedding_dim=2),
    random_seed=1,
    device='cpu',
    optimizer_kwargs=dict(lr=1.0e-01),
    training_kwargs=dict(num_epochs=1024),
)
```

As a general rule of thumb, increase the number of epochs until you observe some form of convergence. Start with a large learning rate. It you observe divergence, decrease the learning rate. Please notice that there is some stochasticity in the training, since we sample negative examples for positive ones. Thus, the loss may fluctuate naturally. To better see the trend, you can smooth the loss by averaging over a window of epochs. 


2. We use a margin-based loss with TransE by default. Thus, it suffices if the model predicts scores such that the scores of positive triples and negative triples are at least one margin apart. Once the model has reached this state, if will not improve further upon these examples, as the embeddings are "good enough". Hence, an optimal solution with margin-based loss might not look like the exact geometric solution. If you want to change that you can switch to a loss function which does not use a margin, e.g. the softplus loss. You can do this by passing `loss="softplus"` to the pipeline, e.g.

```python
result = pipeline(
    training_triples_factory=tf,
    testing_triples_factory=tf,
    model='TransE',
    model_kwargs=dict(embedding_dim=2),
    random_seed=1,
    device='cpu',
    loss='softplus',
)
```



When applying both changes, I obtain a result which is closer to what you would expect.

```python
from pykeen.pipeline import pipeline

result = pipeline(
    training_triples_factory=tf,
    testing_triples_factory=tf,
    model='TransE',
    model_kwargs=dict(embedding_dim=2),
    random_seed=1,
    device='cpu',
    loss='softplus',
    optimizer_kwargs=dict(lr=1.0e-01),
    training_kwargs=dict(num_epochs=128),
)
```

![loss](https://user-images.githubusercontent.com/36984984/94916330-cc19de00-04ae-11eb-8a70-fa3364cfdbff.png)
![emb](https://user-images.githubusercontent.com/36984984/94916377-dd62ea80-04ae-11eb-81ae-d3ea996ada34.png)



@mberr will you add a PR with the code for generating that plot? I'm sure other people will find it useful!

> @mberr will you add a PR with the code for generating that plot? I'm sure other people will find it useful!

The loss plot [is already in the repository](https://pykeen.readthedocs.io/en/latest/reference/pipeline.html#pykeen.pipeline.PipelineResult.plot_losses), although I re-implemented it :see_no_evil: 

As for the second plot, its usage seems to be restricted to this toy example, as long as we do not use some [dimensionality reduction strategy](https://scikit-learn.org/stable/modules/unsupervised_reduction.html) to project the embedding space into two dimensions.

Moreover, the relation arrow heads somehow got lost in my plot compared to @HeikoPaulheim 's version.

Here is the code used to reproduce my second plot:
```python
from matplotlib import pyplot as plt

# draw entities
e_emb = result.model.entity_embeddings.weight.detach().numpy()
e_id_to_label = {
    eid: el
    for el, eid in tf.entity_to_id.items()
}
for i, e in enumerate(e_emb):
    plt.scatter(*e, color='black')
    plt.annotate(e_id_to_label[i], e)

# draw relations
r_emb = result.model.relation_embeddings.weight.detach().numpy()
r_id_to_label = {
    rid: rl
    for rl, rid in tf.relation_to_id.items()
}
for i, r in enumerate(r_emb):
    plt.arrow(0, 0, *r)
    plt.annotate(r_id_to_label[i], r)
```


Thanks @mberr! With these changes, I also got it to converge :-)

As an additional information: The default value of `num_epochs` is `5`, which might be more appropriate for larger datasets than for smaller ones.

https://github.com/pykeen/pykeen/blob/852e25e5deb1b16dd4450769d644b7dc18564b4a/src/pykeen/pipeline.py#L789

In #99, I've built on @mberr's code as well as implemented the functionality for generating this plot for arbitrarily large entity and relation embeddings using a choice of dimensionality reduction techniques that come with `sklearn` (e.g., PCA, KPCA, Gaussian Process). It also comes with a user-facing function `PipelineResult.plot()` that does all the hard work and makes this:

![index](https://user-images.githubusercontent.com/5069736/95676164-720ace00-0bbc-11eb-817a-8fc47129d336.png)

There's a demo notebook here with the toy example: https://github.com/pykeen/pykeen/blob/add-entity_plot/notebooks/Pipeline%20Plots%20Demo.ipynb. ~@mberr @HeikoPaulheim could you suggest tweaks to make it converge and get the desired results?~ Update: I forgot to copy the `loss='softplus'`, even after @mberr wrote up such a nice explanation of why this was important. Now it's looking much nicer as of 18abc448.

![index](https://user-images.githubusercontent.com/5069736/95676264-083ef400-0bbd-11eb-96bb-e9b87e2c9ba5.png)