Repository: pykeen/pykeen
Issue: Are relations with attributes suppoted for KGE training?

Hi, in a knowledge graph, some relations have attributes while some don't. For example, in the below relations, the `has_attribute` relation in the triple `<IfcWorkControl, has_attribute, IfcDateTime>` has a `FinishTime` attribute. The `inherits_from` relation in the triple `<IfcWorkPlan, inherits_from, IfcWorkControl>` does not. I want to consider the relation attribute for the knowledge graph embedding training as the `has_attribute` relation is very generic. I found the RGCN does not support relations with attributes. How can we consider this? Thanks!

   ```
 {
      "source": "IfcWorkControl",
      "target": "IfcDateTime",
      "type": "has_attribute",
      "attribute": "FinishTime"
    },
    {
      "source": "IfcWorkPlan",
      "target": "IfcWorkControl",
      "type": "inherits_from"
    },
    {
      "source": "IfcWorkPlan",
      "target": "IfcWorkPlanTypeEnum",
      "type": "has_attribute",
      "attribute": "PredefinedType"
    },
```

Is the `attribute` a categorical feature, i.e., has only a fixed set of possible values? If yes, you could convert them to individual relations, e.g.,
```json
{
   "source": "IfcWorkControl",
   "target": "IfcDateTime",
   "type": "has_attribute:FinishTime",
 },
 {
   "source": "IfcWorkPlan",
   "target": "IfcWorkControl",
   "type": "inherits_from"
 },
 {
   "source": "IfcWorkPlan",
   "target": "IfcWorkPlanTypeEnum",
   "type": "has_attribute:PredefinedType",
 },
```

> Is the `attribute` a categorical feature, i.e., has only a fixed set of possible values? If yes, you could convert them to individual relations, e.g.,
> 
> {
>    "source": "IfcWorkControl",
>    "target": "IfcDateTime",
>    "type": "has_attribute:FinishTime",
>  },
>  {
>    "source": "IfcWorkPlan",
>    "target": "IfcWorkControl",
>    "type": "inherits_from"
>  },
>  {
>    "source": "IfcWorkPlan",
>    "target": "IfcWorkPlanTypeEnum",
>    "type": "has_attribute:PredefinedType",
>  },

@mberr Thank you for your reply! In the knowledge graph the `attribute` is a fixed set of possible values but the number of values is quite large and the values are quite sparse. For example, if I convert them to individual relations, the total number of unique relation types is `1164` and the total number of triples is `3043` (Originally the number of unique relation types is just `4`). Among the `1164` relation types, `941` only appear once. I think this also causes large size of training parameters? Now I create the individual attributed relations `has_attribute:<attr>` with a threshold (i.e., only the attributes appearing more than 3 times are considered). I wonder are there better solutions to this? 

Your dataset appears quite sparse – with only about three examples per relationship type, it's unlikely that you'll be able to learn good representations. Given that your relation labels are somewhat natural language-like, I'd suggest utilizing transfer learning:

1. Encode relation + attribute strings into vectors with a text encoder, (see the [documentation](https://pykeen.readthedocs.io/en/stable/api/pykeen.nn.init.LabelBasedInitializer.html#pykeen.nn.init.LabelBasedInitializer)).

2. Since your dataset is rather small, keep those representation frozen and learn a small transformation on top of them (see the [documentation](https://pykeen.readthedocs.io/en/stable/tutorial/representations.html#transformed-representations))
 
3. For entities, you have two options:
  - Apply the same text-based technique if entity names contain semantic information, or you have enough training data per entity
  - Train embeddings from scratch


This transfer learning approach should help you make better use of your limited training data.

Here is an example of how you could set it up:

```python
from torch import nn

from pykeen.datasets.utils import get_dataset
from pykeen.models import ERModel
from pykeen.nn import TransformedRepresentation, init
from pykeen.pipeline import pipeline
from pykeen.triples.triples_factory import TriplesFactory

# TODO: we use nations here to have a self-contained example
dataset = get_dataset(dataset="nations")

triples_factory: TriplesFactory = dataset.training

# get label-based features
relation_features = init.LabelBasedInitializer.from_triples_factory(
    triples_factory=triples_factory,
    for_entities=False,
    # note: uses https://huggingface.co/google-bert/bert-base-cased by default
    encoder="transformer",
).as_embedding()

# add transformation, here a simple linear projection
embedding_dim = 32
relation_representations = TransformedRepresentation(
    transformation=nn.Linear(in_features=relation_features.shape[0], out_features=embedding_dim),
    base=relation_features,
)

# compose model
model = ERModel(
    interaction="distmult",
    triples_factory=triples_factory,
    relation_representations=relation_representations,
    # we use default representations, i.e., an embedding, with appropriate dimension
    entity_representations_kwargs={"embedding_dim": embedding_dim},
)

# train model
result = pipeline(dataset=dataset, model=model)

```

> Your dataset appears quite sparse – with only about three examples per relationship type, it's unlikely that you'll be able to learn good representations. Given that your relation labels are somewhat natural language-like, I'd suggest utilizing transfer learning:
> 
> 1. Encode relation + attribute strings into vectors with a text encoder, (see the [documentation](https://pykeen.readthedocs.io/en/stable/api/pykeen.nn.init.LabelBasedInitializer.html#pykeen.nn.init.LabelBasedInitializer)).
> 2. Since your dataset is rather small, keep those representation frozen and learn a small transformation on top of them (see the [documentation](https://pykeen.readthedocs.io/en/stable/tutorial/representations.html#transformed-representations))
> 3. For entities, you have two options:
> 
> * Apply the same text-based technique if entity names contain semantic information, or you have enough training data per entity
> * Train embeddings from scratch
> 
> This transfer learning approach should help you make better use of your limited training data.
> 
> Here is an example of how you could set it up:
> 
> from torch import nn
> 
> from pykeen.datasets.utils import get_dataset
> from pykeen.models import ERModel
> from pykeen.nn import TransformedRepresentation, init
> from pykeen.pipeline import pipeline
> from pykeen.triples.triples_factory import TriplesFactory
> 
> # TODO: we use nations here to have a self-contained example
> dataset = get_dataset(dataset="nations")
> 
> triples_factory: TriplesFactory = dataset.training
> 
> # get label-based features
> relation_features = init.LabelBasedInitializer.from_triples_factory(
>     triples_factory=triples_factory,
>     for_entities=False,
>     # note: uses https://huggingface.co/google-bert/bert-base-cased by default
>     encoder="transformer",
> ).as_embedding()
> 
> # add transformation, here a simple linear projection
> embedding_dim = 32
> relation_representations = TransformedRepresentation(
>     transformation=nn.Linear(in_features=relation_features.shape[0], out_features=embedding_dim),
>     base=relation_features,
> )
> 
> # compose model
> model = ERModel(
>     interaction="distmult",
>     triples_factory=triples_factory,
>     relation_representations=relation_representations,
>     # we use default representations, i.e., an embedding, with appropriate dimension
>     entity_representations_kwargs={"embedding_dim": embedding_dim},
> )
> 
> # train model
> result = pipeline(dataset=dataset, model=model)

@mberr Thank you very much for your detailed help! Will try it out!