Repository: pykeen/pykeen
Issue: Got different scores for the same triples from the same model using `TriplesFactory.from_path` and `TriplesFactory.from_labeled_triples`
Root cause: Documentation Deficiency

**Describe the bug**
Hi, I have trained a model and have a dataset comprised of several triples, and I want to calculate the scores for them. When I used `TriplesFactory.from_path` and `TriplesFactory.from_labeled_triples` respectively, the scores are different. 

**To Reproduce**

```
import...

NEW_TRIPLE = "new triple.txt"
model = torch.load(MODEL)

# using from_path
tf = TriplesFactory.from_path(NEW_TRIPLE)
scores = model.score_hrt(tf.mapped_triples.to(model.device))
print(scores)
```
Output:
```
tensor([[-0.9651],
        [-1.8023],
        [-1.5763],
        [-0.2474],
        [-1.7150],
        [-1.0077]], grad_fn=<ViewBackward>)
```

```
# using from_labeled_triples
# file format: head\trelation\ttail\n
with open(NEW_TRIPLE) as f:
    new_triples = f.readlines()
new_triples = [x.strip() for x in new_triples] 
for new_triple in new_triples:
    new_triple = new_triple.split("\t")
    new_triple = np.array([[new_triple[0],new_triple[1],new_triple[2]]])
    tf = TriplesFactory.from_labeled_triples(new_triple)
    score = model.score_hrt(tf.mapped_triples.to(model.device))
    print(score)
```

Output:
```
tensor([[-1.3907]], grad_fn=<ViewBackward>)
tensor([[-1.3770]], grad_fn=<ViewBackward>)
tensor([[-1.3770]], grad_fn=<ViewBackward>)
tensor([[-1.3907]], grad_fn=<ViewBackward>)
tensor([[-1.3907]], grad_fn=<ViewBackward>)
tensor([[-1.3770]], grad_fn=<ViewBackward>)
```

**Expected behavior**
The inputs are the same triples so the two output scores should be the same.

**Environment:**
 Key             | Value                    |
|-----------------|--------------------------|
| OS              | nt                       |
| Platform        | Windows                  |
| Release         | 10                       |
| User            | Hao Liu                  |
| Time            | Thu Jul  8 20:38:58 2021 |
| Python          | 3.7.10                   |
| PyKEEN          | 1.5.1-dev                |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.9.0                    |
| CUDA Available? | false                    |
| CUDA Version    | N/A                      |
| cuDNN Version   | N/A                      |


you have to put the model in evaluation mode and disable gradient tracking (will write a more thorough explanation later)

you could take inspiration from related code in the prediction module: https://github.com/pykeen/pykeen/blob/55d73324a957f3ce466869899abeac7e34e596fb/src/pykeen/models/predict.py#L307-L340

Thanks @cthoyt , I used `model.eval()` and `with torch.no_grad() `but the situation still did not change.
```python
import...

NEW_TRIPLE = "new triple.txt"
model = torch.load(MODEL)

# using from_path
tf = TriplesFactory.from_path(NEW_TRIPLE)
with torch.no_grad():
    scores = model.score_hrt(tf.mapped_triples.to(model.device))
    print(scores)
```
Output:
```
tensor([[-0.9651],
        [-1.8023],
        [-1.5763],
        [-0.2474],
        [-1.7150],
        [-1.0077]])
```

```python
with open(NEW_TRIPLE) as f:
    new_triples = f.readlines()
new_triples = [x.strip() for x in new_triples] 
for new_triple in new_triples:
    new_triple = new_triple.split("\t")
    new_triple = np.array([[new_triple[0],new_triple[1],new_triple[2]]])
    tf = TriplesFactory.from_labeled_triples(new_triple)
    with torch.no_grad():
        score = model.score_hrt(tf.mapped_triples.to(model.device))
        print(score)
```
Output:
```
tensor([[-1.3907]])
tensor([[-1.3770]])
tensor([[-1.3770]])
tensor([[-1.3907]])
tensor([[-1.3907]])
tensor([[-1.3770]])
```

`TriplesFactory.from_labeled_triples(new_triple)` creates a new triples factory in each inner loop, which likely only has two entities.

You need to make sure that the triples factory you create or use for evaluation uses the same label-to-id mappings, otherwise you likely end up with evaluating the triples `[0, 0, 1]`, `[1, 0, 0]` or `[0, 0, 0]` (which seems to be confirmed from your scores).

> `TriplesFactory.from_labeled_triples(new_triple)` creates a new triples factory in each inner loop, which likely only has two entities.
> 
> You need to make sure that the triples factory you create or use for evaluation uses the same label-to-id mappings, otherwise you likely end up with evaluating the triples `[0, 0, 1]`, `[1, 0, 0]` or `[0, 0, 0]` (which seems to be confirmed from your scores).

Thanks @mberr . What are the measures to ensure the same label-to-id mappings? Sorry I don't get it...

@Hao-666 

When you create the evaluation triples factories, you need to provide `entity_to_id` and `relation_to_id` with which you trained the model. From your code example I cannot directly see where this happens, since I suppose you already trained the model with some other code?

If your model was trained after https://github.com/pykeen/pykeen/pull/498, you may have the mappings stored in your model checkpoint file. Otherwise, you would need to construct the training triples factory first (to be on the safe side using the exactly same code you also used for training), and then use its mappings. Assuming the training triples factory is stored in `training_tf`, the following code would achieve this:
```python
eval_tf = TriplesFactory.from_labeled_triples(
  new_triples,
  entity_to_id=training_tf.entity_to_id,
  relation_to_id=training_tf.relation_to_id,
)
```

Thanks @mberr 

Yes the model was trained after #498. 

Thanks for the example code. After adding the arguments `entity_to_id=tf.entity_to_id, relation_to_id=tf.relation_to_id` to map the ids it worked. Below are the worked codes and outputs for potential latecomers :)  

```python
import ...

TRIPLE = "triple.txt"
NEW_TRIPLE = "new triple.txt"
MODEL = "results_transR/trained_model.pkl"

# train the model
tf = TriplesFactory.from_path(TRIPLE)
training, testing = tf.split()
results = pipeline(
    training=training,
    testing=testing,
    model='TransR',
    training_kwargs=dict(num_epochs=100),  
)
results.save_to_directory('results_transR')

# calculate scores for new triples
model = torch.load(MODEL)
model.eval()

# use TriplesFactory.from_path
tf_eval = TriplesFactory.from_path(NEW_TRIPLE, 
                                   entity_to_id=tf.entity_to_id, 
                                   relation_to_id=tf.relation_to_id)
with torch.no_grad():
    scores = model.score_hrt(tf_eval.mapped_triples.to(model.device))
    print(scores)
    #print(tf_eval.triples)
```

Output:
```
tensor([[-0.2647],
        [-1.4166],
        [-0.3418],
        [-0.3194],
        [-0.3169],
        [-1.4725]])
```

```python
# use from_labeled_triples
with open(NEW_TRIPLE) as f:
    new_triples = f.readlines()
new_triples = [x.strip() for x in new_triples] 
for new_triple in new_triples:
    new_triple = new_triple.split("\t")
    new_triple = np.array([[new_triple[0],new_triple[1],new_triple[2]]])
    tf_eval = TriplesFactory.from_labeled_triples(new_triple, 
                                                  entity_to_id=tf.entity_to_id, 
                                                  relation_to_id=tf.relation_to_id)
    with torch.no_grad():
        score = model.score_hrt(tf_eval.mapped_triples.to(model.device))
        #print(tf_eval.triples)
        print(score)
```
Output:
```
tensor([[-0.3194]])
tensor([[-0.2647]])
tensor([[-0.3418]])
tensor([[-0.3169]])
tensor([[-1.4725]])
tensor([[-1.4166]])
```

The scores are the same. It is just that the positions are different. I used `tf_eval.triples` to check it and it seems that `TriplesFactory.from_path` will sort the triples in an alphabetical order, which is different from reading the `new triple.txt `line by line and causes the different positions of scores.