Repository: pykeen/pykeen
Issue: Logging of TriplesFactory paths in pipeline() is misleading
Root cause: Validation Gap

**Describe the bug**

I train an hpo pipeline and specify my dataset as triple factories using the training, testing and validation keyword arguments. However, in the logs validation and testing is set to None. This means that either the keyword arguments or the logging don't work correctly.

**To Reproduce**

```python
training = TriplesFactory(
    path=training_path,
    create_inverse_triples=True,
)
validation = TriplesFactory(
    path=validation_path,
    entity_to_id=training.entity_to_id,
    relation_to_id=training.relation_to_id,
    create_inverse_triples=True,
)
testing = TriplesFactory(
    path=testing_path,
    entity_to_id=training.entity_to_id,
    relation_to_id=training.relation_to_id,
    create_inverse_triples=True,
)

# initialize logging to a logfile with level DEBUG
logger = getConfiguredLogger(__name__)

result = hpo_pipeline(
    training=training,
    validation=validation,
    testing=testing,
    n_trials=30,
    model='SimplE',
    stopper='early',
    stopper_kwargs=dict(frequency=5, patience=2, relative_delta=0.002, metric='mean_reciprocal_rank'),
    evaluator="rankbased",
    metric="mean_reciprocal_rank",
    direction="maximize",
    negative_sampler_kwargs=dict(num_negs_per_pos=1),
    evaluator_kwargs=dict(filtered=True, ks={1,3,5,10,20}),
    n_jobs = 1,
    evaluation_relation_whitelist=evaluation_relation_whitelist,
)
```
logfile shows the following lines:
```
2020-11-10 10:57:38,676 - root - DEBUG - Run Pipeline based on following config:
2020-11-10 10:57:38,677 - root - DEBUG - training:  path/to/training_set
2020-11-10 10:57:38,683 - root - DEBUG - testing: <None>
2020-11-10 10:57:38,683 - root - DEBUG - validation: <None>
2020-11-10 10:57:38,683 - root - DEBUG - model: <class 'pykeen.models.unimodal.simple.SimplE'>
(output truncated)
```
**Expected behavior**
It prints the paths of my validation and test set in the logfile.

**Environment (please complete the following information):**
 OS: Ubuntu 18.04.5 LTS
Python version: 3.8.5
Version of this software: 1.0.6
Versions of required Python packages:
    click 7.1.2
    click-default-group 1.2.2
    numpy 1.19.2
    torch 1.6.0
    torch-cluster 1.5.7
    torch-geometric 1.6.1
    torch-scatter 2.0.5
    torch-sparse 0.6.7
    torch-spline-conv 1.2.0
    tqdm 4.51.0


## Explanation

Because the relation whitelist was used, a triples factory is derived from the original

https://github.com/pykeen/pykeen/blob/c1a56f50407280df794f6f2a5a7e0e4ff2040161/src/pykeen/triples/triples_factory.py#L659-L666

ultimately, because the new one is created based on the triples and not the path, this is where the path gets set to `"<None>"`.

https://github.com/pykeen/pykeen/blob/c1a56f50407280df794f6f2a5a7e0e4ff2040161/src/pykeen/triples/triples_factory.py#L230-L232

Do you get the same error if you use the paths directly in the `hpo_pipeline` function and not the triples factories?

@cthoyt yes, i do get the same log output.

@fratajcz okay thanks for the feedback. I'm going to look into this now

I was not able to reproduce from the current master branch (since you just wrote pykeen 1.0.6, which is unrelease, I assume you're on the most recent)

```python
import logging
import os

from pykeen.datasets.nations import NATIONS_TEST_PATH, NATIONS_TRAIN_PATH, NATIONS_VALIDATE_PATH
from pykeen.hpo import hpo_pipeline
from pykeen.triples import TriplesFactory


def main():
    logging.basicConfig(level=logging.DEBUG)

    training_path = NATIONS_TRAIN_PATH
    validation_path = NATIONS_VALIDATE_PATH
    testing_path = NATIONS_TEST_PATH

    training = TriplesFactory(
        path=training_path,
        create_inverse_triples=True,
    )
    validation = TriplesFactory(
        path=validation_path,
        entity_to_id=training.entity_to_id,
        relation_to_id=training.relation_to_id,
        create_inverse_triples=True,
    )
    testing = TriplesFactory(
        path=testing_path,
        entity_to_id=training.entity_to_id,
        relation_to_id=training.relation_to_id,
        create_inverse_triples=True,
    )
    result = hpo_pipeline(
        training=training,
        validation=validation,
        testing=testing,
        n_trials=30,
        model='SimplE',
        stopper='early',
        stopper_kwargs=dict(frequency=5, patience=2, relative_delta=0.002, metric='mean_reciprocal_rank'),
        evaluator="rankbased",
        metric="mean_reciprocal_rank",
        direction="maximize",
        negative_sampler_kwargs=dict(num_negs_per_pos=1),
        evaluator_kwargs=dict(filtered=True, ks={1, 3, 5, 10, 20}),
        n_jobs=1,
    )
    result.save_to_directory(os.path.join(os.path.expanduser('~'), 'Desktop', 'hpotest'))


if __name__ == '__main__':
    main()
```

Results:
```
INFO:pykeen.hpo.hpo:Using model: <class 'pykeen.models.unimodal.simple.SimplE'>
INFO:pykeen.hpo.hpo:Using loss: <class 'pykeen.losses.SoftplusLoss'>
INFO:pykeen.hpo.hpo:Using regularizer: <class 'pykeen.regularizers.PowerSumRegularizer'>
INFO:pykeen.hpo.hpo:Using optimizer: <class 'torch.optim.adagrad.Adagrad'>
INFO:pykeen.hpo.hpo:Using training loop: <class 'pykeen.training.slcwa.SLCWATrainingLoop'>
INFO:pykeen.hpo.hpo:Using negative sampler: <class 'pykeen.sampling.basic_negative_sampler.BasicNegativeSampler'>
INFO:pykeen.hpo.hpo:Using evaluator: <class 'pykeen.evaluation.rank_based_evaluator.RankBasedEvaluator'>
INFO:pykeen.hpo.hpo:Attempting to maximize mean_reciprocal_rank
WARNING:pykeen.hpo.hpo:bad range given: range(32, 4000, 100) - not divisible by q
WARNING:pykeen.pipeline:No random seed is specified. Setting to 1990363277.
WARNING:pykeen.utils:No cuda devices were available. The model runs on CPU
DEBUG:root:Run Pipeline based on following config:
DEBUG:root:training: /Users/cthoyt/dev/pykeen/src/pykeen/datasets/nations/train.txt
DEBUG:root:testing: /Users/cthoyt/dev/pykeen/src/pykeen/datasets/nations/test.txt
DEBUG:root:validation: /Users/cthoyt/dev/pykeen/src/pykeen/datasets/nations/valid.txt
DEBUG:root:model: <class 'pykeen.models.unimodal.simple.SimplE'>
DEBUG:root:model_kwargs: {'embedding_dim': 325, 'preferred_device': device(type='cpu'), 'random_seed': 1990363277, 'regularizer': PowerSumRegularizer(), 'loss': SoftplusLoss(
  (softplus): Softplus(beta=1, threshold=20)
)}
DEBUG:root:loss: <class 'pykeen.losses.SoftplusLoss'>
DEBUG:root:loss_kwargs: {}
DEBUG:root:regularizer: <class 'pykeen.regularizers.PowerSumRegularizer'>
DEBUG:root:regularizer_kwargs: {'weight': 0.041280478791912316}
DEBUG:root:optimizer: <class 'torch.optim.adagrad.Adagrad'>
DEBUG:root:optimizer_kwargs: {'lr': 0.0036768511834511207, 'lr_decay': 0.0064744546069331775, 'weight_decay': 0.5529491697507914}
DEBUG:root:training_loop: <class 'pykeen.training.slcwa.SLCWATrainingLoop'>
DEBUG:root:negative_sampler: <class 'pykeen.sampling.basic_negative_sampler.BasicNegativeSampler'>
DEBUG:root:_negative_sampler_kwargs: {'num_negs_per_pos': 1}
DEBUG:root:_training_kwargs: {'num_epochs': 500, 'batch_size': 432}
DEBUG:root:stopper: EarlyStopper(evaluator=<pykeen.evaluation.rank_based_evaluator.RankBasedEvaluator object at 0x13f3bd760>, evaluation_triples_factory=TriplesFactory(path="/Users/cthoyt/dev/pykeen/src/pykeen/datasets/nations/valid.txt"), evaluation_batch_size=None, evaluation_slice_size=None, frequency=5, patience=2, metric='mean_reciprocal_rank', relative_delta=0.002, best_metric=None, best_epoch=None, remaining_patience=2, larger_is_better=True, result_tracker=<pykeen.trackers.ResultTracker object at 0x13f3bd550>, stopped=False)
DEBUG:root:stopper_kwargs: {'frequency': 5, 'patience': 2, 'relative_delta': 0.002, 'metric': 'mean_reciprocal_rank', 'result_callbacks': [<function Objective._update_stopper_callbacks.<locals>._result_callback at 0x1171213a0>], 'stopped_callbacks': [<function Objective._update_stopper_callbacks.<locals>._stopped_callback at 0x13f3ea9d0>]}
DEBUG:root:evaluator: <class 'pykeen.evaluation.rank_based_evaluator.RankBasedEvaluator'>
DEBUG:root:evaluator_kwargs: {'filtered': True, 'ks': {1, 3, 5, 10, 20}}
Training epochs on cpu:   0%|          | 0/500 [00:00<?, ?epoch/s]
```

Thanks for looking into this issue. Weirdly, I can't reproduce the error with the code you provided either. However, my paths are correct and I don't get an error during initializing the factories or during training, testing or validation. Is there any way how i can test if my factories deliver the correct triples?

From

https://github.com/pykeen/pykeen/blob/c1a56f50407280df794f6f2a5a7e0e4ff2040161/src/pykeen/triples/triples_factory.py#L230-L232

the output seems to indicate that `path is None`.

@cthoyt I think it is caused by the relation_restriction for evaluation.

https://github.com/pykeen/pykeen/blob/c1a56f50407280df794f6f2a5a7e0e4ff2040161/src/pykeen/triples/triples_factory.py#L660-L666

here, we do not copy the path to the new, restricted, factory.

Ah yes. Past Charlie is laughing at us. I think the reason it's that way is it would be a bit misleading to say it's just the path. Maybe I can update the repr to include the entity restriction information 

I think a problem is that we cannot pass both, triples and path, to the constructor of TriplesFactory. We should think how to relax that, e.g. not directly use the constructor, but `TriplesFactory.from_path` and `TriplesFactory.from_triples`, and do the check there.

Thanks a lot for figuring this out! So i get that the error is due do the internal handling of the path attribute and should not affect the performance?

> Thanks a lot for figuring this out! So i get that the error is due do the internal handling of the path attribute and should not affect the performance?

Yes, as far as I can judge it now, it should only be a problem with displaying the correct path, and not about using wrong triples.

@fratajcz thanks for being patient, this has been solved with #211