Repository: pykeen/pykeen
Issue: Run best pipeline after HPO using pipeline_from_path
Root cause: Documentation Deficiency

**Describe the bug**
When using pipeline_from_path to run the best pipeline config, `user defined` error occured. 
`OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect error. : '<user defined>'`

**To Reproduce**
```
PATH = "pipeline_config.json"
results = pipeline_from_path(path=PATH)
```
Contents of pipeline_config:
```
{
  "metadata": {
    "best_trial_evaluation": 0.8582068882917007,
    "best_trial_number": 4,
    "git_hash": "UNHASHED",
    "version": "1.5.1-dev"
  },
  "pipeline": {
    "evaluator": "rankbased",
    "filter_validation_when_testing": true,
    "loss": "marginranking",
    "loss_kwargs": {
      "margin": 0,
      "margin_activation": "softplus"
    },
    "model": "transr",
    "model_kwargs": {
      "embedding_dim": 208,
      "relation_dim": 256,
      "scoring_fct_norm": 1
    },
    "negative_sampler": "basic",
    "negative_sampler_kwargs": {
      "num_negs_per_pos": 2
    },
    "optimizer": "adam",
    "optimizer_kwargs": {
      "lr": 0.04169154977419964
    },
    "testing": "<user defined>",
    "training": "<user defined>",
    "training_kwargs": {
      "batch_size": 128,
      "num_epochs": 200
    },
    "training_loop": "slcwa",
    "validation": "<user defined>"
  }
}
```

**Expected behavior**
Run the best pipeline config.

**Environment:**

 Key             | Value                    |
|-----------------|--------------------------|
| OS              | nt                       |
| Platform        | Windows                  |
| Release         | 10                       |
| User            | Hao Liu                  |
| Time            | Mon Jul 26 10:52:36 2021 |
| Python          | 3.7.10                   |
| PyKEEN          | 1.5.1-dev                |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.9.0                    |
| CUDA Available? | false                    |
| CUDA Version    | N/A                      |
| cuDNN Version   | N/A                      |

**Additional information**
The "training", "testing", and "validation" in the best pipeline config file are "user defined", probably that's the reason? How to define them? Thanks!


Hi @Hao-666 ,

you need to replace `<user defined>` by an actual path. It looks like you want to use your own triples, which already have been split into train/validation/test, correct? Then you should put the paths to the respective files there.

Thanks @mberr . Yes I split the triples into train/validation/test sets as below. 
```
tf = TriplesFactory.from_path(TRIPLE)
training, testing, validation = tf.split([.6, .2, .2],random_state=42)
```
But There seems no method to export the split TripleFactory objects as some file formats outside? 

`TriplesFactory.from_path` is a bit magical depending on the file path. Can you please share what that looks like?

Also, do you mean that you get an error when you run this code? Can you please include the full stack trace?

Hi @cthoyt ,

It is like this:
```
TRIPLE = "gold triple.txt"
tf = TriplesFactory.from_path(TRIPLE)
training, testing, validation = tf.split([.6, .2, .2],random_state=42)
print(tf)
```
Output:
`TriplesFactory(num_entities=165, num_relations=5, num_triples=2222, inverse_triples=False, path="C:\Users\......\gold triple.txt")`

There is no error loading the triples. The error occured when trying to run the best pipeline config. As explained by @mberr the `user_defined` strings should be replaced by actual triple paths. But now my problem is that it seems there is no method to export the split triples into some file formats so that I can type that file paths in the pipeline config file. 

Full code:
```
import...
PATH = "pipeline_config.json" #The best pipeline config after HPO
TRIPLE = "gold triple.txt"
tf = TriplesFactory.from_path(TRIPLE)
training, testing, validation = tf.split([.6, .2, .2],random_state=42)
results = pipeline_from_path(path=PATH)
```

@Hao-666 that's a really good point! The triples factories are effectively wrappers around a 3-column numpy array integer representations for triples, and two associated label dictionaries. We don't have functionality built in for exporting these (should be trivial enough with numpy), but this would definitely be nice to have for convenience

obv the HPO pipeline can't handle all possible cases. It's set up for you to completely automate your workflow from end to end. If you want to go back and rerun the best configuration, it won't be possible since the triples factory is split on-the-fly in the pipeline. Therefore the solution (as you've figure out) is to split the triples ahead of time and use them as the arguments for the HPO pipeline in the first place, or in your case, by slightly modifying the output later (however, please be aware of #499, which could throw a wrench in the whole idea due to a bug I don't quite yet understand)

@mberr and the PyKEEN team have had several discussions about making a pipeline for preparing datasets, including pre-splitting and pre-indexing the triples. Maybe we can look back into that

@cthoyt I see. Thanks! 

Given that the triple factories are numpy array representations, I exported the numpy values into txt as training.txt, validation.txt, and testing.txt after spliting the whole dataset and then typed these three paths in the best pipeline config. Here is the summary of the workflow of using HPO to obtain optimized parameters and then using the best pipeline config file to train for reference :)

```
import...
TRIPLE = '... .txt' # dataset path
PATH = 'pipeline_config.json' #The best pipeline config generated after HPO

# load and split dataset 
tf = TriplesFactory.from_path(TRIPLE)
training, testing, validation = tf.split([.6, .2, .2],random_state=42)

# HPO to obtain best config
results_optim = hpo_pipeline(
    training=training,
    testing=testing,
    validation=validation,
    ... # some other arguments
)
results_optim.save_to_directory('...')

# export split training, validation, and testing triples into txt files
TRAINING_TRIPLE = '... .txt'
VALIDATION_TRIPLE = '... .txt'
TESTING_TRIPLE = '... .txt'

for i in range(len(training.triples)):
    with open(TRAINING_TRIPLE, 'a') as f:
        f.write(str(training.triples[i][0]) + "\t" + str(training.triples[i][1]) + "\t" + str(training.triples[i][2]) + "\n")
for i in range(len(validation.triples)):
    with open(VALIDATION_TRIPLE, 'a') as f:
        f.write(str(validation.triples[i][0]) + "\t" + str(validation.triples[i][1]) + "\t" + str(validation.triples[i][2]) + "\n")
for i in range(len(testing.triples)):
    with open(TESTING_TRIPLE, 'a') as f:
        f.write(str(testing.triples[i][0]) + "\t" + str(testing.triples[i][1]) + "\t" + str(testing.triples[i][2]) + "\n")

# replace the user-defined strings in the best pipeline config file with the exported txt paths

# run the best pipeline config obtained by the HPO
results = pipeline_from_path(path=PATH)
```

Numpy has a function for this: https://numpy.org/doc/stable/reference/generated/numpy.savetxt.html

The last thing won't work directly -  you'll want to load the JSON configuration, then manually intervene to edit the testing, training, and validation paths to actually be your new paths, then it should work. Something like:

```python
with open(PATH) as file:
   config = json.load(file)

config['training'] = TRAINING_TRIPLE
...

results = pipeline(**config)
```

@cthoyt Thanks for pointing out!

Do you mean replacing the user-defined strings in the best pipeline config json file with the exported txt paths and then running it with pipeline_from_path won't work directly? I did so just now and it worked.

> @cthoyt Thanks for pointing out!
> 
> Do you mean replacing the user-defined strings in the best pipeline config json file with the exported txt paths and then running it with pipeline_from_path won't work directly? I did so just now and it worked.

No, that works too! But it means you had to do it by hand instead of with code and that's not so great for reproducibility. 

> > @cthoyt Thanks for pointing out!
> > 
> > Do you mean replacing the user-defined strings in the best pipeline config json file with the exported txt paths and then running it with pipeline_from_path won't work directly? I did so just now and it worked.
> 
> No, that works too! But it means you had to do it by hand instead of with code and that's not so great for reproducibility. 

Ohhhh sorry I misunderstood your words. Yes you are right:)

I had this problem when I did hpo with triples factory classes which I had implemented. 
I.e. when I passed a triple factory object to the `hpo_config`, instead of a path I incurred in the `<user defined>` problem. 

This is how I fixed this:

```
training = InductiveNumericLiteralsFactory( ... )  # implemented by me
testing = InductiveNumericLiteralsFactory( ... )  # implemented by me
validation = InductiveNumericLiteralsFactory( ... )  # implemented by me

hpo_config = {
    'optuna': dict(
         n_trials=args.n_trials,
    ),
    'pipeline': dict(
        training=training,   # here I pass the factory object object instead of a path, this seems to cause the issue
        testing=testing,  # here I pass the factory object object instead of a path, this seems to cause the issue
        validation=validation,  # here I pass the factory object object instead of a path, this seems to cause the issue
        ...  # more hpo settings
    )
}

# do hyperparameter optimization; evaluate the various configs on validation set
hpo_result = hpo_pipeline_from_config(hpo_config)

# get best config from hyperparameter optimization
best_cfg = hpo_result._get_best_study_config()

# for some reason, pykeen forgets to set the triples factory in the results 
# so I set them manually - this solves the problem
best_cfg['pipeline']['training'] = training
best_cfg['pipeline']['testing'] = testing
best_cfg['pipeline']['validation'] = validation

# train model with best hyperparameter config; test it on test set
result = pipeline_from_config(best_cfg)

```