Repository: pykeen/pykeen
Issue: Support more init arguments and offline mode for WANDB
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
It is currently not possible to run hyperparameter searches with wandb (which requires `wandb.init(reinit=True, ...)` ) or run experiments in offline mode.

**Describe the solution you'd like**
I have the small adjustments here and can create a pull request: https://github.com/pykeen/pykeen/compare/master...kantholtz:wandb-extension

Is there no better way to set wandb to be in dry run mode other than by setting the environment variable? It seems pretty janky given the other arguments can be passed to init()

Yep, seems janky - I agree. But it's both documented that way (https://docs.wandb.com/library/technical-faq#can-i-run-wandb-offline) and pytorch-lightning does it this way also: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/loggers/wandb.py#L121

I have not read the wandb source code though - maybe there is another way?


Only found this (quick glance): https://github.com/wandb/client/blob/abc508579c61339dde51ca58bde602bb2bba224f/wandb/sdk/wandb_settings.py#L323

I don't think there is another way.

Thanks for the quick reply. I'd appreciate if you could answer one more question: why would you want to do this? To test that logging to wandb works properly? 

If you're in the pykeen world then the dummy logger basically serves as the test system. If it's something else, then I'd be happy to know, and add to the wandb tutorial before accepting the PR

Also @mali-git @migalkin I'd appreciate if you could weigh in too, being the wandb pykeen masters :)

> Also @mali-git @migalkin I'd appreciate if you could weigh in too, being the wandb pykeen masters :)

There is indeed no better way (as of now) to force a dryrun other than the env variable. The PR would be handy I think

First of all thank you very much for the lib and the rapid response!

It is actually a combination of multiple reasons (and maybe I have not understood pykeen good enough and did mistakes):
1. I tried running hpo and that was not possible because `reinit=True` needs to be set (otherwise, wandb assumes multiple trials to be one model - which does not work)
2. Then I tried running hpo with custom data, which I was not able to do. As far as I can see from the source code, this is not possible [at the moment](https://github.com/pykeen/pykeen/blob/ad15d9f64b01c99ae4c7b32cf51ddf7cb98ed060/src/pykeen/hpo/hpo.py#L576) (?)
3. At that point I gave up (after trying to ninja-register my custom dataset to the internals of pykeen (wrong approach?)). I decided that it would be much easier for me to simply re-implement the pipeline my way (which is better for me anyway because pykeen is one of multiple model libraries I use in the same project) and re-use my own infrastructure. There I control with a simple "offline" flag whether I want to upload wandb data. This is practical for prototyping where I want to quickly control via cli if a run should appear in my dashboard or not.

@migalkin thanks for the feedback, this is great.

@kantholtz for your points:

1. It looks like we weren't general enough with the handling of `result_tracker_kwargs` for WANDB. I think the code you wrote takes care of it nicely!
2. Will be addressed in https://github.com/pykeen/pykeen/pull/54 for bring your own data in HPO. This is long standing, and I will have to take care of it soon :)
3. I just made a PR #115 to enable future ninjas, but there are already a lot of documentation on bringing your own data for other parts of the pipeline, and this will most likely be superseded by #54 for your use case

I think we'd be happy to accept your updates basically as is. Please go ahead and make a PR!

Alright, thank you.

I have a small suggestion before doing the PR: I'd like to save the return value of `wandb.init` as a property (e.g. `.run`). This allows to get the properties wandb assigns automatically (e.g. the run's name). It would not interfere with any other result trackers, but it comes in handy for people who use the wandb tracker exclusively. If that's okay with you I would then add this to the PR.

I'm looking forward to #54 :)

> I'm looking forward to #54 :)

JSYK @kantholtz  this is all merged and pushed to PyPI for version 1.0.5