Repository: pykeen/pykeen
Issue: Kaggle notebooks are having trouble loading entrypoints
Root cause: Environment Incompatibility

I use the following code on colab or kaggle
```
!pip install pykeen -q
from pykeen.datasets import OpenBioLink
```
Exception occurs on the import line.
this error message pops out:
```
/usr/local/lib/python3.7/dist-packages/pykeen/datasets/__init__.py in <module>()
     75 }
     76 if not _DATASETS:
---> 77     raise RuntimeError('Datasets have been loaded with entrypoints since PyKEEN v1.0.5. Please reinstall.')
     78 
     79 #: A mapping of datasets' names to their classes

RuntimeError: Datasets have been loaded with entrypoints since PyKEEN v1.0.5. Please reinstall.
```

It would be fine if I restart the kernel such that the python environment is reloaded at least once after the package installation. However it would make kaggle commit (Save version => Save & Run all (commit)) impossible, cause I have to do the restart manually and interactively, but the commit session is not interactive.

Because of this, I can not produce a saved notebook with output file from pykeen.
The only option left is to do everything on interactive session, then download the output file manually, then upload it as a kaggle dataset.

## Solution

From https://github.com/pykeen/pykeen/issues/373#issuecomment-821060699:

> I just found the workaound. It should work on both colab and kaggle.
> ```python
>  from pkg_resources import require
>  require('pykeen')
>  ```
> https://colab.research.google.com/drive/1yFWDQ3OybultFHaNdi_gJCHpBJiB4Iem?usp=sharing


Hi @jerryIsHere , doing `pip install --upgrade` should do the trick. Colab is probably caching an old version. 

![image](https://user-images.githubusercontent.com/50871412/114839115-fc501b00-9e07-11eb-8e0e-c71d280f41f0.png)
it is already 1.4.0
I already tried `pip install --upgrade pykeen`
BTW if I remember correctly, I tried `!pip show pykeen` few days ago, the package was 1.4.0 already before I call `pip install --upgrade pykeen`

Hmm... I've never worked with notebooks in Kaggle and just realized in my previous comment I was getting confused with Google Colab. Could you give me some information on how to get this up and running so i can try myself?

I can also explain why the exception is happening: the datasets are loaded through python entrypoints, so you can add more stuff without modifying the PyKEEN code itself. PyKEEN also internally takes advantage of this for its 25+ datasets, so this exception gets raised if PyKEEN was not installed in the normal way, which should result in the entrypoints getting registered. I hate to say it, but can you try uninstalling and then reinstalling to see what happens?

Yes, but the same issue also happens on google colab too.
I was trying to search for system/utility library which can mimic the loading process of python entry-point on jupyter notebook as a workaround, but didn't find any suitable module. 
I was thinking maybe some python expert knows something related.

I have a few examples in [this folder](https://github.com/pykeen/pykeen/tree/master/notebooks) that now have links to open on colab. I was just using colab yesterday on a new notebook so that shouldn't be an issue.

I just made a notebook on Kaggle and was able to get everything working: https://www.kaggle.com/charlestapleyhoyt/pykeen-demo. Note: I had to turn the internet on, but everything else is vanilla.

Wow, really crazy. It worked then when I saved it, it re-ran and got the same error you were having (though it's now definitely obvious that it's using the latest code, since I just introduced the `pykeen.env()` function in my last commit)

I just found the workaound. It should work on both colab and kaggle.
```
from pkg_resources import require
require('pykeen')
```
https://colab.research.google.com/drive/1yFWDQ3OybultFHaNdi_gJCHpBJiB4Iem?usp=sharing



@jerryIsHere thank you very much for bringing this to my attention, and also for contributing a fix! I've just submitted a PR that updates the docs to reflect this and will give you a shout-out in the next release. Would you like me to write your name just as your github handle, or would you like to give your full name?

> @jerryIsHere thank you very much for bringing this to my attention, and also for contributing a fix! I've just submitted a PR that updates the docs to reflect this and will give you a shout-out in the next release. Would you like me to write your name just as your github handle, or would you like to give your full name?

My pleasure. Thanks for your quick and responsive support too.
Just write my name as my github handle

Hello, I am trying to get HelloWorld Google Colab to work, however I still have the same problems mentioned here and in #588. I hope it is okay, that I mention this on this issue, since the error message sent me right here.

So the error is still the same:
![Untitled](https://user-images.githubusercontent.com/14311131/143218973-910fc6ed-b7b2-4ff2-9c26-d50d6a644874.png)

Using
```python
from pkg_resources import require
require('pykeen')
```

leads to the scipy version problem mentioned in #588.

After 
```
!python3 -m pip uninstall pykeen
!python3 -m pip install pykeen
```
the above error persists and the outputs of the environment info is:
```
!pip show pykeen
Name: pykeen
Version: 1.6.0
Summary: A package for training and evaluating multimodal knowledge graph embeddings
Home-page: https://github.com/pykeen/pykeen
Author: "Mehdi Ali, Max Berrendorf, Charles Tapley Hoyt, Laurent Vermue"
Author-email: pykeen2019@gmail.com
License: MIT
Location: /usr/local/lib/python3.7/dist-packages
Requires: torch, dataclasses-json, class-resolver, requests, tqdm, sklearn, docdata, scipy, optuna, tabulate, more-click, pystow, click-default-group, click, numpy, pandas
Required-by: 
```

```
>>> pykeen.env()
Key | Value
-- | --
OS | posix
Platform | Linux
Release | 5.4.104+
Time | Wed Nov 24 10:09:27 2021
Python | 3.7.12
PyKEEN | 1.6.1-dev
PyKEEN Hash | UNHASHED
PyKEEN Branch |  
PyTorch | 1.10.0+cu111
CUDA Available? | false
CUDA Version | 11.1
cuDNN Version | 8005


```

@dobraczka can you check the version of scipy you have installed?

Scipy is at 1.7.2 in all cases. Which makes it weird that `require('pykeen')` is complaining about that.

I was able to reproduce this issue... `pip freeze` shows the new version of Scipy. I have no idea why the `pkg_resources` is getting the old version :/

Hi there, I'm running into the same issues as described above on both Colab and Kaggle. Interestingly, the version conflict I get is different from each on Colab I get a complaint regarding scipy, even though `pip freeze` is showing I have the latest installed. In the Kaggle notebook, the complaint seems to be with pytorch versions. I know in both instances Colab and Kaggle notebooks come with a set of cached preinstalled packages, it's a really strange bug and I've never encountered it with either notebook provider before. I'm not sure if there is a way to run a notebook with a completely clean env from the start. 

Colab:

```python
---------------------------------------------------------------------------

ContextualVersionConflict                 Traceback (most recent call last)

<ipython-input-2-23fa38cc61db> in <module>()
      1 from pkg_resources import require
----> 2 require('pykeen')

1 frames

/usr/local/lib/python3.7/dist-packages/pkg_resources/__init__.py in resolve(self, requirements, env, installer, replace_conflicting, extras)
    775                 # Oops, the "best" so far conflicts with a dependency
    776                 dependent_req = required_by[req]
--> 777                 raise VersionConflict(dist, req).with_context(dependent_req)
    778 
    779             # push the new requirements onto the stack

ContextualVersionConflict: (scipy 1.4.1 (/usr/local/lib/python3.7/dist-packages), Requirement.parse('scipy>=1.5.0'), {'pykeen'})
```
Kaggle 
```python
---------------------------------------------------------------------------
ContextualVersionConflict                 Traceback (most recent call last)
/tmp/ipykernel_54/571153087.py in <module>
      1 from pkg_resources import require
----> 2 require('pykeen')

/opt/conda/lib/python3.7/site-packages/pkg_resources/__init__.py in require(self, *requirements)
    889         included, even if they were already activated in this working set.
    890         """
--> 891         needed = self.resolve(parse_requirements(requirements))
    892 
    893         for dist in needed:

/opt/conda/lib/python3.7/site-packages/pkg_resources/__init__.py in resolve(self, requirements, env, installer, replace_conflicting, extras)
    780                 # Oops, the "best" so far conflicts with a dependency
    781                 dependent_req = required_by[req]
--> 782                 raise VersionConflict(dist, req).with_context(dependent_req)
    783 
    784             # push the new requirements onto the stack

ContextualVersionConflict: (torch 1.9.1+cpu (/opt/conda/lib/python3.7/site-packages), Requirement.parse('torch>=1.10; platform_system != "Windows"'), {'pykeen'})```

> Hi there, I'm running into the same issues as described above on both Colab and Kaggle. Interestingly, the version conflict I get is different from each on Colab I get a complaint regarding scipy, even though `pip freeze` is showing I have the latest installed. In the Kaggle notebook, the complaint seems to be with pytorch versions. I know in both instances Colab and Kaggle notebooks come with a set of cached preinstalled packages, it's a really strange bug and I've never encountered it with either notebook provider before. I'm not sure if there is a way to run a notebook with a completely clean env from the start.
> 
> Colab:
> 
> ```python
> ---------------------------------------------------------------------------
> 
> ContextualVersionConflict                 Traceback (most recent call last)
> 
> <ipython-input-2-23fa38cc61db> in <module>()
>       1 from pkg_resources import require
> ----> 2 require('pykeen')
> 
> 1 frames
> 
> /usr/local/lib/python3.7/dist-packages/pkg_resources/__init__.py in resolve(self, requirements, env, installer, replace_conflicting, extras)
>     775                 # Oops, the "best" so far conflicts with a dependency
>     776                 dependent_req = required_by[req]
> --> 777                 raise VersionConflict(dist, req).with_context(dependent_req)
>     778 
>     779             # push the new requirements onto the stack
> 
> ContextualVersionConflict: (scipy 1.4.1 (/usr/local/lib/python3.7/dist-packages), Requirement.parse('scipy>=1.5.0'), {'pykeen'})
> ```
> 
> Kaggle
> 
> ```python
> ---------------------------------------------------------------------------
> ContextualVersionConflict                 Traceback (most recent call last)
> /tmp/ipykernel_54/571153087.py in <module>
>       1 from pkg_resources import require
> ----> 2 require('pykeen')
> 
> /opt/conda/lib/python3.7/site-packages/pkg_resources/__init__.py in require(self, *requirements)
>     889         included, even if they were already activated in this working set.
>     890         """
> --> 891         needed = self.resolve(parse_requirements(requirements))
>     892 
>     893         for dist in needed:
> 
> /opt/conda/lib/python3.7/site-packages/pkg_resources/__init__.py in resolve(self, requirements, env, installer, replace_conflicting, extras)
>     780                 # Oops, the "best" so far conflicts with a dependency
>     781                 dependent_req = required_by[req]
> --> 782                 raise VersionConflict(dist, req).with_context(dependent_req)
>     783 
>     784             # push the new requirements onto the stack
> 
> ContextualVersionConflict: (torch 1.9.1+cpu (/opt/conda/lib/python3.7/site-packages), Requirement.parse('torch>=1.10; platform_system != "Windows"'), {'pykeen'})```
> ```

Also seeing this issue in Colab.