Repository: pykeen/pykeen
Issue: Getting random dataset partition despite setting random seed
Root cause: Validation Gap

Hello,

When I try to use the Hetionet dataset, even if I set a random seed, I get different train-validation-test partition everytime I initialize a new instance. Is it normal?

For example, if I do this, I will get different result between runs.

```python
from pykeen.datasets import Hetionet
test1 = Hetionet(random_state=42)
testset_entry = test1.testing.mapped_triples
testset_entry = testset_entry.detach().cpu().numpy()
np.sum(testset_entry[:,1]==8)
```



No, this is not desired. Here's a more reproducible way to test this, which confirms the issue:

```python
a = Hetionet(random_state=42)
b = Hetionet(random_state=42)
testing_a = a.testing.mapped_triples.detach().numpy()
testing_b = b.testing.mapped_triples.detach().numpy()
assert (testing_a == testing_b).all()
```

Can you please share your version information by pasting the output of `pykeen version` here?

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | nt                       |
| Platform        | Windows                  |
| Release         | 10                       |
| User            | *****|
| Time            | Tue Jun 22 16:12:07 2021 |
| Python          | 3.7.4                    |
| PyKEEN          | 1.5.0                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.8.1+cpu                |
| CUDA Available? | false                    |
| CUDA Version    | N/A                      |
| cuDNN Version   | N/A                      |


@pitehu thanks again for pointing this out. I've already started a PR #500 with a unit test that will pass once the code is fixed. I'll let you know when it's done :)

@cthoyt I don't know if it helps but on colab the random seed behaves as expected, somehow. 
Here's the environment. On my local machine, even if I switch to the same version of everything, it still doesn't work
<html>
<body>
<!--StartFragment-->

Key | Value
-- | --
OS | posix
Platform | Linux
Release | 5.4.104+
User | root
Time | Tue Jun 22 15:20:56 2021
Python | 3.7.10
PyKEEN | 1.5.1-dev
PyKEEN Hash | UNHASHED
PyKEEN Branch |  
PyTorch | 1.9.0+cu102
CUDA Available? | false
CUDA Version | 10.2
cuDNN Version | 7605

<!--EndFragment-->
</body>
</html>

This should be solved with #500. @pitehu please let me know if the issue persists for you. Thanks for being patient!

Thanks you very much! I will let you know if I still find issues.


From: Charles Tapley ***@***.***>
Sent: Tuesday, November 9, 2021 3:20 PM
To: ***@***.***>
Cc: Tiancheng ***@***.***>; ***@***.***>
Subject: Re: [pykeen/pykeen] Getting random dataset partition despite setting random seed (#499)


This should be solved with #500<https://github.com/pykeen/pykeen/pull/500>. @pitehu<https://github.com/pitehu> please let me know if the issue persists for you. Thanks for being patient!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/pykeen/pykeen/issues/499#issuecomment-964194643>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHIEFNAZUMTDRG6I3SZAQCLULEUZNANCNFSM47DUA73Q>.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.