torch 1.8.0 incompatibility: pytorch-lightening needs to be updated to 1.2.3, but is not enough.
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Due to `torch` 1.8.0 just was released a few days ago, `pytorch-lightning` is still incompatible with the new torch.
Since NeMo setup pulls the latest stable torch release, imports this problem.
The problem is not visible during the install, but when using `pytorch-lightning` within NeMo tutorials for example.
 
**Steps/Code to reproduce bug**

I installed NeMo from r1.0.0rc17
```
python -m pip install git+https://github.com/NVIDIA/NeMo.git@r1.0.0rc1#egg=nemo_toolkit[all]
```
It was working while the official stable pytorch was 1.7.1.
Since PyTorch 1.8.0 came out, in fresh venv NeMo's requirements are pulling the latest torch, which is 1.8.0,
With torch 1.8.0 the second ASR tutorial notebook fails while using `pytorch-lightning` because `torchtext.data.Batch` was deprecated (and moved to` torchtext.legacy.data.Batch`, which breaks PyTorch Lightning. For more information read: https://github.com/PyTorchLightning/pytorch-lightning/pull/6168

**Expected behavior**

Since the bug is indirectly caused by the incompatibility between the current `pytorch-lightning` installed by NeMo and the latest `torch` 1,8.0, the only temporary solution I see that NeMo should stick with `torch<=1.7.1` and it's associated libraries.
I tried to edit the requirements by myself, then `./reinstall.sh`, unfortunately something is still pulling 1.8.0 somewhere at the end of the setup process.
The expected behavior is to be able to stick to torch==1.7.1

**Environment overview (please complete the following information)**

 - Environment location: WSL2 with GPU support and Ubuntu 20.04

**Environment details**

- OS version: Ubuntu 20.04 running in WSL2 with GPU support.
- PyTorch version: 1.8.0
- Python version: 3.8.5

**Additional context**

I think, I completely described the problem and the possible solution.


PTL was recently fixed this problem here: https://github.com/PyTorchLightning/pytorch-lightning/commit/60d0c95e07b8edfa22f5892ebb72d93b1d35832f 
and they released `pytorch-lightening` 1.2.3 .

Therefore I updated in `requirements/requirements.txt`
```
-pytorch-lightning>=1.1.0,<=1.1.5
+pytorch-lightning>=1.2.3,<=1.2.3
```
Then I made a fresh new `venv` and installed NeMo from `r1.0.0rc1` with `./reinstall.sh`

That problem now related to `pytorch-lightening` version incompatibility with `torch` 1.8.0 has been solved.
Then I start the first tutorial, in the following cell, now it crashes with another problem.

```
quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained('QuartzNet15x5Base-En')
[NeMo I 2021-03-09 22:19:02 cloud:56] Found existing object /home/tibor/.cache/torch/NeMo/NeMo_1.0.0rc1/QuartzNet15x5Base-En/00869f9c89b8393ca3de640e0c536bd2/QuartzNet15x5Base-En.nemo.
[NeMo I 2021-03-09 22:19:02 cloud:62] Re-using file from: /home/tibor/.cache/torch/NeMo/NeMo_1.0.0rc1/QuartzNet15x5Base-En/00869f9c89b8393ca3de640e0c536bd2/QuartzNet15x5Base-En.nemo
[NeMo I 2021-03-09 22:19:02 common:615] Instantiating model from pre-trained checkpoint
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-6-38ceefde5424> in <module>
----> 1 quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained('QuartzNet15x5Base-En')

~/asr/NeMo/nemo/core/classes/common.py in from_pretrained(cls, model_name, refresh_cache, override_config_path, map_location, strict, return_config)
    616         if class_ is None:
    617             class_ = cls
--> 618         instance = class_.restore_from(
    619             restore_path=nemo_model_file_in_cache,
    620             override_config_path=override_config_path,

~/asr/NeMo/nemo/core/classes/modelPT.py in restore_from(cls, restore_path, override_config_path, map_location, strict, return_config)
    417         global _MODEL_RESTORE_PATH
    418         _MODEL_RESTORE_PATH = os.path.abspath(os.path.expanduser(restore_path))
--> 419         return cls._default_restore_from(restore_path, override_config_path, map_location, strict, return_config)
    420 
    421     @classmethod

~/asr/NeMo/nemo/core/classes/modelPT.py in _default_restore_from(cls, restore_path, override_config_path, map_location, strict, return_config)
    370                     model_weights = path.join(tmpdir, _MODEL_WEIGHTS)
    371                     OmegaConf.set_struct(conf, True)
--> 372                     instance = cls.from_config_dict(config=conf)
    373                     instance = instance.to(map_location)
    374                     instance.load_state_dict(torch.load(model_weights, map_location=map_location), strict=strict)

~/asr/NeMo/nemo/core/classes/common.py in from_config_dict(cls, config)
    433         else:
    434             # models are handled differently for now
--> 435             instance = cls(cfg=config)
    436 
    437         if not hasattr(instance, '_cfg'):

~/asr/NeMo/nemo/collections/asr/models/ctc_models.py in __init__(self, cfg, trainer)
     87     def __init__(self, cfg: DictConfig, trainer: Trainer = None):
     88         # Get global rank and total number of GPU workers for IterableDataset partitioning, if applicable
---> 89         self.global_rank = 0
     90         self.world_size = 1
     91         self.local_rank = 0

~/venv/pt8nemo/lib/python3.8/site-packages/torch/nn/modules/module.py in __setattr__(self, name, value)
    993                     buffers[name] = value
    994                 else:
--> 995                     object.__setattr__(self, name, value)
    996 
    997     def __delattr__(self, name):

AttributeError: can't set attribute
```

This looks like now the model is not compatible with `torch` 1.8.0.

Therefore, I can still continue my work only if I apply https://github.com/NVIDIA/NeMo/pull/1869
Looks like, it will take some time and effort to upgrade NeMo to use torch 1.8.0.

A temporary workaround for myself it is here https://github.com/NVIDIA/NeMo/pull/1869

@tiborkiss Please note that NeMo RC1 does not work on lightning 1.2.0. In fact, RC1 pins Lightning <= 1.1.5. If you need Lightning 1.2.0 or high, please use the NeMo main branch

Thank you for the tips @blisc .
Looks like the shortest workaround is to stick with torch 1.7.1

In the setup.py we have a section related to `SUPPORTED_TORCH_VERSION` where is the `REMOVE AFTER 21.03 Container is released !` comment. 
At least until this container is released it works, therefore I close this PR.

@tiborkiss Sorry, what do you mean? Did you find a fix? Thanks!

> @tiborkiss Sorry, what do you mean? Did you find a fix? Thanks!

Yes, in this branch, this file it does that fix. https://github.com/NVIDIA/NeMo/blob/r1.0.0rc1/setup.py
As soon as is not removed (read the comments there), it is enough. I just use this branch and it works.