zhiminy/EvalEng huggingface dataset:
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211129+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211128+cu102

Base PyTorch commit: b10929a14a32c90ac5943cc3f27a3a9105e6f53f

Affected PyTorch version: 1.11.0.dev20211129+cu102

Affected PyTorch commit: 6ae34ea6f8b65d2ad36e417c3d525507901cfd7f

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +7.40500%
- test_eval[timm_vovnet-cpu-eager]: -10.63166%


cc @xuzhao9


Bisection returns empty result. Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211130+cu102'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211129+cu102

Base PyTorch commit: 6ae34ea6f8b65d2ad36e417c3d525507901cfd7f

Affected PyTorch version: 1.11.0.dev20211130+cu102

Affected PyTorch commit: 1342f19a8c2d92ed86bc32a0b3a6fc9b35b3e3e3

Affected Tests:
- test_eval[timm_vovnet-cpu-eager]: +12.27904%


cc @xuzhao9


Bisection returns empty result. Closed.
Repository: pytorch/benchmark
Issue: Densenet121: Training with batch size 256 will encounter CUDA OOM on CI's Tesla T4
Root cause: Unimplemented Feature Gap

In the original paper, densenet121 is trained with a batch size of 256 on the ImageNet dataset. They are also training densenet121 on the TitanX GPU with 12GB device memory. However CI's Tesla T4 gets out of memory with 16GB device memory.
Paper: https://arxiv.org/pdf/1608.06993.pdf

In the original repo, DenseNet has an option `-optMemory 4` which significantly reduces the memory footprint and is able to train on TitanX. TorchVision's version of densenet121 has memory_efficient option, that still fails to run on T4 (16GB mem).

Repo: https://github.com/liuzhuang13/DenseNet#memory-efficient-implementation-newly-added-feature-on-june-6-2017
Densenet121 Torchvision: https://github.com/pytorch/vision/blob/d367a01a18a3ae6bee13d8be3b63fd6a581ea46f/torchvision/models/densenet.py#L162

Densenet121 train/example on T4 fails with CUDA OOM:
```
(py37) aaronshi@pytorch-benchmark-dev-0:~/data/as-use/benchmark$ python test.py -k densenet121 --verbose
test_densenet121_check_device_cpu (__main__.TestBenchmark) ... ok
test_densenet121_check_device_cuda (__main__.TestBenchmark) ... ok
test_densenet121_eval_cpu (__main__.TestBenchmark) ... ok
test_densenet121_eval_cuda (__main__.TestBenchmark) ... ok
test_densenet121_example_cpu (__main__.TestBenchmark) ... ok
test_densenet121_example_cuda (__main__.TestBenchmark) ... ERROR
test_densenet121_train_cpu (__main__.TestBenchmark) ... ok
test_densenet121_train_cuda (__main__.TestBenchmark) ... ERROR
```

Torchvision authors may want to implement `-optMemory 4` optimization to allow training on a single device.

Hi @fmassa, author of memory efficient densenet121 in torchvision. Do you know if anyone would be interested in taking a look at implementing this `-optMemory 4` feature to torchvision's densenet121? 

For reference, Memory Efficient PR: https://github.com/pytorch/vision/pull/1003

Fixed by https://github.com/pytorch/benchmark/pull/781
Repository: pytorch/benchmark
Issue: opacus_cifar10 model is broken because of 1.0 API change
Root cause: External Dependency Breakage

Error message:
`ModuleNotFoundError: No module named 'opacus.utils.module_modification'`

caused by https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/opacus_cifar10/__init__.py#L5.

Also, the `PrivacyEngine` class is not accepting `batch_size` argument anymore on 1.0.0:
https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/opacus_cifar10/__init__.py#L37

Context: [Opacus 1.0 API change](https://github.com/pytorch/opacus/blob/main/Migration_Guide.md)
Repository: pytorch/benchmark
Issue: tacotron2: train/eval/example disabled on CUDA
Root cause: Resource Mishandling

After aligning the tacotron2's parameters to match with the source paper and source code (such as batch size), the tests for train, eval, and example run into `RuntimeError: CUDA out of memory.` due to CI's Tesla T4 GPU.

Aligned with Source: https://github.com/pytorch/benchmark/pull/610

Failed Tests due to `RuntimeError: CUDA out of memory.`:
- test_tacotron2_train_cuda
- test_tacotron2_example_cuda
- test_tacotron2_eval_cuda

TODO: Re-enable these tests once the CI upgrades the GPU. 

Fixed by https://github.com/pytorch/benchmark/pull/781
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211206+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211205+cu113

Base PyTorch commit: 6a4fa860262951ca14db0c53fc7d60c3dbc48af4

Affected PyTorch version: 1.11.0.dev20211206+cu113

Affected PyTorch commit: 6a4fa860262951ca14db0c53fc7d60c3dbc48af4

Affected Tests:
- test_eval[timm_efficientnet-cpu-jit]: -9.70146%


cc @xuzhao9


Two pytorch versions are from the same commit: ignore this noise event.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211206+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211205+cu113

Base PyTorch commit: 6a4fa860262951ca14db0c53fc7d60c3dbc48af4

Affected PyTorch version: 1.11.0.dev20211206+cu113

Affected PyTorch commit: 6a4fa860262951ca14db0c53fc7d60c3dbc48af4

Affected Tests:
- test_eval[timm_efficientnet-cpu-jit]: -8.62954%
- test_eval[yolov3-cpu-eager]: +18.53711%


cc @xuzhao9


Noise. Closed this issue.
Repository: pytorch/benchmark
Issue: The test code, which tests the correctness of FastRnn, allows small difference between the output of FastRnn and the output of cudnn's rnn
Root cause: Documentation Deficiency

Hi, 

First, I want to thank you guys for sharing the code. I learned a lot from it. It gives me a good start for implement my own GRU layer.   :-)

One thing that I am very confusing is the outputs from cudnn's lstm and the fastrnn's lstm are different. Based on the assertions in [your test code](https://github.com/pytorch/benchmark/blob/877113367236968470e5694a4f1343e5654c11bd/rnns/fastrnns/custom_lstms.py#L398-L400), I guess it is a common case which I do not need to worry too much. But, I am curious what cause the difference. 

Thanks for your time :-)

Closed as they are not relevant anymore.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211208+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211206+cu113

Base PyTorch commit: 6a4fa860262951ca14db0c53fc7d60c3dbc48af4

Affected PyTorch version: 1.11.0.dev20211208+cu113

Affected PyTorch commit: bd8a4a937216433b281a333905175214a12f5fc1

Affected Tests:
- test_eval[yolov3-cpu-eager]: -15.72157%


cc @xuzhao9


Confirmed to be noise, closing this issue.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211209+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211208+cu113

Base PyTorch commit: bd8a4a937216433b281a333905175214a12f5fc1

Affected PyTorch version: 1.11.0.dev20211209+cu113

Affected PyTorch commit: 8b20dde93240642b3fce14b304e2d5e6d09d9891

Affected Tests:
- test_eval[densenet121-cpu-eager]: +9.64176%


cc @xuzhao9


Confirmed to be noise. Closed.
Repository: pytorch/benchmark
Issue: TODO: Update test_bench.py and test.py to use the same EXCLUDELIST to skip tests
Root cause: Architectural Constraint

Create a central EXCLUDELIST that contains all the tests that need to be skipped, and share the same list between test.py and test_bench.py.

At the moment, we only have the list in test.py. This will allow us to have a clear list of skipped tests.

Fixed by https://github.com/pytorch/benchmark/pull/781
Repository: pytorch/benchmark
Issue: Add fairseq roberta model
Root cause: Unimplemented Feature Gap

We can add more NLP models through fairseq: https://github.com/pytorch/fairseq. Roberta is a good candidate (https://github.com/pytorch/fairseq/tree/main/examples/roberta)
The benchmark could use a minimal/synthesized GLUE benchmark task dataset

@xuzhao9 Just want to confirm that this is still a desired addition?

@joecummings Actually this can be closed since we recently added fambench_xlmr in https://github.com/pytorch/benchmark/pull/835. Thanks for the reminder though.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211214+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211213+cu113

Base PyTorch commit: e948856ce7ead632483e35714a9a27ea2fa68ee7

Affected PyTorch version: 1.11.0.dev20211214+cu113

Affected PyTorch commit: 3d358a76783acef87f282cc0fb20818eeb4c133c

Affected Tests:
- test_eval[shufflenet_v2_x1_0-cpu-jit]: -9.58774%


cc @xuzhao9


Root cause commit: https://github.com/pytorch/pytorch/commit/3d358a7678, trying to confirm.

Confirmed speedup
Repository: pytorch/benchmark
Issue: nits for running benchmark models locally
Root cause: Validation Gap

moved from original issue in https://github.com/pytorch/hub/issues/148 on behalf of @zdevito 

I am working on python packaging for PyTorch and just used the benchmark models to verify that the packaging approach would work and not get caught up in the complexity of existing models. I used the ability to loop through the models to get a handle on the torch.nn.Module for each one, saved it in the package, and reloaded it. It illustrated a lot of shortcomings of my initial code and I was able to quickly try fixes and see how they would work. Pretty cool! I think the benchmark suite is going to be really useful for these type of design questions in addition to purely perf improvements. Thanks for helping to put it together.

As part of the process of using the benchmarks, I ran into a few nits in some of the benchmarks that only got uncovered when trying to use things locally.

Couldn't easily work around
Background-Matting - does not work on my local machine

- [x]  has hard-coded circleci paths,
- [x]  expects local directory to be a specific value

tacotron2

- [x]  requires a GPU even if you ask for a cpu model, because it calls .cuda in load_model()

Require workarounds
Overall

 

- [x] the use of sys.path modifications to load files means that error messages are confusing:

e.g. (File "hubconf.py", line 74, in __init__, which hubconf.py is that?). Would be to treat models/ as part of the path and to load the submodules from there.
BERT-pytorch

- [x]  expects local directory to be a specific value

attention-is-all-you-need-pytorch

- [x] expects local directory to be a specific value

fastNLP

- [x] expects local directory to be a specific value

demucs

 

- [x] get_module does not return a torch.nn.Module (returns a lambda)
- [x] doesn't do anything with the jit flag (should throw if it is not supported)
- [x] puts ScriptModule annotation on model, but doesn't actually script the model

moco

- [x] default device is set to 'cuda' but the runbook specifies the default device is 'cpu',

causes model to fail in unexpected way when cuda is not installed
Repository: pytorch/benchmark
Issue: Enable train tests when CI infra supports better GPU
Root cause: Resource Mishandling

We have to disable GPU train tests for a few models, because their original batch size doesn't fit into our current CI infra.

List of models that disabled train because of insufficient GPU memory:
- densenet121
- demucs
- hf_T5
- squeezenet1_1
- tacotron2
- timm_nfnet

Fixed by https://github.com/pytorch/benchmark/pull/781
Repository: pytorch/benchmark
Issue: Enable inference optimizations for models that support it
Root cause: Unimplemented Feature Gap

Some models code has support for inference specific optimization, like fp16:

https://github.com/pytorch/benchmark/blob/1ef02faa0396d8b1d9ddf56796563e59ba36627b/torchbenchmark/models/tacotron2/waveglow/inference.py#L52
https://github.com/pytorch/benchmark/blob/0954aa616e584a63d60cff6fd56ab40f0ca928a2/torchbenchmark/models/yolov3/__init__.py#L63

We should enable it `eval` settings to be representative.

Broader question is whether it's fair to enable `.half()` for models that don't have it natively. It might be considered a very common optimization so it's reasonable to do so for at least the simple ones (like torchvision ones). More intrusive optimizations aren't "fair" because real users are less likely to do so in their code unguided (and today PyTorch doesn't really guide them)

I submitted https://github.com/pytorch/benchmark/pull/660 to enable `half()` or fp16 for inference on yolov3 and tacotron2.
I see some models also support `amp`, is it the same as `half()`?

Also, I see models could enable `amp` or `half()` for training but they are usually not the default. Should we enable them for train by default as well?

Added https://github.com/pytorch/benchmark/pull/663, enabling `half()` for all torchvision models GPU inference. CPU doesn't support `half()`

Added https://github.com/pytorch/benchmark/pull/776, support amp mode for all models, but not enabled by default because of the performance and correctness concerns.

Resolved by https://github.com/pytorch/benchmark/pull/776. Now all models support `amp` and we provide option to enable `half` for models that support it.
Repository: pytorch/benchmark
Issue: Benchmark test failing due to CUDA out of memory, GPU 8GB
Root cause: Resource Mishandling

RTX 3070ti 8gb NVIDIA-SMI 460.91.03    Driver Version: 460.91.03    CUDA Version: 11.2 

`pytest test_bench.py --ignore_machine_config`

Is 8gb too small for hobbyist? 

Do we need to add the following to the script depending on GPU memory?
```
import gc
gc.collect()
torch.cuda.empty_cache()
```

In the past few months, we increases the default batch size for most of the models to stick to the following principle:
- Train batch sizes should be consistent with either the research paper, or a reference correct implementation (such as default value in github repo);
- Eval batch sizes should be the smallest int that is able to saturate the GPU on our CI platform (which uses an Nvidia T4 16GB)

This is to simulate the GPU utilization and framework overhead that occurs in real-world training/inference scenario.

So yes, unfortunately the default batch size will be too large if you are using a 8gb GPU. However, most models should now support `train_bs` and `eval_bs` arguments, which allows the framework to run the model with smaller batch sizes. We welcome user's contribution to add an option, e.g., "pytest test_bench.py --single-batch-size" to run the suite with `batch_size==1` for all the models.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211217+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211216+cu113

Base PyTorch commit: 3d358a76783acef87f282cc0fb20818eeb4c133c

Affected PyTorch version: 1.11.0.dev20211217+cu113

Affected PyTorch commit: 4a6a5d163012a81268f0db20e9fcc3a926dd4d54

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +11.01573%


cc @xuzhao9


Bisection job: https://github.com/pytorch/benchmark/actions/runs/1616495926.
Noise and empty result, closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211223+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211222+cu113

Base PyTorch commit: 6217fee96bf1dc4e1343935d7ff87053f9bf22e4

Affected PyTorch version: 1.11.0.dev20211223+cu113

Affected PyTorch commit: a421ee0e52e57e854656d40df3d8280af27ab4a0

Affected Tests:
- test_eval[timm_regnet-cpu-eager]: +8.54308%


cc @xuzhao9


Bisection job: https://github.com/pytorch/benchmark/actions/runs/1616495926
Empty result(noise)
Repository: pytorch/benchmark
Issue: update unidash/scuba 
Root cause: Unimplemented Feature Gap

- [ ] add new models
- [ ] update queries to move from hub to benchmark or include both in the same query
 

issues with fastrnns 

- [ ] circle_build_num and circle_project_name missing
- [ ] git * variables null on recent entries

issues with benchmark

- [ ] git * variables null on recent entries
- [ ] no data from learning to paint in the last few days?
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211224+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211223+cu113

Base PyTorch commit: a421ee0e52e57e854656d40df3d8280af27ab4a0

Affected PyTorch version: 1.11.0.dev20211224+cu113

Affected PyTorch commit: 795af1578ccade97792646e02e5ec3c295385501

Affected Tests:
- test_eval[densenet121-cpu-eager]: +7.90262%


cc @xuzhao9


Bisection job: https://github.com/pytorch/benchmark/actions/runs/1619807130
Empty result (noise). Closed.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211227+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211226+cu113

Base PyTorch commit: 795af1578ccade97792646e02e5ec3c295385501

Affected PyTorch version: 1.11.0.dev20211227+cu113

Affected PyTorch commit: 795af1578ccade97792646e02e5ec3c295385501

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +7.35548%


cc @xuzhao9


Bisection job: https://github.com/pytorch/benchmark/actions/runs/1627746031
Failed. Looking into it.

Two commits are the same so this is a noise.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211228+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211227+cu113

Base PyTorch commit: 795af1578ccade97792646e02e5ec3c295385501

Affected PyTorch version: 1.11.0.dev20211228+cu113

Affected PyTorch commit: 795af1578ccade97792646e02e5ec3c295385501

Affected Tests:
- test_train[hf_Bert-cpu-eager]: -7.24271%


cc @xuzhao9


Bisection job: https://github.com/pytorch/benchmark/runs/4652165514?check_suite_focus=true
Failed. Looking into it.

Two commits are the same so this is a noise signal.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20211229+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20211228+cu113

Base PyTorch commit: 795af1578ccade97792646e02e5ec3c295385501

Affected PyTorch version: 1.11.0.dev20211229+cu113

Affected PyTorch commit: a0c99a8d3bce895c6bb6a220e98ca84274eb33dd

Affected Tests:
- test_eval[drq-cuda-eager]: +8.27124%


cc @xuzhao9


Bisection job: https://github.com/pytorch/benchmark/actions/runs/1634817119
Empty result (noise), closing it.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20220102+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220101+cu113

Base PyTorch commit: ce86881afadc0fea628c7e47d64a4073f3e09894

Affected PyTorch version: 1.11.0.dev20220102+cu113

Affected PyTorch commit: ce86881afadc0fea628c7e47d64a4073f3e09894

Affected Tests:
- test_train[hf_Bert-cpu-eager]: +8.26311%


cc @xuzhao9


Bisection job failed (https://github.com/pytorch/benchmark/actions/runs/1646657501) Looking into it.

Two commits are the same so this is a noise.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20220103+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220102+cu113

Base PyTorch commit: ce86881afadc0fea628c7e47d64a4073f3e09894

Affected PyTorch version: 1.11.0.dev20220103+cu113

Affected PyTorch commit: fa09099ba35fcd42347732ca3a5f8ddaf145da1b

Affected Tests:
- test_eval[tts_angular-cuda-eager]: +7.54299%


cc @xuzhao9


Bisection job: https://github.com/pytorch/benchmark/actions/runs/1650287468

Bisection result is empty. Closing it.
Repository: pytorch/benchmark
Issue: niter not used consistently across benchmark suite
Root cause: Architectural Constraint

A bunch of benchmarks use `niterations`, like StarGan: https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/pytorch_stargan/__init__.py#L96

Some others don't have the attribute at all.

A few months ago, @wconstab suggested removing the `niter` arguments from all benchmarks. I like the idea and believe we should do the warmup in the framework, not the model code.

I think we should do that and make sure:
1. data is already prefetched to the device
2. in the pytest framework, warmup the test for a fixed number of iterations (e.g. 5)

i'm not super rigid about removing niter, but we should definitely make it consistent at minimum.  I have fixed a few niterations->niter in my wconstab/ltc branch, only to realize i missed other models.

Closed #697
Repository: pytorch/benchmark
Issue: Gate CUDA failures behind some kind of flag?
Root cause: Architectural Constraint

Currently, a bunch of models error when you call `get_module()` due to CI reasons. However, if you have an A100 they run perfectly fine. So, it's a bit annoying to need to modify the benchmarks a ton just to get a model that should work on your GPU.

Good point, I think we should have an abstract layer to determine what tests (and configurations) are available given the current user platform.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20220110+cu113'
Root cause: Others


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220108+cu113

Base PyTorch commit: 0eb2fc608c0ea8c4150a6ba823e80a52e4d9d158

Affected PyTorch version: 1.11.0.dev20220110+cu113

Affected PyTorch commit: 2bed616e0fb1a937b1bb1eee8e6cf83734e839bd

Affected Tests:
- test_train[hf_Bert-cpu-eager]: -9.45017%


cc @xuzhao9


Bisection run: https://github.com/pytorch/benchmark/actions/runs/1679774388, empty result, noise.
Repository: pytorch/benchmark
Issue: Performance Signal Detected by TorchBench CI on '1.11.0.dev20220110+cu113'
Root cause: Algorithmic Error


TorchBench CI has detected a performance signal.

Base PyTorch version: 1.11.0.dev20220108+cu113

Base PyTorch commit: 0eb2fc608c0ea8c4150a6ba823e80a52e4d9d158

Affected PyTorch version: 1.11.0.dev20220110+cu113

Affected PyTorch commit: 2bed616e0fb1a937b1bb1eee8e6cf83734e839bd

Affected Tests:
- test_train[hf_Bert-cpu-eager]: -7.56587%


cc @xuzhao9


Bisection run: https://github.com/pytorch/benchmark/actions/runs/1684739683, empty result, closed.
Repository: pytorch/benchmark
Issue: Support for different Python versions
Root cause: External Dependency Breakage

I wonder what's the long-term plan to support new Python versions?

As a concrete example, currently it's not possible to install benchmark on Python 3.9 as is:
demucs has a dependency on lameenc==1.2.2, which is not available on Python 3.9.

So when Python 3.9 needs to be supported, will older models that can't be run with 3.9 be just removed (or maybe updated)?
Or should there be a mechanism to only install what is supported - different set of models for different Python versions, possibly with different version of dependencies?


I believe the long-term goal is to keep models updated for the Python versions we support. For example, when we decide to support Python 3.9, we should update demucs model (or remove it, if it can't be fixed).