NeMo 0.10.1, jasper_eval -> TypeError: only integer tensors of a single element can be converted...
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
Using environment - Docker image  nvcr.io/nvidia/nemo:v0.10 with additionally installed CTC DECODERS (https://github.com/NVIDIA/OpenSeq2Seq).

By this tutorial:
https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/v0.10.1/asr/tutorial.html

running jasper_eval.py leads to the error:

File "/workspace/nemo/examples/asr/jasper_eval.py", line 158, in main
logprob.append(batch[j][: evaluated_tensors[4][i][j], :].cpu().numpy())
TypeError: only integer tensors of a single element can be converted to an index

What am i doing wrong?
Is there a Docker image with fully preinstalled libs (CTC DECODERS, etc.) to run the examples?

Full Traceback:
root@a44a85ffe228:/workspace# python /workspace/nemo/examples/asr/jasper_eval.py --model_config=/workspace/model/quartznet15x5_ru.yaml --eval_datasets=/workspace/dataset/test_quartznetru.json --load_dir=/workspace/model --lm_path=/workspace/model/lm.binary
/opt/conda/lib/python3.6/site-packages/torchaudio/backend/utils.py:54: UserWarning: "sox" backend is being deprecated. The default backend will be changed to "sox_io" backend in 0.8.0 and "sox" backend will be removed in 0.9.0. Please migrate to "sox_io" backend. Please refer to https://github.com/pytorch/audio/issues/903 for the detail.
  '"sox" backend is being deprecated. '
[NeMo I 2021-01-07 14:40:21 collections:138] Dataset loaded with 15 files totalling 0.13 hours
[NeMo I 2021-01-07 14:40:21 collections:139] 0 files were filtered totalling 0.00 hours
[NeMo I 2021-01-07 14:40:21 jasper_eval:117] Evaluating 15 examples
[NeMo I 2021-01-07 14:40:21 features:144] PADDING: 16
[NeMo I 2021-01-07 14:40:21 features:152] STFT using conv
[NeMo I 2021-01-07 14:40:25 jasper_eval:130] ================================
[NeMo I 2021-01-07 14:40:25 jasper_eval:131] Number of parameters in encoder: 18894656
[NeMo I 2021-01-07 14:40:25 jasper_eval:132] Number of parameters in decoder: 36900
[NeMo I 2021-01-07 14:40:25 jasper_eval:133] Total number of parameters in model: 18931556
[NeMo I 2021-01-07 14:40:25 jasper_eval:134] ================================
[NeMo I 2021-01-07 14:40:25 actions:1454] Restoring JasperEncoder from /workspace/model/JasperEncoder-STEP-10150.pt
[NeMo I 2021-01-07 14:40:25 actions:1454] Restoring JasperDecoderForCTC from /workspace/model/JasperDecoderForCTC-STEP-10150.pt
Selected optimization level O1:  Insert automatic casts around Pytorch functions and Tensor methods.

Defaults for this optimization level are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Warning:  multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback.  Original ImportError was: ImportError('/opt/conda/lib/python3.6/site-packages/amp_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN3c10lsERSoNS_12TensorTypeIdE',)
[NeMo I 2021-01-07 14:40:25 actions:728] Evaluating batch 0 out of 1
[NeMo I 2021-01-07 14:40:25 jasper_eval:152] Greedy WER inf%
Traceback (most recent call last):
  File "/workspace/nemo/examples/asr/jasper_eval.py", line 207, in <module>
    main()
  File "/workspace/nemo/examples/asr/jasper_eval.py", line 158, in main
    logprob.append(batch[j][: evaluated_tensors[4][i][j], :].cpu().numpy())
TypeError: only integer tensors of a single element can be converted to an index


**Steps/Code to reproduce bug**
Ubuntu 20.04
Install Docker
Pull nvcr.io/nvidia/nemo:v0.10

Install SWIG
apt-get update
apt-get install swig -y

Install torchaudio
pip install torchaudio

Install CTC DECODERS
git clone https://github.com/NVIDIA/OpenSeq2Seq -b ctc-decoders
cd /workspace/OpenSeq2Seq/decoders
./setup.sh 

PREPARE DATASET
copy data and model inside container

**Environment overview (please complete the following information)**
nvcr.io/nvidia/nemo:v0.10

**Additional context**
NVIDIA GTX 980Ti


can you please switch to the NeMo version 1.0.0b3 or above ?

Looks like this is still an issue with 1.0.0b3, but the fix @ https://github.com/NVIDIA/NeMo/issues/1581#issuecomment-748399696
worked for me.