ASR model not accepting audio argument for direct transcription of numpy or PyTorch tensor
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I am trying to use the NeMo Canary ASR model for transcribing audio in memory.
However when i try to use the audio argument of the class like it is documented here: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/results.html#transcribing-inference

i get the error `Error while processing audio: EncDecMultiTaskModel.transcribe() got an unexpected keyword argument 'audio'`

and looking at the code, it seems it expects a list of files or input manifest and nothing else.


**Steps/Code to reproduce bug**
```py
import numpy as np
from nemo.collections.asr.models import EncDecMultiTaskModel

canary_model = EncDecMultiTaskModel.from_pretrained('nvidia/canary-1b')

sample_rate = 16000
duration = 1
audio = np.random.randint(-32768, 32768, size=(sample_rate * duration), dtype=np.int16)

predicted_text = canary_model.transcribe(
    audio=audio,
    batch_size=16,  # batch size to run the inference with
)
```


**Expected behavior**

Transcribing correctly without error or having to write files

**Environment overview (please complete the following information)**

 - Environment location: Local windows PC
 - Method of NeMo install: pip install git+https://github.com/NVIDIA/NeMo.git@r1.23.0#egg=nemo_toolkit[asr]

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Windows 11
- PyTorch version: 2.3.0
- Python version: 3.11


I'm in the same situation.  I get : 

```
    transcript = asr_model.transcribe(batch, batch_size=1)
  File "/home/ubuntu/miniconda3/envs/nemo202403/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/miniconda3/envs/nemo202403/lib/python3.10/site-packages/nemo/collections/asr/models/rnnt_models.py", line 286, in transcribe
    fp.write(json.dumps(entry) + '\n')
  File "/home/ubuntu/miniconda3/envs/nemo202403/lib/python3.10/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/home/ubuntu/miniconda3/envs/nemo202403/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/ubuntu/miniconda3/envs/nemo202403/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/ubuntu/miniconda3/envs/nemo202403/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Tensor is not JSON serializable
```


@jprobichaud

The only way i got it working is by writing temporary files i can use. (json manifest + audio file)

You can have a look at my code here:
https://github.com/Sharrnah/whispering/blob/main/Models/STT/nemo_canary.py#L189


Canary at this moment does not support audio as a tensor. We are working on improving this support in the coming release, but as the model is new, it will take a little bit of time to mature to full support of all of ASR domains features 

Fyi @pzelasko @krishnacpuvvada 

@titu1994 Do other models also not support ndarray or tensor as inputs currently?

Mlat other ASR models should support it, Multi Task models are still being developed 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.