parakeet-tdt-1.1b unable to calculate ASR confidence scores!
State: closed | Pull request: no
Labels: name: bug

name: stale

When following the NeMo tutorial for ASR Confidence (https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/ASR_Confidence_Estimation.ipynb)

I am able to obtain confidence scores from both "parakeet-rnnt-1.1b" as well as "parakeet-ctc-0.6b". However I am unable to get the same results when using "parakeet-tdt-1.1b" and I get the following exception:

```
Transcribing:   0%|                                                                             | 0/1 [00:00<?, ?it/s]
INFO:     127.0.0.1:58692 - "POST /v1/audio/transcriptions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/middleware/cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/routing.py", line 758, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/routing.py", line 778, in app
    await route.handle(scope, receive, send)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/routing.py", line 299, in handle
    await self.app(scope, receive, send)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/routing.py", line 79, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/starlette/routing.py", line 74, in app
    response = await func(request)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/jamie/nvidia_asr/parakeet_streaming_2.0_confidence.py", line 116, in transcriptions
    transcript = transcribe(audio_path=upload_name)
  File "/home/jamie/nvidia_asr/parakeet_streaming_2.0_confidence.py", line 157, in transcribe
    transcript = transcriber.transcribe([audio_path], return_hypotheses=True)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/nemo/collections/asr/models/rnnt_models.py", line 304, in transcribe
    best_hyp, all_hyp = self.decoding.rnnt_decoder_predictions_tensor(
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/nemo/collections/asr/parts/submodules/rnnt_decoding.py", line 502, in rnnt_decoder_predictions_tensor
    hypotheses = self.compute_confidence(hypotheses)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/nemo/collections/asr/parts/submodules/rnnt_decoding.py", line 582, in compute_confidence
    [hyp.frame_confidence[ts][offset]]
IndexError: list index out of range
```

TDTs are a recent model, and not all models support all features (due to architecture or training loss). We will look into it, but for now confidence metrics are computable only on rnnt and ctc models. 

Thank you! The TDT model is very impressive, and would love to switch to it, but our application requires confidence metrics. Thanks for looking into it!

Hey NeMo  team! Just following up on using confidence scores from TDT-1.1b model. This is a key enabler for our project.

Hi @asifjahmed
We are working on it. However, please consider that TDT confidence estimation is neither formally defined nor easily supported by the NeMo design. So, my estimate for this feature is ~2 weeks.

Hey @GNroy -

Thanks very much for the update! With a timeframe in mind I can plan accordingly. Thanks and looking forward to migrating to the awesome TDT model soon.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

Hi, I saw this was just marked as completed! Great! Can you link to the commit or LMK how to use this? Thank you!

You need to add `include_duration_confidence=True` to the decoding config. @GNroy are docs updated on how to use this feature. 

TDT confidence does not require additional parameters by default. It runs just like a regular confidence estimation.
`tdt_include_duration` is an optional flag indicating that TDT confidence will be computed with consideration of duration probabilities as well.

Awesome work @GNroy ! We need to add the comment about tdt include duration flag to the docs for confidence estimation 

I did add documentation about the flag (e.g. https://github.com/NVIDIA/NeMo/blob/b1628cf231eff0ca96a94b5f840b0dcbb7f2d667/nemo/collections/asr/parts/submodules/rnnt_decoding.py#L99) and also mentioned it in the confidence tutorial.

Perfect thanks !