Prompt formatter API and canary transcribe tensor input support
State: closed | Pull request: yes
Labels: name: ASR

name: common

name: Run CICD

# What does this PR do ?

Generic prompt formatter for text modality with several out-of-the-box prompt format definitions. See the class documentation for more details.

Also, enables support for tensor/array inputs in Canary. Example snippet:

```python
import os
from tempfile import NamedTemporaryFile

import lhotse

from nemo.collections.asr.models.aed_multitask_models import EncDecMultiTaskModel


def main():
    model = EncDecMultiTaskModel.from_pretrained("nvidia/canary-1b")

    path = ...
    rec = lhotse.Recording.from_file(path)
    audio = rec.load_audio()[0]

    TARGET_LANG = "es"

    # Array list input, legacy API for Canary-1B
    results = model.transcribe(
        audio=[audio],
        batch_size=1,
        num_workers=0,
        source_lang="en",
        target_lang=TARGET_LANG,
        task="asr",
        pnc="yes",
    )

    print(results)

    # Array list input, explicit single-turn prompt   
    results = model.transcribe(
        audio=[audio],
        batch_size=1,
        num_workers=0,
        role="user",
        slots={
            "source_lang": "en",
            "target_lang": TARGET_LANG,
            "task": "asr",
            "pnc": "yes",
        },
    )

    print(results)

    # Array list input, explicit multi-turn prompt   
    results = model.transcribe(
        audio=[audio],
        batch_size=1,
        num_workers=0,
        turns=[{
            "role": "user",
            "slots": {
                "source_lang": "en",
                "target_lang": TARGET_LANG,
                "task": "asr",
                "pnc": "yes",
            },
        }],
    )

    print(results)

    # Audio path input, explicit single-turn prompt   
    results = model.transcribe(
        audio=path,
        batch_size=1,
        num_workers=0,
        role="user",
        slots={
            "source_lang": "en",
            "target_lang": TARGET_LANG,
            "task": "asr",
            "pnc": "yes",
        },
    )

    print(results)

    # Legacy JSON manifest with slot values API for Canary-1B 
    with NamedTemporaryFile("w", suffix=".json") as f:
        lhotse.serialization.save_to_jsonl(
            [
                {
                    "audio_filepath": path,
                    "text": "irrelevant",
                    "duration": rec.duration,
                    "task": "asr",
                    "pnc": "yes",
                    "source_lang": "en",
                    "target_lang": TARGET_LANG,
                }
            ],
            f.name,
        )
        f.flush()
        os.fsync(f.fileno())

        results = model.transcribe(
            audio=f.name,
            batch_size=1,
            num_workers=0,
        )

        print(results)


if __name__ == "__main__":
    main()

```

We can also now provide these values dynamically to `transcribe_speech.py` from CLI. Example:

```bash
# Legacy Canary-1B format
python ~/code/NeMo/examples/asr/transcribe_speech.py \
  audio_dir=wavs \
  output_filename=out.json \
  batch_size=1 \
  pretrained_name=nvidia/canary-1b \
  +prompt.source_lang=en \
  +prompt.target_lang=es \
  +prompt.task=asr \
  +prompt.pnc=yes

# Explicit single-turn format
python ~/code/NeMo/examples/asr/transcribe_speech.py \
  audio_dir=wavs \
  output_filename=out.json \
  batch_size=1 \
  pretrained_name=nvidia/canary-1b \
  +prompt.role=user \
  +prompt.slots.source_lang=en \
  +prompt.slots.target_lang=es \
  +prompt.slots.task=asr \
  +prompt.slots.pnc=yes

# Explicit multi-turn format
python ~/code/NeMo/examples/asr/transcribe_speech.py \
  audio_dir=wavs \
  output_filename=out.json \
  batch_size=1 \
  pretrained_name=nvidia/canary-1b \
  +prompt.turns='[{role:user,slots:{source_lang:en,target_lang:es,task:asr,pnc:yes}}]'
```

**Collection**: ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)