Fine-tuning Conformer CTC with Conformer-CTC streaming model
State: closed | Pull request: no
Labels: name: stale

Is it possible to pre-train a model with conformer-ctc bpe type model and then finetune it with the conformer-ctc streaming model yaml?

should be possible. Are you facing any issues while finetuning? 
@VahidooX to confirm. 

Not for offline model to streaming. You can do this if you keep the encoder architecture the same, with the flags for loading a checkpoint inside the maybe init from checkpoint function. You will have to exclude the decoder and joint weights when you specify the RNNT model checkpoint inside your ctc train script. 

https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/core/api.html#nemo.core.ModelPT.maybe_init_from_pretrained_checkpoint

The convolutions in streaming models are different from the offline models, so not sure if it is possible. Names can be different. Have not tried it, but it is better to start from a streaming model instead. We have some pretrained streaming checkpoints already for English. You may keep the decoder if load from a ctc to another ctc.

Oh I just realized user wants to train offline first and use it for streaming, sadly this may not be possible as Vahid pointed out. I will update my comment above 

The conformer CTC model is an encoder only architecture. So I tried to initialize and offline Conformer-CTC for the conformer-CTC streaming architecture and then ran the finetune on just 100hrs of train data with the streaming configs. It still worked and gave much lower WER in the zeroth epoch itself. Which means that it is indeed possible!!

That's pretty amazing ! Wasn't there shape mismatch when loading the checkpoint ?

Sorry my mistake. There was no error as such and the online Conf-CTC checkpoint loaded correctly. On fine-tuning with conf-CTC streaming, it ran without an error but to my surprise the .nemo model generated still contains model.config which is non-streaming. If that is the case then how did some part of the streaming configs enabled training to proceed as usual in the first place? Please comment if this sounds like an error - 

'''
[NeMo W 2024-01-04 04:28:17 exp_manager:900] The checkpoint callback was told to monitor a validation value and trainer's max_steps was set to -1. Please ensure that max_steps will run for at least 1 epochs to ensure that checkpointing will not error out.
[NeMo W 2024-01-04 04:28:19 modelPT:142] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config :
    manifest_filepath: ../../data/hindi_normalized/filtered_hindi_v1_manifest.json
    sample_rate: 16000
    batch_size: 24
    shuffle: true
    num_workers: 16
    pin_memory: true
    use_start_end_token: false
    trim_silence: false
    max_duration: 30
    min_duration: 0.1
    is_tarred: false
    tarred_audio_filepaths: null
    shuffle_n: 2048
    bucketing_strategy: synced_randomized
    bucketing_batch_size: 8
    
    [NeMo W 2024-01-04 04:30:11 modelPT:486] Trainer wasn't specified in model constructor. Make sure that you really wanted it.
[NeMo W 2024-01-04 04:30:11 lr_scheduler:890] Neither `max_steps` nor `iters_per_batch` were provided to `optim.sched`, cannot compute effective `max_steps` !
    Scheduler will not be instantiated !
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 1 processes
----------------------------------------------------------------------------------------------------

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name              | Type                              | Params
------------------------------------------------------------------------
0 | preprocessor      | AudioToMelSpectrogramPreprocessor | 0
1 | encoder           | ConformerEncoder                  | 115 M
2 | decoder           | ConvASRDecoder                    | 66.2 K
3 | loss              | CTCLoss                           | 0
4 | spec_augmentation | SpectrogramAugmentation           | 0
5 | _wer              | WERBPE                            | 0
6 | spec_augment      | SpectrogramAugmentation           | 0
------------------------------------------------------------------------
115 M     Trainable params
0         Non-trainable params
115 M     Total params
460.710   Total estimated model params size (MB)
2024-01-04 04:30:13.561892: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
SLURM auto-requeueing enabled. Setting signal handlers.
[W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
'''


what is the script you used to finetune streaming model? and model config. 
CC: @VahidooX 

I used the script "examples/asr/speech_to_text_finetune.py" and large topology of the config "examples/asr/conf/conformer/cache_aware_streaming/conformer_ctc_bpe_streaming.yaml" with 17 conformer layers. Please note that the initial pre-trained model is conformer ctc model with 17 layers (non-streaming).

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.

> Sorry my mistake. There was no error as such and the online Conf-CTC checkpoint loaded correctly. On fine-tuning with conf-CTC streaming, it ran without an error but to my surprise the .nemo model generated still contains model.config which is non-streaming. If that is the case then how did some part of the streaming configs enabled training to proceed as usual in the first place? Please comment if this sounds like an error -
> 
> ''' [NeMo W 2024-01-04 04:28:17 exp_manager:900] The checkpoint callback was told to monitor a validation value and trainer's max_steps was set to -1. Please ensure that max_steps will run for at least 1 epochs to ensure that checkpointing will not error out. [NeMo W 2024-01-04 04:28:19 modelPT:142] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader. Train config : manifest_filepath: ../../data/hindi_normalized/filtered_hindi_v1_manifest.json sample_rate: 16000 batch_size: 24 shuffle: true num_workers: 16 pin_memory: true use_start_end_token: false trim_silence: false max_duration: 30 min_duration: 0.1 is_tarred: false tarred_audio_filepaths: null shuffle_n: 2048 bucketing_strategy: synced_randomized bucketing_batch_size: 8
> 
> ```
> [NeMo W 2024-01-04 04:30:11 modelPT:486] Trainer wasn't specified in model constructor. Make sure that you really wanted it.
> ```
> 
> ## [NeMo W 2024-01-04 04:30:11 lr_scheduler:890] Neither `max_steps` nor `iters_per_batch` were provided to `optim.sched`, cannot compute effective `max_steps` !
> Scheduler will not be instantiated !
> Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
> ## distributed_backend=nccl
> All distributed processes registered. Starting with 1 processes
> LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
> 
> ## | Name | Type | Params
> ## 0 | preprocessor | AudioToMelSpectrogramPreprocessor | 0
> 1 | encoder | ConformerEncoder | 115 M
> 2 | decoder | ConvASRDecoder | 66.2 K
> 3 | loss | CTCLoss | 0
> 4 | spec_augmentation | SpectrogramAugmentation | 0
> 5 | _wer | WERBPE | 0
> 6 | spec_augment | SpectrogramAugmentation | 0
> 115 M Trainable params 0 Non-trainable params 115 M Total params 460.710 Total estimated model params size (MB) 2024-01-04 04:30:13.561892: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. SLURM auto-requeueing enabled. Setting signal handlers. [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) '''

Is there any upadate on this?