Some Questions about SSL Pre-training
State: closed | Pull request: no

Hi, 
The SSL pre-training code & tutorial are very great; however, I have some questions about configurations of models in SSL pre-training (there are some ambiguities for me to set parameters correctly). I have prepared a `citrinet_ssl_512.yaml` config file for SSL pre-training of Citrinet512 model as below: (I have implement it based on the [Nemo citrinet_ssl_1024.yaml](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/ssl/citrinet/citrinet_ssl_1024.yaml) file. But, as Citrinet1024 is a little huge model I decided to switch to Citrinet512 model for my research experiments). 

[citrinet_ssl_512.zip](https://github.com/NVIDIA/NeMo/files/8178032/citrinet_ssl_512.zip)

My questions are as follows:
1- In the above uploaded file, I have written `doubt` comment beside some of parameters. Can you please kindly check these parameters and let me know if they are set correctly or not? I also suggest that add this yaml file (and also yaml config files for other ASR models in the [SSL conf folder](https://github.com/NVIDIA/NeMo/tree/main/examples/asr/conf/ssl) of NeMo (_e.g._, yaml config file for QuartzNet, Citrinet256, contextnet256, etc.)). 
2- If we set `precision: 16` is it adequate for training the model with fp16? or we need apply extra changes?
3- I have set `min_duration: 2.0`. But, in my data set there are audio utterances with duration less than 2 seconds. So, it means that all of these utterances will be removed form the training process? Is there any way to handle this issue and keep them? (I have realized that there is a relation between `spec_augment` parameters and  `min_duration`. As I have set `patch_size: 16` & `mask_patches: 10`, so 16*10=160, and it means that the minimum acceptable duration of an utterance must be 1.6 seconds. As a result, I have set `min_duration: 2.0`. Am I correct?)
4- I have set the `max_duration: 60.0`. Is this parameter important and related to `spec_augment` category parameters?
5- I have set `trim_silence: false` & `pin_memory: true` & add `num_workers: 16`. Do the values of these parameters have a special effect on the training process? 
6- I have set `num_negatives: 40` & `sample_from_same_utterance_only: true` & `sample_from_non_masked: false`. Do you think the values of these parameters are set correctly? I have set these values based on your [tutorial file](https://nbviewer.org/github/NVIDIA/NeMo/blob/main/tutorials/asr/Self_Supervised_Pre_Training.ipynb), and other `spec_augment` category parameters' values.
7- About loading `init_checkpoint` (_i.e._, `+init_from_pretrained_model`) I have previously send an [issue](https://github.com/NVIDIA/NeMo/issues/3728). What I learned from that is that to do this, I must first download the model manually from NGC and then enter the .nemo file address of the model to code. Did I get it right?
In fact, I want to evaluate the impact of loading another model's weight at the beginning of SSL process for my self, and I know that it is not a formal acceptable task.
8- After the process of SSL training has been finished, can one extract the log probability of the SSL model for evaluation? I mean that, can we extract log probability of model for each word (or token) to for example evaluate (analyze) which words the model transcribe with more skepticism and which words it is more confident about transcribing. Can you help me with this too?

I apologize to you that my number of questions has increased and I thank you in advance if you can guide me.
Best