using new_vocabulary() with BPE model keeps the old tokenizer path in final model
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

It seems that while the vocabulary is updated and the model is trained, trying to restore the model from checkpoint still uses the old tokenizer path.

**Steps/Code to reproduce bug**
I did something very similar to what's in the 08_ASR_with_Subword_Tokenization.ipynb notebook:

```
model_cfg = ASRModel.restore_from(restore_path="stt_en_citrinet_1024.nemo", return_config=True)
classpath = model_cfg.target
imported_class = model_utils.import_class_by_path(classpath)

trainer = pl.Trainer(
            logger=False,  # Provided by exp_manager
            checkpoint_callback=False,
            accelerator="ddp", gpus=4, max_epochs=1)
citrinet = imported_class.restore_from(restore_path=model_path)
citrinet.set_trainer(trainer)

model_cfg["validation_ds"]["manifest_filepath"]= "/path/to/my/val.manifest"
model_cfg["test_ds"]["manifest_filepath"]= "/path/to/my/test.manifest"
model_cfg["train_ds"]["manifest_filepath"]= "/path/to/my/train.manifest"

new_opt = copy.deepcopy(model_cfg["optim"])
new_opt["lr"] = 0.0003

citrinet.change_vocabulary(
    new_tokenizer_dir="/my/bpe_exp01/tokenizer_spe_unigram_v3000/",
    new_tokenizer_type="bpe"
)

model_cfg["decoder"] = citrinet._cfg.decoder
citrinet.setup_training_data(train_data_config=model_cfg["train_ds"])
citrinet.setup_validation_data(val_data_config=model_cfg["validation_ds"])
citrinet.setup_test_data(test_data_config=model_cfg["test_ds"])

trainer.fit(citrinet)
citrinet.save_to(save_path="./citrinet_1024-adapted03.nemo")
```
The model is trained and I get a fina .nemo file at the end.  Now, if I look at the config inside the final "adapted" model, I see:

```
'tokenizer': {'dir': '../../manifests/asrset_1.4/tokenizers/no_appen/tokenizer_spe_unigram_v1024/', 'type': 'bpe'}
```

**Expected behavior**

I would have expect the tokenizer path to be the right one.  Also, for a model that I let the training happen for a longer number of epochs, I can't covert a checkpoint into a final nemo package because `load_from_checkpoint()` throws:
```
FileNotFoundError: Could not find ../../manifests/asrset_1.4/tokenizers/no_appen/tokenizer_spe_unigram_v1024/tokenizer.model
```

How can I fix that?

**Environment overview (please complete the following information)**

 - Environment location: AWS
 - Method of NeMo install: from source

**Environment details**

- OS version : ubuntu 16.04
- PyTorch version 1.7.1
- Python version : 3.8
- CUDA 10.2


> The model is trained and I get a fina .nemo file at the end. Now, if I look at the config inside the final "adapted" model, I see

Quite a silent bug there, sending a patch in https://github.com/NVIDIA/NeMo/pull/2126

We normally follow this procedure for finetuning our models as well, but didnt face it till now since we explicitly update portions of the config (including model.cfg.tokenizer) + call model.change_vocabulary(). This should not be the case, and the PR above should automate it so that just calling change_vocabulary() does this for you.

> Also, for a model that I let the training happen for a longer number of epochs, I can't covert a checkpoint into a final nemo package because load_from_checkpoint()

`load_from_checkpoint` is a Pytorch Lightning mechanism, it has no reference to models tokenizers (one of the reasons we use .nemo files). You will need to do the following : 

```python
from nemo.collections.asr.models import EncDecCTCModelBPE
import torch

# initialize new model with correct tokenizer + random weight initialization
new_model = EncDecCTCModelBPE(cfg=cfg, trainer=None)

state_dict = torch.load("/path/to/pytorch-lightning/file.ckpt")["state_dict"]
new_model.load_state_dict(state_dict)

new_model.save_to("path to new nemo model.nemo")
```

 

Thanks a lot for this quick reply!  I was able to "salvage" my model by effectively hacking temporarily `ctc_bpe_models.py` `__init__` method and forcing `cfg.tokenizer.dir` to the desired value.

Of course, it will be much better now that this will be fixed in the main code!  Thanks again!

Glad it worked. It will hopefully not be needed again with patch.

In general the models are pytorch modules, so stuff like state_dict should work just fine. Pytorch lightning checkpoints put the actual state_dict inside the "state_dict" sublevel. 

Thanks.  Now I just have to find a way to make this fine-tuning actually improve accuracy on our data.  So far, while the speed of Citrinet model is really appealing, I've been unable to train a good model from scratch and got limited success with fine-tuning.  

Right now, I'm trying freezing the encoder for one epoch (so that I don't destroy a good encoder starting point because I'm flipping the tokenizer, and unfreezing the encoder after.  Let's see how well that goes.

I saw a few concerning things in the snippet above - such as ultra large vocabulary of 3000. In an ablation in the paper, we see smaller vocabulary of 1024 tends to do better. The LR is also too small. 

This is the finetuning recipe I use, maybe it will be something of a guideline - 
```yaml
    model.spec_augment.freq_masks=0 \
    model.spec_augment.time_masks=2 \
    model.optim.lr=0.01 \
    model.optim.name='novograd' \
    model.optim.betas=[0.8,0.25] \
    model.optim.weight_decay=0.001 \
    model.optim.sched.warmup_steps=1000 \
    model.optim.sched.min_lr=0.00001 \
    trainer.gpus=-1 \
    trainer.accelerator='ddp' \
    trainer.max_epochs=100 \
```


Heres a snippet that might help. Im working on moving this to a notebook tutorial eventually - https://gist.github.com/titu1994/273416f3d4104d237ae477a1e7526837

That's very interesting. Thanks a lot, I will definitely have a look at
these values and the sample script.

I got many times the remark about the large vocabulary size.  The thing is,
with other model architecture, like the CTC + Attention, we are seeing
interesting behavior thanks to that.  By interesting, I mean that not only
the accuracy seems better, but also the decoding time goes down.  But
perhaps the simpler decoder strategy citrinet uses makes that less
interesting.

Also, I'm trying to fine-tune with 10,000 hours of audio.  Because of that,
running for 100 epochs is quite long (we don't have 32 GPUs lying
around!).I tried to follow the basic values from
https://developer.nvidia.com/blog/jump-start-training-for-speech-recognition-models-with-nemo/
(even though I'm not going into another language, but just changing the
English data.)



On Tue, Apr 27, 2021 at 9:43 PM Somshubra Majumdar ***@***.***>
wrote:

> Heres a snippet that might help. Im working on moving this to a notebook
> tutorial eventually -
> https://gist.github.com/titu1994/273416f3d4104d237ae477a1e7526837
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/NVIDIA/NeMo/issues/2121#issuecomment-828076728>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACGTL25N3DK2KCCBO24O5WTTK5R2ZANCNFSM43TI4YMA>
> .
>


CTC + Attention is I assume an Autoregressive model (or maybe a Wav2Vec). If it's Autoregressive, it's decoding is slower than the straightforward CTC argmax decoding. 

Larger vocabulary reduces number of decoding timesteps required, which is probably what you are observing. Citrinet does not use attention at all, and does not train well with such large vocabularies. 

Thanks, I'll take good note of this.  We were coming to the same
conclusion, but having an expert confirming our intuition is always a good
thing!  Thanks a lot again for all these details!

On Tue, Apr 27, 2021 at 11:30 PM Somshubra Majumdar <
***@***.***> wrote:

> CTC + Attention is I assume an Autoregressive model (or maybe a Wav2Vec).
> If it's Autoregressive, it's decoding is slower than the straightforward
> CTC argmax decoding.
>
> Larger vocabulary reduces number of decoding timesteps required, which is
> probably what you are observing. Citrinet does not use attention at all,
> and does not train well with such large vocabularies.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/NVIDIA/NeMo/issues/2121#issuecomment-828115159>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACGTL27BKN43ZJXRMPTWOB3TK56NTANCNFSM43TI4YMA>
> .
>