Intent and slot classification tutorial training presents constant loss
State: closed | Pull request: no
Labels: name: bug

name: stale

Hello,
I have cloned the intent and slot classification tutorial and running it on Sagemaker.
The branch is "Main", Environment Pytorch 1.12 and python 3.8.

I have tried running via Jupiter notebook and via terminal. 
The result is always the same I have the training loss stack at 4.13 after 22 Epochs and still the same afterwards.

Main modification to the existing is the max number of Epochs = 50.
I assume this is not the expected behaviour. Can anybody help ? 
Just an example of the output.

``
Epoch 26: 100%|| 346/346 [00:49<00:00,  6.99it/s, loss=4.13, v_num=4-50, lr=0.000]Epoch 26, global step 8424: 'val_loss' was not in top 3
``               
.....                                                                                                                                              
``
Epoch 28: 100% || 346/346 [00:49<00:00,  6.97it/s, loss=4.13, v_num=4-50, lr=0.000]Epoch 28, global step 9048: 'val_loss' was not in top 3                                                   
``

UPDATE: I noticed the following.

1) running in colab for an older version essentially same issue.
``
https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/nlp/Joint_Intent_and_Slot_Classification.ipynb#scrollTo=zt-cZuNVl6la
``

I also noticed in logs something strange. Is the training set as the testing set not well balanced ?there seems not to be enough representation of the labels for intent and slot.

``
[NeMo I 2023-01-28 23:00:43 intent_slot_classification_descriptor:87]  Stats calculating for train mode...
[NeMo I 2023-01-28 23:00:43 intent_slot_classification_descriptor:112] Three most popular intents in train mode:
[NeMo I 2023-01-28 23:00:43 data_preprocessing:194] label: 0, 175 out of 9960 (1.76%).
[NeMo I 2023-01-28 23:00:43 data_preprocessing:194] label: 2, 175 out of 9960 (1.76%).
[NeMo I 2023-01-28 23:00:43 data_preprocessing:194] label: 6, 175 out of 9960 (1.76%).
[NeMo I 2023-01-28 23:00:43 intent_slot_classification_descriptor:118] Three most popular slots in train mode:
[NeMo I 2023-01-28 23:00:43 data_preprocessing:194] label: 54, 53689 out of 65658 (81.77%).
[NeMo I 2023-01-28 23:00:43 data_preprocessing:194] label: 38, 1290 out of 65658 (1.96%).
[NeMo I 2023-01-28 23:00:43 data_preprocessing:194] label: 12, 1233 out of 65658 (1.88%).
[NeMo I 2023-01-28 23:00:43 intent_slot_classification_descriptor:121] Total Number of Intents: 9960
[NeMo I 2023-01-28 23:00:43 intent_slot_classification_descriptor:122] Intent Label Frequencies: {0: 175, 2: 175, 6: 175, 7: 175, 8: 175, 9: 175, 11: 175, 13: 175, 14: 175, 15: 175, 16: 175, 17: 175, 18: 175, 19: 175, 20: 175, 22: 175, 23: 175, 24: 175, 25: 175, 27: 175, 28: 175, 30: 175, 35: 175, 36: 175, 37: 175, 39: 175, 41: 175, 42: 175, 43: 175, 44: 175, 45: 175, 46: 175, 47: 175, 48: 175, 49: 175, 51: 175, 52: 175, 53: 175, 55: 175, 57: 175, 58: 175, 59: 175, 61: 175, 63: 175, 62: 171, 56: 165, 60: 163, 38: 162, 26: 151, 4: 142, 50: 134, 5: 126, 32: 126, 29: 114, 21: 110, 1: 106, 54: 97, 33: 89, 10: 79, 12: 79, 3: 72, 40: 70, 34: 69, 31: 35}
[NeMo I 2023-01-28 23:00:43 intent_slot_classification_descriptor:123] Total Number of Slots: 65658
[NeMo I 2023-01-28 23:00:43 intent_slot_classification_descriptor:124] Slots Label Frequencies: {54: 53689, 38: 1290, 12: 1233, 46: 1076, 14: 637, 36: 599, 5: 579, 18: 526, 19: 458, 11: 423, 52: 356, 28: 345, 23: 330, 26: 317, 43: 298, 6: 252, 4: 247, 20: 233, 13: 207, 35: 176, 37: 167, 48: 158, 9: 138, 44: 137, 41: 136, 2: 134, 7: 133, 34: 131, 45: 120, 53: 114, 16: 106, 47: 95, 33: 91, 39: 87, 42: 70, 25: 66, 49: 60, 22: 51, 40: 48, 8: 42, 27: 37, 1: 34, 32: 33, 0: 28, 24: 23, 3: 23, 50: 23, 29: 22, 30: 22, 51: 21, 17: 15, 15: 9, 10: 8, 31: 4, 21: 1}
[NeMo I 2023-01-28 23:00:43 intent_slot_classification_descriptor:128] Intent Weights: {0: 0.8892857142857142, 2: 0.8892857142857142, 6: 0.8892857142857142, 7: 0.8892857142857142, 8: 0.8892857142857142, 9: 0.8892857142857142, 11: 0.8892857142857142, 13: 0.8892857142857142, 14: 0.8892857142857142, 15: 0.8892857142857142, 16: 0.8892857142857142, 17: 0.8892857142857142, 18: 0.8892857142857142, 19: 0.8892857142857142, 20: 0.8892857142857142, 22: 0.8892857142857142, 23: 0.8892857142857142, 24: 0.8892857142857142, 25: 0.8892857142857142, 27: 0.8892857142857142, 28: 0.8892857142857142, 30: 0.8892857142857142, 35: 0.8892857142857142, 36: 0.8892857142857142, 37: 0.8892857142857142, 39: 0.8892857142857142, 41: 0.8892857142857142, 42: 0.8892857142857142, 43: 0.8892857142857142, 44: 0.8892857142857142, 45: 0.8892857142857142, 46: 0.8892857142857142, 47: 0.8892857142857142, 48: 0.8892857142857142, 49: 0.8892857142857142, 51: 0.8892857142857142, 52: 0.8892857142857142, 53: 0.8892857142857142, 55: 0.8892857142857142, 57: 0.8892857142857142, 58: 0.8892857142857142, 59: 0.8892857142857142, 61: 0.8892857142857142, 63: 0.8892857142857142, 62: 0.9100877192982456, 56: 0.9431818181818182, 60: 0.9547546012269938, 38: 0.9606481481481481, 26: 1.0306291390728477, 4: 1.095950704225352, 50: 1.1613805970149254, 5: 1.2351190476190477, 32: 1.2351190476190477, 29: 1.3651315789473684, 21: 1.4147727272727273, 1: 1.4681603773584906, 54: 1.6043814432989691, 33: 1.7485955056179776, 10: 1.9699367088607596, 12: 1.9699367088607596, 3: 2.1614583333333335, 40: 2.2232142857142856, 34: 2.255434782608696, 31: 4.446428571428571}
[NeMo I 2023-01-28 23:00:43 intent_slot_classification_descriptor:130] Slot Weights: {54: 0.02223512857720982, 38: 0.9254122621564482, 12: 0.9681928776819287, 46: 1.109462656302805, 14: 1.8740687883545026, 36: 1.9929579602367582, 5: 2.061799340555817, 18: 2.2695471828551677, 19: 2.606510520047638, 11: 2.8221792392005156, 52: 3.353319713993871, 28: 3.4602371541501977, 23: 3.617520661157025, 26: 3.765873243475767, 43: 4.005979255643685, 6: 4.737229437229437, 4: 4.833124769966875, 20: 5.1235271166601635, 13: 5.767061923583663, 35: 6.782851239669421, 37: 7.14839412084921, 48: 7.555581127733026, 9: 8.650592885375493, 44: 8.713735899137358, 41: 8.777807486631016, 2: 8.908819538670285, 7: 8.975803144224196, 34: 9.112838306731437, 45: 9.948181818181817, 53: 10.47177033492823, 16: 11.262092624356775, 47: 12.566124401913875, 33: 13.118481518481518, 39: 13.721630094043887, 42: 17.054025974025976, 25: 18.087603305785123, 49: 19.896363636363635, 22: 23.40748663101604, 40: 24.870454545454546, 8: 28.423376623376623, 27: 32.26437346437346, 1: 35.11122994652406, 32: 36.175206611570246, 0: 42.63506493506493, 24: 51.903557312252964, 3: 51.903557312252964, 50: 51.903557312252964, 29: 54.26280991735537, 30: 54.26280991735537, 51: 56.846753246753245, 17: 79.58545454545454, 15: 132.64242424242425, 10: 149.22272727272727, 31: 298.44545454545454, 21: 1193.7818181818182}
[NeMo I 2023-01-28 23:00:43 intent_slot_classification_descriptor:87]  Stats calculating for test mode...
``