Repository: mlcommons/inference
Issue: Unstable RNN-T audio_preprocessor with padding
Root cause: Validation Gap

When tracking a mysterious accuracy issue in RNNT, I found the same wavelength won’t generate same feature value, the output feature might have slight difference when the waveform have different padding.
The unstable issue is caused by this line that process waveform and transfer to features in input
`feature, feature_length = self.audio_preprocessor.forward((waveform_batch, waveform_lengths))        `

See the following data.  These two inputs comes from same index.  The initial waveform is the same.   After padding the second run has some padding in waveform.  Then we can see features is slightly different, and the final transcript is different.
I checked pytorch/parts/features.py and there are comments that masks should be done and all those mask code is commented.

First accuracy run with bs=4
```
idx=1130 i=0 waveform_list=tensor([ 0.0007,  0.0018,  0.0009,  ..., -0.0016, -0.0008, -0.0010])
idx=1130 i=0 waveform=tensor([ 0.0007,  0.0018,  0.0009,  ..., -0.0016, -0.0008, -0.0010]) sum=-3.613037109375
idx=1130 i=0 feature=tensor([[-0.7800, -0.4811, -0.7984,  ..., -1.7430, -0.7438, -0.5575],
        [-1.1574, -0.8114, -0.8207,  ..., -1.2880, -1.0434, -0.7543],
        [-1.3713, -1.1696, -1.0467,  ..., -1.0476, -1.1701, -0.9726],
        ...,
        [-1.4445, -0.8084, -0.6845,  ..., -1.1323, -1.1321,  3.5289],
        [-1.4877, -1.1429, -0.9883,  ..., -1.7387, -1.1706,  3.2552],
        [-1.7105, -1.0932, -0.9568,  ..., -1.5876, -1.0704,  3.1636]],
       device='dpcpp')
idx=1130 i=0 feature_length=82 transcript=[15, 8, 0, 14, 15, 0, 9, 0, 1, 13, 0, 17, 21, 5, 20, 21, 18, 5, 0, 8, 5, 0, 4, 9, 4, 14, 27, 20]
```
Second accuracy run with bs=128
```
idx=1130 i=56 waveform_list=tensor([ 0.0007,  0.0018,  0.0009,  ..., -0.0016, -0.0008, -0.0010])
idx=1130 i=56 waveform=tensor([0.0007, 0.0018, 0.0009,  ..., 0.0000, 0.0000, 0.0000]) sum=-3.613037109375
idx=1130 i=56 feature=tensor([[-0.7798, -0.4809, -0.7983,  ..., -5.0028, -5.0028, -5.0028],
        [-1.1572, -0.8112, -0.8206,  ..., -4.2505, -4.2505, -4.2505],
        [-1.3711, -1.1694, -1.0465,  ..., -3.9795, -3.9795, -3.9795],
        ...,
        [-1.4723, -0.7939, -0.6617,  ..., -6.5171, -6.5171,  3.8320],
        [-1.4750, -1.1170, -0.9565,  ..., -6.1042, -6.1042,  3.4496],
        [-1.6823, -1.0494, -0.9096,  ..., -5.9635, -5.9635,  3.3144]],
       device='dpcpp')
idx=1130 i=56 feature_length=82 transcript=[15, 8, 0, 14, 15, 0, 9, 0, 1, 13, 0, 17, 21, 5, 5, 18, 0, 8, 5, 0, 4, 9, 4, 14, 27, 20]
```                                      

@galv @nvpohanh any ideas on this?

@galv may have a better idea than I do about why those masking codes were commented-out. 

> I checked pytorch/parts/features.py and there are comments that masks should be done and all those mask code is commented.

@delock Could you point out which specific lines? Thanks

The second transcript is different on position 15.
The first element of feature in the first run is -0.7800, the first element of feature in the second run is -0.7798.

> @galv may have a better idea than I do about why those masking codes were commented-out.
> 
> > I checked pytorch/parts/features.py and there are comments that masks should be done and all those mask code is commented.
> 
> @delock Could you point out which specific lines? Thanks

I see complaint in the code itself
https://github.com/mlcommons/inference/blob/master/speech_recognition/rnnt/pytorch/parts/features.py#L157
https://github.com/mlcommons/inference/blob/master/speech_recognition/rnnt/pytorch/parts/features.py#L213
And also these commented code that seems related to masked behavior
https://github.com/mlcommons/inference/blob/master/speech_recognition/rnnt/pytorch/parts/features.py#L218
After all the value after audio preprocessing is not stable to #padding in waveform (input)

Wow, my notes to self come in handy, huh?

Anyway,

> https://github.com/mlcommons/inference/blob/master/speech_recognition/rnnt/pytorch/parts/features.py#L157

This problem is very, very minor, if I remember my past's self's reasoning correctly. Consider an input of size 2x20, with sequence lengths [10, 20].

Basically, for the first item in the batch, before preemphasis, all times < 10 are meaningful, and all values time >= 10 should have a value of 0.0. After preemphasis, the value at index 10 will actually contain `-self.preemph * value at index 9`. Since we aren't doing padding immediately after preemphasis, index 10 is being treated as having a non-zero value, when it really shouldn't be. However, I doubt that is a meaningful issue. In addition, this has nothing to do with batching.

Regarding my other comments, it's clear that I wouldn't be able to diagnose quite what's going on there without putting more time into it, which speaking frankly, would take a while for me to get around to! So in other words, a contribution here would be very much welcome.

Yes the direct cause is very minor, we can see the difference in feature tensor in the example is about 0.02%.   However this could tip over greedy decoder's selection when two outputs have very close value.
In the example above, the first transcript is
oh no i am queture he didn't
the second transcript is
oh no i am queer he didn't
I didn't hear the original audio to see which one is more close.

A following question is, sound like this is an issue not easy to fix.   And this would cause accuracy stability issue, that the same sample in different batch might have different value simply because the longest sample in the batch is different.  Can we make RNN-T waive of audit test that mandates same sample will always get same inference result?

> Wow, my notes to self come in handy, huh?
> 
> Anyway,
> 
> > https://github.com/mlcommons/inference/blob/master/speech_recognition/rnnt/pytorch/parts/features.py#L157
> 
> This problem is very, very minor, if I remember my past's self's reasoning correctly. Consider an input of size 2x20, with sequence lengths [10, 20].
> 
> Basically, for the first item in the batch, before preemphasis, all times < 10 are meaningful, and all values time >= 10 should have a value of 0.0. After preemphasis, the value at index 10 will actually contain `-self.preemph * value at index 9`. Since we aren't doing padding immediately after preemphasis, index 10 is being treated as having a non-zero value, when it really shouldn't be. However, I doubt that is a meaningful issue. In addition, this has nothing to do with batching.
> 
> Regarding my other comments, it's clear that I wouldn't be able to diagnose quite what's going on there without putting more time into it, which speaking frankly, would take a while for me to get around to! So in other words, a contribution here would be very much welcome.



> Can we make RNN-T waive of audit test that mandates same sample will always get same inference result?

@georgelyuan @nv-rborkar Could you comment on this? ^^^

There is another source of instability. RNNT encoder is also sensitive to padding.

Index 342, no padding:
Input feature without padding
```
tensor([[[-1.4145, -0.5309, -0.2315,  ..., -0.2998, -0.2471, -0.3127]],                           
                                                                                                  
        [[-0.5919, -0.3745, -0.4387,  ..., -1.0306, -0.7269, -0.6440]],                           
                                                                                                  
        [[-0.6148, -0.8319, -0.8317,  ..., -0.7402, -0.1455, -0.2249]],                           
                                                                                                  
        ...,                                                                                      
                                                                                                  
        [[-0.2204, -0.9337, -0.7143,  ..., -0.9848, -0.8590, -0.6262]],                           
                                                                                                  
        [[-0.1166, -0.0019, -0.1328,  ..., -0.4671, -0.3745, -0.6702]],                           
                                                                                                  
        [[ 0.0853, -0.2338, -0.2812,  ..., -0.5097, -0.6032, -0.7745]]])             
```             
input shape: torch.Size([101, 1, 240])                                                                         
input sequence length tensor([101])                                                                                     
result of encoding:
```
tensor([[[ 1.6638e-02,  2.7330e-02,  4.0094e-02,  ...,  1.7563e-02,                               
          -8.4806e-02,  1.5089e-02],                                                              
         [-1.4059e-02,  7.7343e-02,  2.7369e-02,  ...,  3.3430e-03,                               
          -1.3860e-01,  1.7699e-02],                                                              
         [-1.8549e-02,  2.9244e-02,  6.1052e-03,  ...,  1.2046e-03,                               
          -3.0347e-02,  3.4044e-02],                                                              
         ...,                                                                                     
         [-1.3019e-01, -4.7606e-03,  3.8779e-03,  ...,  8.2823e-05,                               
          -1.9144e-02,  4.9786e-02],                                                              
         [-1.5974e-01, -2.2053e-03,  2.0833e-03,  ..., -3.3526e-04,                               
          -2.1200e-03,  1.9253e-02],                                                              
         [-1.2397e-01, -2.5250e-03,  2.3010e-03,  ..., -7.2155e-04,                               
           6.0329e-03,  1.2191e-02]]])                                                    
```
encoder output shape: torch.Size([1, 51, 1024])                                                                         
encoder output length: tensor([51], dtype=torch.int32)                                                                   
decode result pad # = 0: [19, 15, 0, 8, 5, 0, 3, 1, 18, 5, 19, 0, 8, 21]so he cares hu
----------------------------------------------------------------------
Same index (342), pad input features with 0.0
Input, note the zero padding at the end
```
tensor([[[-1.4145, -0.5309, -0.2315,  ..., -0.2998, -0.2471, -0.3127]],

        [[-0.5919, -0.3745, -0.4387,  ..., -1.0306, -0.7269, -0.6440]],

        [[-0.6148, -0.8319, -0.8317,  ..., -0.7402, -0.1455, -0.2249]],

        ...,

        [[-0.1166, -0.0019, -0.1328,  ..., -0.4671, -0.3745, -0.6702]],

        [[ 0.0853, -0.2338, -0.2812,  ..., -0.5097, -0.6032, -0.7745]],

        [[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000]]])
```
input shape, note the increase of input size (102 vs 101): torch.Size([102, 1, 240])
input length, note the length is 101 that marks the boundary of input features: tensor([101])
Output: note how the last line differ from no padding with large relative error.  While all the steps before that give same value. 
```
tensor([[[ 1.6638e-02,  2.7330e-02,  4.0094e-02,  ...,  1.7563e-02,
          -8.4806e-02,  1.5089e-02],
         [-1.4059e-02,  7.7343e-02,  2.7369e-02,  ...,  3.3430e-03,
          -1.3860e-01,  1.7699e-02],
         [-1.8549e-02,  2.9244e-02,  6.1052e-03,  ...,  1.2046e-03,
          -3.0347e-02,  3.4044e-02],
         ...,
         [-1.3019e-01, -4.7606e-03,  3.8779e-03,  ...,  8.2823e-05,
          -1.9144e-02,  4.9786e-02],
         [-1.5974e-01, -2.2053e-03,  2.0833e-03,  ..., -3.3526e-04,
          -2.1200e-03,  1.9253e-02],
         [-1.1567e-01, -2.1508e-03,  1.6184e-03,  ..., -2.4320e-04,
           3.4836e-03,  9.8401e-03]]])

```
Output shape and output length, note the shape and length is the same (51), means all the values relevant to greedy decoder.
torch.Size([1, 51, 1024])
tensor([51], dtype=torch.int32)
Output, note how the last character of no padding is thrown away because of difference in encoder result.
pad # = 1: [19, 15, 0, 8, 5, 0, 3, 1, 18, 5, 19, 0, 8]so he cares h

This problem can be reproduced with pytorch 1.7.1 installed from pip for python 3.7

@galv @delock @nvpohanh @georgelyuan @christ1ne had a one off meeting:
- this is a real issue
- @delock will generate the numpy arrays which can reproduce the issue
- @galv can look into debugging this
- we agreed to make this issue an exception in compliance test 01 if we cannot have a speedy resolution for v1.0 submission.

To be clear on the last point: we agree that this can fall under the "Non-Determinism" bucket in TEST01: https://github.com/mlcommons/inference/tree/master/compliance/nvidia/TEST01#non-determinism

That means, TEST01 should still be run, but the submitter can submit an explanation about the non-determinism and just make sure that the accuracy doesn't drop by more than 1%.

The following files contains input dump and transcript of the batch.  The index that has problem is 342 and 1887.  The first one falls into instability of encoder, the second one falls into instability of audio preprocessor.  @galv 
[bs2.txt](https://github.com/mlcommons/inference/files/5877750/bs2.txt)
[bs4.txt](https://github.com/mlcommons/inference/files/5877751/bs4.txt)


I've begun look into this. I have some ideas as to what causes the stability based on looking at the code However, it's hard to consider it fixed until I've nailed down a reproducer test case. @delock can you clarify the meaning of "input for idx[414, 342] 
" in bs2.txt? I'm supposing that 342 and 414 are the query sample indexes used to create the batches. Is that right?

I am going to try to set up a reproducer unit test, but the more information, the better.

Yes, 342 and 414 is the sample index in the query.  The value after equal sign the  waveform value used to reproduce the issue.