Repository: mlcommons/inference
Issue: [GNMT] Difference in Beam Search Algorithm between PyTorch and TensorFlow 
Root cause: Algorithmic Error

TensorFlow GNMT implements beam search algorithm according to the paper (https://arxiv.org/abs/1609.08144); in particular, length and coverage penalty has been taken into consideration before applying TopK, while some of PyTorch implementations does not do that, e.g.

- https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/Translation/GNMT/seq2seq/models/seq2seq_base.py#L61-L64
- https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/Translation/GNMT/seq2seq/inference/beam_search.py

Such difference entails mathematically different computations and makes unequal level for benchmarking performance. **So the question is such variation allowed in the submission?** 



For MLPerf Inference 0.5 both the TF and the PyTorch reference implementations are equally valid. If either of the reference implementations applies the length and convergence penalty before applying TopK, then it is valid. Otherwise, it seems like these calculations would yield very different results and consequently could not be considered mathematically equivalent or an approximation.

@tjablin Only the TF version is the reference as the PyTorch SW has not been vetted for equivalence. That's why the PyTorch version is not under v0.5 directory but in others/cloud/translation/gnmt/pytorch. Let's clarify this point again in the meeting today. 

@christ1ne Thanks for the correction. I had totally forgotten that NMT did not have a PyTorch reference. Sorry for spreading misinformation.

@DilipSequeira To be clear, the rules are:

> All implementations are allowed as long as the latency and accuracy bounds are met and the reference weights are used. Reference weights may be modified according to the quantization rules.

So taking:
> length and coverage penalty has been taken into consideration before applying TopK

Isn't invalid, but it seems like a radical enough change that it would require weight supplements. Weight supplements aren't valid.

> The following techniques are disallowed:
> Wholesale weight replacement or supplements

If you have a submission that applies length and coverage penalties before applying TopK and does not use weight then this would be a valid submission, in my opinion.

Sorry again for the confusion.


To be clear, you should do your own experiments to determine if you can reach the required level of accuracy with your proposed change without weight supplements.

@tjablin there is no weight in beam search. Am I missing something?

If changing the beam search algorithm in the proposed manner reduces accuracy for the overall model then it might require retraining or weight supplements to meet the required accuracy.

I thought the criteria is whether the proposed change is mathematically equivalent to the reference.  @tjablin 

The rules say:
> All implementations are allowed as long as the latency and accuracy bounds are met and the reference weights are used. Reference weights may be modified according to the quantization rules.



Let's clarify this during the meeting later. There is no weights in the beam search. 

WG: @mdfaijulamin this proposal is allowed as long as you can hit the accuracy threshold. 

but you cannot do the following: 
https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#82-model-equivalence

Treating beams in a beam search differently. For example, employing different precision for different beams.
Changing the number of beams per beam search relative to the reference.