Repository: mlcommons/inference
Issue: [llama] Question about the Inference accuracy 
Root cause: Algorithmic Error

**Premiss** 

1. The inference code for causal-lm in HF is written in left-padding convention, the HF will select the last index of the output tensor as the logits of the next words.
2. The Llama inference code with batching is manually right-padded
- which will cause an HF warning
**"A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer."**
- The output may produce gibberish words at the beginning of output tokens, e.g. "��� Driver turned onto highway....", ""
- Also, it may be logically not the right approach.

3. I also tried to use **left padding <\unk> token (id, 0) to do HF FP32 batch inference (BS=4)**, the total tested sample is 164,  of which outputs of 139 samples are the same as run **HF Fp32 with no padding (BS=1)**, the others (25) non-consistency results I think maybe caused by numerically unstable.

**Question**
1. Maybe the results of HF FP32 inference without batching (no padding) or left padding batching results are the most accurate results, we need to set this as our accuracy target?

@nv-alicheng 
#1544 

I think the key question here is: is the accuracy test expected to pass with batch size > 1, or not?  If not, then this is not an issue, but the expectation should be clarified.  If it is expected to pass with batch size > 1, then these padding issues need to be resolved.

Ah shoot - Sorry it took a while for a reply, but this looks like an actual bug with the SUT. I just ran an experiment with sample_count=100.

The script on top-of-tree shows the following with sample count 100:
{'rouge1': 46.799, 'rouge2': 24.0965, 'rougeL': 30.7241, 'rougeLsum': 44.3092, 'gen_len': 91291, 'gen_num': 100}

Changing the padding parameters from
```
(0, max_seq_len - self.data_object.input_lens[q.index], 0, 0)
```
to
```
(max_seq_len - self.data_object.input_lens[q.index], 0, 0, 0)
```
removes the warning and has a higher rouge score:
{'rouge1': 47.6373, 'rouge2': 25.5539, 'rougeL': 32.0018, 'rougeLsum': 45.2576, 'gen_len': 97811, 'gen_num': 100}

I can submit a PR with this fix.
@nvzhihanj We might have to take another week to re-run the fp32 accuracy with this fix.

@nv-alicheng Is there a PR for padding parameters changes? 

For the key setting like padding_side='left', greedy search and more, is there a source of truth place to capture the final setting? 

@FanhaiLu1 https://github.com/mlcommons/inference/pull/1572 was merged last week for the padding parameters change.

> For the key setting like padding_side='left', greedy search and more, is there a source of truth place to capture the final setting?

Can you elaborate on this?

Suppose I'm the new member who will participate the ML Perf, I will first check .../lama2-70b/README.md for Reference Implementation details, it's a good document, but the README doesn't share the inference requirements. Shall I use top k or beam search? Can I use quantization? Is there any tech approach I can not leverage? It would be helpful if there are several rules or requirements about the inference. 

I may miss something, please correct me if I'm wrong. Thanks!

Hi @FanhaiLu1 

Rules for the inference submission are described in https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc 

_Q: What algorithm is used for the auto-regressive decoding loop?_

_A: The algorithms used by the benchmarks (greedy search and beam search) are described at a high level here: https://huggingface.co/blog/how-to-generate. Specifically, GPT-J uses a beam width of 4 and enable early termination, while **Llama2 uses greedy search**._

Quantization rules and disallowed techniques are also explained in the inference rules.

Ah I see what you are asking now. @nvzhihanj Can you add this to the README?

I will add a link to the inference rules and the above details in a new PR.

Thanks @nv-alicheng @nv-alicheng and @szutenberg for the explanation and adding the link!