Repository: mlcommons/inference
Issue: Running whisper reference implementation produces errors in the loadgen log file
Root cause: Validation Gap

```
:::MLLOG {"key": "generic_message", "value": "Ending naturally: Minimum query count and test duration met.", "time_ms": 177.942689, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "issue_query_controller.cc", "line_no": 482, "pid": 28141, "tid": 28141}}
:::MLLOG {"key": "error_runtime", "value": "n_tokens argument missing or attempted to record 0 as number of tokens", "time_ms": 177.942689, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": true, "is_warning": false, "file": "logging.cc", "line_no": 443, "pid": 28141, "tid": 28513}}
:::MLLOG {"key": "error_runtime", "value": "n_tokens argument missing or attempted to record 0 as number of tokens", "time_ms": 177.942689, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": true, "is_warning": false, "file": "logging.cc", "line_no": 443, "pid": 28141, "tid": 28513}}
:::MLLOG {"key": "error_runtime", "value": "n_tokens argument missing or attempted to record 0 as number of tokens", "time_ms": 177.942689, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": true, "is_warning": false, "file": "logging.cc", "line_no": 443, "pid": 28141, "tid": 28513}}
:::MLLOG {"key": "error_runtime", "value": "n_tokens argument missing or attempted to record 0 as number of tokens", "time_ms": 177.942689, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": true, "is_warning": false, "file": "logging.cc", "line_no": 443, "pid": 28141, "tid": 28513}}
:::MLLOG {"key": "error_runtime", "value": "n_tokens argument missing or attempted to record 0 as number of tokens", "time_ms": 177.942689, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": true, "is_warning": false, "file": "logging.cc", "line_no": 443, "pid": 28141, "tid": 28513}}
:::MLLOG {"key": "error_runtime", "value": "n_tokens argument missing or attempted to record 0 as number of tokens", "time_ms": 177.942689, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": true, "is_warning": false, "file": "logging.cc", "line_no": 443, "pid": 28141, "tid": 28513}}
:::MLLOG {"key": "error_runtime", "value": "n_tokens argument missing or attempted to record 0 as number of tokens", "time_ms": 177.942689, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": true, "is_warning": false, "file": "logging.cc", "line_no": 443, "pid": 28141, "tid": 28513}}
:::MLLOG {"key": "error_runtime", "value": "n_tokens argument missing or attempted to record 0 as number of tokens", "time_ms": 177.942689, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": true, "is_warning": false, "file": "logging.cc", "line_no": 443, "pid": 28141, "tid": 28513}}
:::MLLOG {"key": "error_runtime", "value": "n_tokens argument missing or attempted to record 0 as number of tokens", "time_ms": 177.942689, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": true, "is_warning": false, "file": "logging.cc", "line_no": 443, "pid": 28141, "tid": 28513}}
```

This sounds a little bit familiar.  I believe the number of tokens isn't being reported to loadgen.  Can you try the following:

Replace here which adds the number of tokens to the results: https://github.com/mlcommons/inference/blob/b9ed3c7fec1f16fe0576852bd3ade78100d86672/speech2text/reference_SUT.py#L218
`results.append(vllm_text, len(output.outputs[0].token_ids)) `

Here, propagate the number of tokens: https://github.com/mlcommons/inference/blob/b9ed3c7fec1f16fe0576852bd3ade78100d86672/speech2text/reference_SUT.py#L224
```
for i,result_tuple in enumerate(results):
            result, n_tokens = result_tuple 
```

And here, report the number of tokens to loadgen: https://github.com/mlcommons/inference/blob/b9ed3c7fec1f16fe0576852bd3ade78100d86672/speech2text/reference_SUT.py#L236
`self.output_queue.put((qid[i], n_tokens, response_array))`


AR: Please verify it.

@keithachorn-intel Thanks. That worked. Here's the full change needed. Can you please open a PR?
```
diff --git a/speech2text/reference_SUT.py b/speech2text/reference_SUT.py
index 5b3b6c5..6f975bb 100644
--- a/speech2text/reference_SUT.py
+++ b/speech2text/reference_SUT.py
@@ -215,15 +215,15 @@ class Instance(mp.Process):
         for output in outputs:
             request_id = int(output.request_id)
             vllm_text = output.outputs[0].text
-            results.append(vllm_text)
+            results.append((vllm_text, len(output.outputs[0].token_ids)))
             query_ids.append(self.query_idx_mapping[request_id])
             qid.append(self.qid_mapping[request_id])

         self.num_samples += len(results)

-        for i, result in enumerate(results):
+        for i, result_tuple in enumerate(results):
             # Whisper outputs space in the front and capitalizes things
-            result = result.lower().strip()
+            result, n_tokens = result_tuple
             transcript = []
             for s in result:
                 if s in labels_dict:
@@ -233,7 +233,7 @@ class Instance(mp.Process):
             assert len(transcript) == 1
             response_array = array.array('q', transcript[0])

-            self.output_queue.put((qid[i], response_array))
+            self.output_queue.put((qid[i], n_tokens, response_array))
             print(f"Finished {qid[i]}")
         return True

@@ -330,14 +330,13 @@ class vllmSUT:
     def response_loadgen(self):
         keep_alive = True
         while keep_alive:
-            result = self.output_queue.get()
-            if result is None:
+            qid, n_tokens, response_array = self.output_queue.get()
+            if qid is None:
                 keep_alive = False
             else:
-                qid, response_array = result
                 bi = response_array.buffer_info()
                 response = lg.QuerySampleResponse(qid, bi[0],
-                                                  bi[1] * response_array.itemsize)
+                                                  bi[1] * response_array.itemsize, n_tokens)
                 lg.QuerySamplesComplete([response])

     def stop(self):
```

But the accuracy we got by running the reference implementation is `95.33295` which is even below the 99% requirement. Is the reference accuracy taken at fp32 and the reference implementation having bfloat16 as the dtype?

> But the accuracy we got by running the reference implementation is `95.33295` which is even below the 99% requirement. Is the reference accuracy taken at fp32 and the reference implementation having bfloat16 as the dtype?

Are you building the dataset (dev-all-repack) using the embedded script, or downloading from MLC.  Using the build script, you should see > 97%.

Thanks @keithachorn-intel for the information. I was downloading the preprocessed dataset. I'm not sure why that is there if the accuracy is bad. @anandhu-eng can you please remove that from the workflow and add the processing scripts instead?

@arjunsuresh  - I think removing this line contributed to the accuracy issue you saw.  I'm factoring this into the PR.

>  @anandhu-eng can you please remove that from the workflow and add the processing scripts instead?

Hi @arjunsuresh , Could you please retry with the latest changes made by Keith? If it still doesn’t work,  I’ll be glad to update the flow accordingly.

@keithachorn-intel Is there a PR for the relevant change? 

> [@keithachorn-intel](https://github.com/keithachorn-intel) Is there a PR for the relevant change?

Yes, but it was already merged: https://github.com/mlcommons/inference/pull/2305

> [@keithachorn-intel](https://github.com/keithachorn-intel) but that PR is only dealing with token latencies right?

The PR ensures the loadgen gets the correct number of tokens returned value, not the latency.  This lack of this value led to the error at the start of this thread.  It didn't affect runtime performance or accuracy, but did affect the compliance checks.

Thanks Keith, all is working fine now.