Repository: mlcommons/inference
Issue: Compliance TEST04-B bug & workaround from v0.5 present in v0.7
GitHub labels: inference v1.0

Issue https://github.com/mlperf/inference/issues/520 from v0.5 does not appear to have been resolved yet, even though we are now getting ready to submit for v0.7. I will assume my workaround from v0.5 (described and approved in #520) is still sufficient to use for v0.7 unless anyone suggests otherwise.

@nv-rborkar @georgelyuan ^^^

A good change for the next version of TEST04-B (in MLPerf v1.0) would be to avoid using one query filled with the same repeating index like this (which is not representative of any of the MLPerf benchmarks):
[1, 1, 1, 1, ...]

and instead change it to use a query with unique indices, but then repeat that same query many times, like this:
[1, 821, 14, 2482, ...]
[1, 821, 14, 2482, ...]
[1, 821, 14, 2482, ...]
...

This is more aligned with what the actual MLPerf workloads' characteristics are, while still testing to make sure caching isn't happening across queries.

EDIT: Just reiterating that this only affects the Offline usage of TEST04-B.

@mthmsn does this bug affect reference implementation of classification & object detection benchmarks alone or did you observe similar behavior with reference implementations of other benchmarks as well?

Yes the WAR used in v0.5 is sufficient to use. Do you have a patch for the WAR - wonder if we can share it with others who might be facing this issue.

Above is an interesting idea. We do similar for Multi-stream (if spq < performance_sample_count) but since Offline issues just one query, the test just fills the query with the same index repeatedly. 

 

@nv-rborkar I have only tried this with the classification & object detection benchmarks.

Thank you, we will continue to use the same workaround.

If other people have this same problem, they can refer to the changed 3 lines of code from our submission last year (we included the modified version of `main.py` with only those three lines being different from the regular `main.py`). The 3 lines that were changed are highlighted here:
https://github.com/mlperf/inference_results_v0.5/blob/master/closed/CentaurTechnology/code/resnet/python/main_audit-test04b-bug-workaround.py#L356-L358

Keep in mind that the rest of `main.py` has had other minor changes for v0.7, so users implementing this workaround should just adopt those 3 lines and not copy the entire file. Users should also understand that this is used solely for Offline TEST04-B, as the regular version was fine for all other tests and scenarios.

can we close this one?

This problem will crop up for every MLPerf submission round unless this compliance test is changed to accommodate TF-Lite better. I think the suggested changes a few comments further up need to be implemented before the next submission round.

Agreed, we should try to resolve any remaining issues with this test.

@mthmsn can you explain your proposed fix some more? In Offline we typically just issue one query in my understanding. So I am not following the proposed issuing of multiple queries. 
In case of Multi-stream, we do issue queries the way you have explained above for TEST04-B.

> instead change it to use a query with unique indices, but then repeat that same query many times, like this:
> [1, 821, 14, 2482, ...]
> [1, 821, 14, 2482, ...]
> [1, 821, 14, 2482, ...]
> ...
> This is more aligned with what the actual MLPerf workloads' characteristics are, while still testing to make sure caching isn't happening across queries.

@mthmsn I have another question:

In a "normal" run (not TEST-04B), the Offline scenario is already doing what you mentioned: if QSL size is 16, then LoadGen will issue one single query containing indices of [0, 1, 2, ..., 15, 0, 1, 2, ..., 15, 0, 1, 2, ..., 15, 0, 1, 2, ...]. How would your proposal be different from a normal run and help us to capture potential caching? Thanks

@nvpohanh That is an interesting observation. I would have a couple guesses about the potential underlying cause:

1. The issue could arise when the QSL size is lower than the expected batch size in TF-Lite, thereby forcing multiple identical indices within the same batch. In fact, in our `user.conf` file, we manually set the `performance_sample_count_override` to 1024 to ensure this doesn't become a problem in normal runs:
https://github.com/mlperf/inference_results_v0.7/blob/master/closed/CentaurTechnology/code/python-code/user.conf#L5-L7

For TEST04-B, the `*.Offline.performance_issue_same = 1` setting essentially is the equivalent to overriding our `performance_sample_count_override` value, correct? If that's true, then that seems like problematic behavior. Perhaps the QSL size for TEST04-B should not be 1, but be at least the `performance_sample_count_override` specified in `user.conf`. Loadgen could guarantee that the query size is at least several times larger than that `performance_sample_count_override` to ensure that any caching behavior is detected.

2. Alternatively, this could be due input data pointer sharing problems, perhaps with how TF-Lite uses the data pointers, or perhaps related to when Python's global interpreter lock (GIL) is released and multiple threads are trying to access the same input data source at the same time.

Reason 1 above would definitely affect our v0.5 submission.

Both reasons 1 and 2 above could affect our v0.7 submission.

It's not clear to me at the moment what underlying issue is the real culprit here, the symptoms brought up in the other github issues with regard to this may just be symptoms rather than the problem itself.

I'm happy to help you dig further into this, but please be aware that I (and probably most submitters) have other high priorities now that the main MLPerf submission push is over, so any contributions I can provide will be slow-coming.

I'll also add my unsolicited 2-cents here about this test in general:

The main thing that troubles me with TEST04-B is that it doesn't really reflect the actual "normal" run, it feels like there is too much behavior about it that changes, thus sort of making it into the equivalent of its own "scenario" that an MLPerf user must now add support for and debug (which ended up being a LOT of effort for this one compliance test). I remember for MLPerf v0.5 I spent at least a few days, maybe a full week debugging TEST04-B for Offline mode, which is discouraging for a new submitter when they already have everything up and running great for all of the other scenarios / tests. I will just reiterate my desire that this compliance test become more representative of the actual workload.

@nv-rborkar We don't have to implement my suggested fix, it might make more sense to discuss this further before zeroing in on a solution. The main point I was making in my suggested solution was to avoid repeating the same index through the entire query (e.g., see my other post above).

@mthmsn 

Regarding below:
> For TEST04-B, the *.Offline.performance_issue_same = 1 setting essentially is the equivalent to overriding our performance_sample_count_override value, correct? If that's true, then that seems like problematic behavior. Perhaps the QSL size for TEST04-B should not be 1, but be at least the performance_sample_count_override specified in user.conf. Loadgen could guarantee that the query size is at least several times larger than that performance_sample_count_override to ensure that any caching behavior is detected.

In TEST04-B, QSL is NOT 1. QSL size is still determined by what you set the performance_sample_count_overrride to be as it can be seen here: https://github.com/mlperf/inference/blob/master/loadgen/test_settings_internal.cc#L133-L136

`performance_sample_count` on the other hand is passed here: https://github.com/mlperf/inference/blob/master/loadgen/test_settings_internal.cc#L114-L116

So in TEST04-A and TEST04-B, QSL size is same as performance_sample_count - just that each sample in QSL is either unique (TEST04-A) or same (TEST04-B).

Re:
> It's not clear to me at the moment what underlying issue is the real culprit here, the symptoms brought up in the other github issues with regard to this may just be symptoms rather than the problem itself.

The other symptoms were TEST04-B being significantly slower than TEST04-A because accesses are from non-contiguous memory locations but that is allowed - only in the event TEST04-B is significantly fast, it might be a violation as it hint towards caching behavior which is not allowed by rules.

Re:

> The main thing that troubles me with TEST04-B is that it doesn't really reflect the actual "normal" run..

Compliance tests are desired to be different than normal runs in order to stress-test the SUT but under the right constraints. We should correct the constraints if they are wrong but I don't see an issue in the test constraints yet.

Can you share more details of the failure - such as which benchmarks does it happen for & if this failure happens when running test04-b on both reference implementation as well as your submission implementation?




any action needed here? @mthmsn 

@christ1ne yes, further action is needed. I think what would be helpful here is if someone (perhaps in MLCommons?) could run all of the compliance tests for all of the models that MLPerf provided just to flesh out for sure what is really supported by the MLPerf reference implementations.

Looking at @nv-rborkar's last question, I think it's reasonable to assume that probably no one knows yet if the reference implementation can run all of the compliance tests successfully for all of the "supported" frameworks (i.e., where a model is provided, like TF-Lite). I think there's an up-front compliance test compatibility burden here that perhaps shouldn't fall on submitters to prove out.

@mthmsn can we incorporate the WAR from you  in a PR since the compliance test does not check performance? 

@christ1ne that's fine with me. I don't currently have proper privileges in this repo to submit a PR. The code change is really minimal (so someone else can do it), but it touches `main.py`, so the change should probably exist only in a separate TEST04-B version of the `main.py` script (`main_compliance-test04b-issue-workaround.py`?). Note that this is exactly what we did for our submission, we had the "normal" version of `main.py`, and a separate TEST04-B-Offline version of `main.py` (both files included in our submission)

It involves changing these two lines of code: https://github.com/mlperf/inference/blob/58db6aa33bf9befb8b79bf28335b3295b0fcc909/vision/classification_and_detection/python/main.py#L287-L288

From this:
```
            results = self.model.predict({self.model.inputs[0]: qitem.img})
            processed_results = self.post_process(results, qitem.content_id, qitem.label, self.result_dict)
```
To this:
```
            for i in range(len(qitem.query_id)):
                results = self.model.predict({self.model.inputs[0]: qitem.img})
                processed_results.extend(self.post_process(results, qitem.content_id, qitem.label, self.result_dict))
```

Putting links to the old issue threads here for reference, so this history of this bug and the various explanations don't get overlooked:
#503 "TEST04-B break offline contiguous mode"
#520 "TEST04-B WAR in main for Offline mode to provide queries which are not from contiguous memory locations"


@georgelyuan @nv-rborkar nv-rborkar  do you want to take another look into this issue?

I will take a look at it, this week & get back.

@nv-rborkar any feedback? Thanks!

@christ1ne I have added the WAR that @mthmsn had used to test04-B folder & readme instructions in above PR

thanks @nv-rborkar. I will close this issue now and please reopen if needed @mthmsn