Repository: mlcommons/inference
Issue: LoadGen may mishandle RNG seeds on 32-bit system
Root cause: Validation Gap

I've run our LoadGen integration on a 32-bit Raspberry Pi 4 (`armv7l`):
```
$ uname -a
Linux rpi4 4.19.66-v7l+ #1253 SMP Thu Aug 15 12:02:08 BST 2019 armv7l GNU/Linux
```
with the [RNG seeds required for v0.5](https://github.com/mlperf/inference/pull/486).

Unfortunately, the logs suggest that all the modified seeds were set to `2^32-1`:
```
$ grep rng_seed mlperf_log* -R
mlperf_log_detail.txt:"pid": 30290, "tid": 30290, "ts": 57536ns : qsl_rng_seed : 4294967295
mlperf_log_detail.txt:"pid": 30290, "tid": 30290, "ts": 57536ns : sample_index_rng_seed : 4294967295
mlperf_log_detail.txt:"pid": 30290, "tid": 30290, "ts": 57536ns : schedule_rng_seed : 4294967295
mlperf_log_detail.txt:"pid": 30290, "tid": 30290, "ts": 57536ns : accuracy_log_rng_seed : 0
mlperf_log_detail.txt:"pid": 30290, "tid": 30290, "ts": 60092ns : qsl_rng_seed : 4294967295
mlperf_log_detail.txt:"pid": 30290, "tid": 30290, "ts": 60092ns : sample_index_rng_seed : 4294967295
mlperf_log_detail.txt:"pid": 30290, "tid": 30290, "ts": 60092ns : schedule_rng_seed : 4294967295
mlperf_log_detail.txt:"pid": 30290, "tid": 30290, "ts": 60092ns : accuracy_log_rng_seed : 0
```
I don't yet know if it's simply an issue with printing, or it runs deeper.

On a 64-bit system I get the seeds printed as expected:
```
$ grep rng_seed output/tf-cpu/mobilenet/mlperf_log_detail.txt 
"pid": 26017, "tid": 26017, "ts": 15333ns : qsl_rng_seed : 3133965575612453542
"pid": 26017, "tid": 26017, "ts": 15333ns : sample_index_rng_seed : 41592772053807303
"pid": 26017, "tid": 26017, "ts": 15333ns : schedule_rng_seed : 3622009729038561421
"pid": 26017, "tid": 26017, "ts": 15333ns : accuracy_log_rng_seed : 0
"pid": 26017, "tid": 26017, "ts": 15920ns : qsl_rng_seed : 3133965575612453542
"pid": 26017, "tid": 26017, "ts": 15920ns : sample_index_rng_seed : 41592772053807303
"pid": 26017, "tid": 26017, "ts": 15920ns : schedule_rng_seed : 3622009729038561421
"pid": 26017, "tid": 26017, "ts": 15920ns : accuracy_log_rng_seed : 0
```

Here's the first 5 and last 5 queries from the 32-bit Raspberry Pi:
```
"set" : "[221,412,538,907,728,...,560,509,835,906,488]"
```
And here's the first five and last 5 queries from two different 64-bit machines:
```
"set" : "[961,348,364,445,952,...,967,913,824,565,614]"
...
"set" : "[961,348,364,445,952,...,967,913,824,565,614]"
```

Trying with smaller seeds:
```
+++ b/program/image-classification-tflite-loadgen/user.conf
@@ -3,9 +3,9 @@
 # Model maybe '*' as wildcard. In that case the value applies to all models.
 # All times are in milli seconds
 
-*.SingleStream.qsl_rng_seed = 3133965575612453542
-*.SingleStream.sample_index_rng_seed = 41592772053807303
-*.SingleStream.schedule_rng_seed = 3622009729038561421
+*.SingleStream.qsl_rng_seed = 123
+*.SingleStream.sample_index_rng_seed = 456
+*.SingleStream.schedule_rng_seed = 789
 *.SingleStream.target_latency = 10
 *.SingleStream.target_latency_percentile = 90
 *.SingleStream.min_duration = 60000
@@ -16,9 +16,9 @@ ssd-mobilenet.SingleStream.performance_sample_count_override = 256
 ssd-resnet34.SingleStream.performance_sample_count_override = 64
 gnmt.SingleStream.performance_sample_count_override = 3903900
```
gives me on:
- 64-bit:
```
"set" : "[733,472,546,675,296,...,146,690,74,197,434]"
```
- 32-bit:
```
"set" : "[733,472,546,675,296,...,146,690,74,197,434]"
```
as expected.

So it's not just printing. The traversal order is also affected.

running this with a debugger on windows ... I see issues too.
I see that we have the correct string in hand but strtoul() is returning the wrong value.
Looking.

std::mt19937 ought to be the same everywhere, but potentially std::uniform_distribution is different? If people end up with slightly different random numbers, I'm basically okay with it.

@tjablin Care to specify 32-bit seeds?

strtoul() should be strtoull() - at least that fixes it for me.

Specifying 32-bit seeds won't resolve the issue. std::mt19937 requires 64-bit seeds. Even if the inital seed is less than 2^32, the seed will be 64-bits after the first round through the pseudo-rng.

I have a PR here https://github.com/mlperf/inference/pull/489 ... I know, its a little late for v0.5.

>  Even if the inital seed is less than 2^32, the seed will be 64-bits after the first round through the pseudo-rng.

OK, maybe I got lucky with the example above.
 

> Specifying 32-bit seeds won't resolve the issue. std::mt19937 requires 64-bit seeds. Even if the inital seed is less than 2^32, the seed will be 64-bits after the first round through the pseudo-rng.

The issue is in the initial parsing of text -> number - if a `long` is only 32 bits on a given platform, `strtoul` will only output a 32-bit result

Yes,  after some debugging we found the same and #489 uses now strtoull().