Repository: mlcommons/inference
Issue: Crash in loadgen
Root cause: Validation Gap

https://github.com/mlperf/inference/blob/master/loadgen/logging.cc#L363

/usr/include/c++/10/bits/atomic_base.h:423: std::__atomic_base<_IntTp>::__int_type std::__atomic_base<_IntTp>::load(std::memory_order) const [with _ITp = long int; std::__atomic_base<_IntTp>::__int_type = long int; std::memory_order = std::memory_order]: Assertion '__b != memory_order_release' failed.

The C++ spec says:

order must be one of std::memory_order_relaxed, std::memory_order_consume, std::memory_order_acquire or std::memory_order_seq_cst. Otherwise the behavior is undefined.

I hit this issue on Fedora 31 and 32 because:

$ python -c "import sysconfig; print(sysconfig.get_config_var('CFLAGS'))"

-Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 **-Wp,-D_GLIBCXX_ASSERTIONS** -fexceptions -fstack-protector-strong -grecord-gcc-switches   -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv

By default, it defines "_GLIBCXX_ASSERTIONS", which will turn on libstdc++ assertions.

But on Ubuntu 19.04, the output is different:

-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g   -fstack-protector-strong -Wformat -Werror=format-security  -g -flto -fuse-linker-plugin -ffat-lto-objects

I think that's why most of you didn't hit this issue before.

See: https://fedoraproject.org/wiki/Changes/HardeningFlags28



@snnn  do you have a proposed resolution?

Change the line 363 to something like:

```
  return max_latency_.load(std::memory_order_acquire);
```

std::memory_order_release is not ok.


@nvpohanh @tjablin any thoughts on this?

definitely an issue. I think windows debug builds have been complaining about this as well.

I am not familiar with this part of the code. @tjablin would be the right person to ask

> Change the line 363 to something like:
> 
> ```
>   return max_latency_.load(std::memory_order_acquire);
> ```
> 
> std::memory_order_release is not ok.

Do you know the reason? Why is memory_order_release not ok.

According to C++ standard, it is not allowed.

@briandersn Could you help to reply the question?

I do see that `memory_order_release` is not allowed in atomic load(): https://en.cppreference.com/w/cpp/atomic/atomic/load

As the model of x86 atomic is accquire-release, so the atomic release load is correct, and won't assert.
But for other architecture, atomic release order is for write. when atomic load is needed, the a order of accquire is needed fisrt to make sure the before release write is done and the result is synced from the write core to the load core.

so the below code should be a bug , really need change from memory_order_release to **memory_order_acq_rel**

QuerySampleLatency AsyncLog::GetMaxLatencySoFar() {
  return max_latency_.load(std::**memory_order_release**);
}


Just had occasion to test this today

**memory_order_acq_rel** causes the same assertion failure

**memory_order_acquire** did let it run to completion

Looks like @tjablin fixed this in #878. Can this be closed now?

Yeap, should be good.