Repository: tensorflow
Issue: OMP_PROC_BIND or OMP_PLACES either ignored or respected incorrectly
Labels: stat:awaiting tensorflower,type:bug,type:performance,TF 2.11

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

2.11

### Custom Code

Yes

### OS Platform and Distribution

_No response_

### Mobile device

_No response_

### Python version

3.10.4

### Bazel version

5.1.1

### GCC/Compiler version

11.3.0

### CUDA/cuDNN version

N/A

### GPU model and memory

N/A

### Current Behaviour?

When I run TensorFlow on CPU and try to enable core binding with `OMP_PROC_BIND=close`, all threads get bound to core 0 (rather than thread 0 to core 0, thread 1 to core 1, etc).

Expected output:

```
$ OMP_PROC_BIND=true OMP_PLACES=cores python tf_example.py
Inter_op_threads: 1
Intra_op_threads: 4
Thread count: 4
Child affinity is {96}
Child affinity is {97}
Child affinity is {98}
Child affinity is {99}
```
I.e. I'd expect four threads, bound to subsequent cores. Now, I guess TensorFlow simply uses some threads for management of the framework. Though I'm surprised by the large number of threads, 11 extra threads on top of the Intra_op_thread count, see the log output for the custom TF-2.11 case, this is not really an 'issue' (although one might wonder how management threads ought to behave, they should probably remain unbound even if the compute threads are bound 1 per core).

What I'm seeing however is that with our custom built TF-2.11, all threads are bound to the first core in my cgroup (core ID 96 in this case). For tf-nightly, all binding is completely ignored.

Not sure if it's useful, but the Custom built had this build command:

<details>
<summary>build command</summary>

```
bazel --output_user_root=/tmp/jenkins/build/TensorFlow/2.11.0/foss-2022a/TensorFlow/bazel-root --local_startup_timeout_secs=300 --host_jvm_args=-Xms512m --host_jvm_args=-Xmx4096m build --config=noaws --config=nogcp --config=nohdfs --compilation_mode=opt --config=opt --subcommands --verbose_failures --jobs=128 --copt="-fPIC" --distinct_host_configuration=false --action_env=CPATH='/sw/arch/RHEL8/EB_production/2022/software/cURL/7.83.0-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/double-conversion/3.2.0-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/flatbuffers/2.0.7-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/giflib/5.2.1-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/hwloc/2.7.1-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/ICU/71.1-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/JsonCpp/1.9.5-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/libjpeg-turbo/2.1.3-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/libpng/1.6.37-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/LMDB/0.9.29-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/nsync/1.25.0-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/protobuf/3.19.4-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/pybind11/2.9.2-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/snappy/1.1.9-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/SQLite/3.38.3-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/zlib/1.2.12-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/OpenSSL/1.1/include' --host_action_env=CPATH='/sw/arch/RHEL8/EB_production/2022/software/cURL/7.83.0-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/double-conversion/3.2.0-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/flatbuffers/2.0.7-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/giflib/5.2.1-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/hwloc/2.7.1-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/ICU/71.1-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/JsonCpp/1.9.5-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/libjpeg-turbo/2.1.3-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/libpng/1.6.37-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/LMDB/0.9.29-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/nsync/1.25.0-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/protobuf/3.19.4-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/pybind11/2.9.2-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/snappy/1.1.9-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/SQLite/3.38.3-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/zlib/1.2.12-GCCcore-11.3.0/include:/sw/arch/RHEL8/EB_production/2022/software/OpenSSL/1.1/include' --action_env=LIBRARY_PATH='/sw/arch/RHEL8/EB_production/2022/software/cURL/7.83.0-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/double-conversion/3.2.0-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/flatbuffers/2.0.7-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/giflib/5.2.1-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/hwloc/2.7.1-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/ICU/71.1-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/JsonCpp/1.9.5-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/libjpeg-turbo/2.1.3-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/libpng/1.6.37-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/LMDB/0.9.29-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/nsync/1.25.0-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/protobuf/3.19.4-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/pybind11/2.9.2-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/snappy/1.1.9-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/SQLite/3.38.3-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/zlib/1.2.12-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/OpenSSL/1.1/lib' --host_action_env=LIBRARY_PATH='/sw/arch/RHEL8/EB_production/2022/software/cURL/7.83.0-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/double-conversion/3.2.0-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/flatbuffers/2.0.7-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/giflib/5.2.1-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/hwloc/2.7.1-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/ICU/71.1-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/JsonCpp/1.9.5-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/libjpeg-turbo/2.1.3-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/libpng/1.6.37-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/LMDB/0.9.29-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/nsync/1.25.0-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/protobuf/3.19.4-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/pybind11/2.9.2-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/snappy/1.1.9-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/SQLite/3.38.3-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/zlib/1.2.12-GCCcore-11.3.0/lib:/sw/arch/RHEL8/EB_production/2022/software/OpenSSL/1.1/lib' --action_env=PYTHONNOUSERSITE='1' --host_action_env=PYTHONNOUSERSITE='1' --action_env=PYTHONPATH --host_action_env=PYTHONPATH  //tensorflow/tools/pip_package:build_pip_package
```

</details>

I'm not 100% sure of all the details of the custom build, I used EasyBuild to build TF from sources, and the build recipy wasn't made by me.

I don't understand enough of the TensorFlow threading model to know how to debug this issue. My specific questions would be:

- Why does my custom TF 2.11 build bind all threads to one core? Other GOMP-based packages (e.g. `scipy`) do show correct binding on my system, with the same (OMP_) environment variables.
- Why does tf-nightly not bind threads at all?

Any general explanations of which potential threading models can be used in TF are also welcome.

### Standalone code to reproduce the issue

```shell
import tensorflow as tf

tf.config.threading.set_inter_op_parallelism_threads(1)
tf.config.threading.set_intra_op_parallelism_threads(4)

print(f"Inter_op_threads: {tf.config.threading.get_inter_op_parallelism_threads()}")
print(f"Intra_op_threads: {tf.config.threading.get_intra_op_parallelism_threads()}")

A = tf.random.normal([20000,20000])
for i in range(0,1):
    B = tf.multiply(A,A)

import psutil
import os

current_process = psutil.Process()
threads = current_process.threads()
print(f"Thread count: {len(threads)}")
for thread in threads:
    print('Child affinity is {}'.format(os.sched_getaffinity(thread.id)))
```


### Relevant log output

```shell
# output for custom build TF-2.11
$ OMP_PROC_BIND=true OMP_PLACES=cores python tf_example.py
Inter_op_threads: 1
Intra_op_threads: 4
Thread count: 15
Child affinity is {96}
Child affinity is {96}
Child affinity is {96}
Child affinity is {96}
Child affinity is {96}
Child affinity is {96}
Child affinity is {96}
Child affinity is {96}
Child affinity is {96}
Child affinity is {96}
Child affinity is {96}
Child affinity is {96}
Child affinity is {96}
Child affinity is {96}
Child affinity is {96}

# output for tf-nightly
# Not sure what threading model tf-nightly uses, so I've set both OMP and KMP variables:
$ OMP_PROC_BIND=true OMP_PLACES=cores KMP_AFFINITY=granularity=fine,verbose,compact,1,0 python tf_example.py

Inter_op_threads: 1
Intra_op_threads: 4
Thread count: 14
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}
Child affinity is {96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}


# Note that this is generated on an HPC system in which I'm in a CGROUP with access to core 96-127, hence the core IDs start at 96.
```
</details>