Repository: tensorflow
Issue: `tensorflow/python/compiler/xla/jit_test` fails on PPC with $HOME unset
Labels: stat:awaiting tensorflower,type:bug,type:build/install,TF 2.13

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

tf 2.13.0

### Custom code

Yes

### OS platform and distribution

_No response_

### Mobile device

_No response_

### Python version

3.11

### Bazel version

6.3.1

### GCC/compiler version

12.3

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

Running the test `//tensorflow/python/compiler/xla:jit_test_cpu` through Bazels build system with a pre-installed numpy (build from source) yields a segmentation fault I cannot explain.

It happened after upgrading our toolchain from GCC 11 & Python 3.10 to GCC 12 and Python 3.11 but includes also a couple other software versions which changed but trying to narrow them down wasn't successful. But seemingly only Python and OpenSSL (instead of BoringSSL which doesn't work on PPC) as well as the mentioned numpy should be involved.

After a lot of digging I found that omitting `$HOME` when executing the test (which Bazel does) triggers the segmentation fault.   
Adding `--test_env HOME=/non-existing` works around this.

I further traced it to `testJITCreateOpsLambda` in particular keeping only the single `compute` call at https://github.com/tensorflow/tensorflow/blob/v2.13.0/tensorflow/python/compiler/xla/jit_test.py#L76 is enough while `self.compute(False, create_ops)` works which means it is related to XLA compilation.

Also replacing [`random_uniform`](https://github.com/tensorflow/tensorflow/blob/v2.13.0/tensorflow/python/compiler/xla/jit_test.py#L70) by `constant_op.constant(1)` avoids the error, so it is related to how that call is compiled/run but I failed to further follow how that happens

I'm at loss how to debug this further and whether this is an issue with TensorFlows XLA compilation or a bug elsewhere only triggered by that special environment

### Standalone code to reproduce the issue

```shell
`bazel test --config=noaws --config=nogcp --config=nohdfs --compilation_mode=opt --config=opt --copt="-fPIC" --action_env=CPATH='/sw/installed/OpenSSL/1.1/include' --host_action_env=CPATH='/sw/installed/OpenSSL/1.1/include' --action_env=LIBRARY_PATH='/sw/installed/OpenSSL/1.1/lib' --host_action_env=LIBRARY_PATH='/sw/installed/OpenSSL/1.1/lib' --action_env=PYTHONPATH --host_action_env=PYTHONPATH --action_env PYTHON_BIN_PATH --action_env PYTHON_LIB_PATH --python_path=$(which python)  -- //tensorflow/python/compiler/xla:jit_test_cpu`

or reduced to the actual invocation after bazel failed:

`export PYTHONPATH=bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/com_google_protobuf/python:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/tblib_archive/src:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/absl_py:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/astunparse_archive:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/com_google_protobuf:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/dill_archive:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/gast_archive:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/opt_einsum_archive:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/six_archive:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/tblib_archive:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/termcolor_archive:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/typing_extensions_archive:bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/wrapt:$PYTHONPATH
(cd /dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/bazel-root/9c88b62e77874bb73aea75868f86ebae/execroot/org_tensorflow && \
cd - &&
  exec env - \
    LD_LIBRARY_PATH=$LD_LIBRARY_PATH \
    PATH=$PATH \
    PYTHONNOUSERSITE=1 \
    PYTHONPATH=$PYTHONPATH \
    HOME2='/fake' \
python bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/compiler/xla/jit_test.py)`
```


### Relevant log output

```shell
Running tests under Python 3.11.3: /beegfs/ws/1/s3248973-EasyBuild/easybuild-ml/software/Python/3.11.3-GCCcore-12.3.0/bin/python
[ RUN      ] JITTest.testJITCreateOpsLambda
2023-09-08 10:32:04.231849: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled
2023-09-08 10:32:04.404724: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x201108083080 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2023-09-08 10:32:04.404765: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2023-09-08 10:32:04.603193: I ./tensorflow/compiler/jit/device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
Fatal Python error: Segmentation fault

Thread 0x0000200000048800 (most recent call first):
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1455 in _call_tf_sessionrun
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1362 in _run_fn
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1379 in _do_call
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1372 in _do_run
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1192 in _run
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 969 in run
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2059 in run
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/compiler/xla/jit_test.py", line 55 in compute
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/compiler/xla/jit_test.py", line 81 in testJITCreateOpsLambda
  File "/beegfs/ws/1/s3248973-EasyBuild/easybuild-ml/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 579 in _callTestMethod
  File "/beegfs/ws/1/s3248973-EasyBuild/easybuild-ml/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 623 in run
  File "/beegfs/ws/1/s3248973-EasyBuild/easybuild-ml/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/case.py", line 678 in __call__
  File "/beegfs/ws/1/s3248973-EasyBuild/easybuild-ml/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/suite.py", line 122 in run
  File "/beegfs/ws/1/s3248973-EasyBuild/easybuild-ml/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/suite.py", line 84 in __call__
  File "/beegfs/ws/1/s3248973-EasyBuild/easybuild-ml/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/suite.py", line 122 in run
  File "/beegfs/ws/1/s3248973-EasyBuild/easybuild-ml/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/suite.py", line 84 in __call__
  File "/beegfs/ws/1/s3248973-EasyBuild/easybuild-ml/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/runner.py", line 217 in run
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/absl_py/absl/testing/_pretty_print_reporter.py", line 86 in run
  File "/beegfs/ws/1/s3248973-EasyBuild/easybuild-ml/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/main.py", line 274 in runTests
  File "/beegfs/ws/1/s3248973-EasyBuild/easybuild-ml/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/unittest/main.py", line 102 in __init__
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537 in _run_and_get_tests_result
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568 in run_tests
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156 in _run_in_app
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049 in main
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51 in g_main
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/absl_py/absl/app.py", line 258 in _run_main
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/absl_py/absl/app.py", line 312 in run
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60 in main_wrapper
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489 in benchmarks_main
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62 in main
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56 in main
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/compiler/xla/jit_test.py", line 321 in <module>

Extension modules: google.protobuf.pyext._message, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, tensorflow.python.framework.fast_tensor_util (total: 15)
*** Received signal 11 ***
*** BEGIN MANGLED STACK TRACE ***
/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/../../../_solib_ppc/_U_S_Stensorflow_Clibtensorflow_Uframework_Uimport_Ulib___Utensorflow/libtensorflow_framework.so.2(+0xd76744)[0x200b0de86744]
[0x2000000504d8]
/beegfs/ws/1/s3248973-EasyBuild/easybuild-ml/software/Python/3.11.3-GCCcore-12.3.0/lib/libpython3.11.so.1.0(+0x14090c)[0x2000001b090c]
[0x2000000504d8]
/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/../../libtensorflow_cc.so.2(_ZN3xla3cpu13CpuExecutable22ExecuteComputeFunctionEPKNS_20ExecutableRunOptionsEN4absl12lts_202301254SpanIKNS_23MaybeOwningDeviceMemoryEEEPNS_19HloExecutionProfileE+0x150)[0x200b15956c40]
/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/../../libtensorflow_cc.so.2(+0x5be7c04)[0x200b15957c04]
/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/../../libtensorflow_cc.so.2(_ZN15stream_executor4host10HostStream8WorkLoopEv+0x1ac)[0x200b1d822b0c]
/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/../../libtensorflow_cc.so.2(+0xdab3274)[0x200b1d823274]
/dev/shm/s3248973-EasyBuild/TensorFlow/2.13.0/foss-2022b/TensorFlow/tensorflow-2.13.0/bazel-out/ppc-opt/bin/tensorflow/python/compiler/xla/jit_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/../../../_solib_ppc/_U_S_Stensorflow_Clibtensorflow_Uframework_Uimport_Ulib___Utensorflow/libtensorflow_framework.so.2(+0xd85088)[0x200b0de95088]
/lib64/libpthread.so.0(+0x8b94)[0x2000008c8b94]
/lib64/libc.so.6(clone+0xe4)[0x200000a585f4]
*** END MANGLED STACK TRACE ***

*** Begin stack trace ***
	tsl::CurrentStackTrace[abi:cxx11]()
	
	__kernel_sigtramp_rt64
	
	__kernel_sigtramp_rt64
	xla::cpu::CpuExecutable::ExecuteComputeFunction(xla::ExecutableRunOptions const*, absl::lts_20230125::Span<xla::MaybeOwningDeviceMemory const>, xla::HloExecutionProfile*)
	
	stream_executor::host::HostStream::WorkLoop()
	
	
	
	clone
*** End stack trace ***
```