Repository: pytorch/benchmark
Issue: Diagnose broken bisection workflows
Root cause: Environment Incompatibility

- [x] https://github.com/pytorch/benchmark/actions/runs/4599515585 (bisection-gh4597620224)
   https://github.com/pytorch/benchmark/actions/runs/4620952443 (Empty result)

- [x] https://github.com/pytorch/benchmark/actions/runs/4589889196 (bisection-gh4589099003)
    https://github.com/pytorch/benchmark/actions/runs/4622635280
    https://github.com/pytorch/benchmark/actions/runs/4635223730
    https://github.com/pytorch/benchmark/actions/runs/4639108972
    https://github.com/pytorch/benchmark/actions/runs/4648633190
    https://github.com/pytorch/benchmark/actions/runs/4635223730 (Empty result)

- [x] https://github.com/pytorch/benchmark/actions/runs/4584339159 (bisection-gh4583551519)
       https://github.com/pytorch/benchmark/actions/runs/4639749526 (Empty result)

- [x] https://github.com/pytorch/benchmark/actions/runs/4610516127 (bisection-gh4608671211)
      Empty result
- [x] https://github.com/pytorch/benchmark/actions/runs/4580280066 (bisection-gh4579260372)
https://github.com/pytorch/benchmark/actions/runs/4624067638
  Empty result

They seem to have the same profiler compilation error:
```
 /data/nvme/srcs/pytorch/torch/csrc/profiler/kineto_client_interface.cpp:25:8: error: ‘void torch::profiler::impl::{anonymous}::LibKinetoClient::prepare(bool, bool, bool, bool, bool)’ marked ‘override’, but does not override
   25 |   void prepare(
      |        ^~~~~~~
/data/nvme/srcs/pytorch/torch/csrc/profiler/kineto_client_interface.cpp: In constructor ‘torch::{anonymous}::RegisterLibKinetoClient::RegisterLibKinetoClient()’:
/data/nvme/srcs/pytorch/torch/csrc/profiler/kineto_client_interface.cpp:76:44: error: cannot declare variable ‘client’ to be of abstract type ‘torch::profiler::impl::{anonymous}::LibKinetoClient’
   76 |     static profiler::impl::LibKinetoClient client;
      |                                            ^~~~~~
/data/nvme/srcs/pytorch/torch/csrc/profiler/kineto_client_interface.cpp:21:7: note:   because the following virtual functions are pure within ‘torch::profiler::impl::{anonymous}::LibKinetoClient’:
   21 | class LibKinetoClient : public libkineto::ClientInterface {
      |       ^~~~~~~~~~~~~~~
In file included from /data/nvme/srcs/pytorch/third_party/kineto/libkineto/include/libkineto.h:26,
                 from /data/nvme/srcs/pytorch/torch/csrc/profiler/kineto_client_interface.cpp:2:
/data/nvme/srcs/pytorch/third_party/kineto/libkineto/include/ClientInterface.h:17:16: note: 	‘virtual void libkineto::ClientInterface::warmup(bool)’
   17 |   virtual void warmup(bool setupOpInputsCollection) = 0;
      |                ^~~~~~
/data/nvme/srcs/pytorch/third_party/kineto/libkineto/include/ClientInterface.h:20:16: note: 	‘virtual void libkineto::ClientInterface::set_withstack(bool)’
   20 |   virtual void set_withstack(bool withStack) = 0;
      |                ^~~~~~~~~~~~~
```

Wondering it could because of staled conda environment and kineto packages.

Looks like this is because checking out pytorch commits but not submodule commits, still digging.

https://github.com/pytorch/benchmark/actions/runs/4580280066 (bisection-gh4579260372)

[running] https://github.com/pytorch/benchmark/actions/runs/4624067638

https://github.com/pytorch/benchmark/actions/runs/4584339159 (bisection-gh4583551519)

[failed] https://github.com/pytorch/benchmark/actions/runs/4639749526

https://github.com/pytorch/benchmark/actions/runs/4639749526 try-2 - empty result.

New linking error: /usr/bin/ld: /data/nvme/srcs/pytorch/build/lib/libtorch_cpu.so: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
 
The reason is because somehow the conda environment installed libstdcxx-ng package, and pytorch uses it for compiling but not linking. Setting up the `LD_LIBRARY_PATH` and `LIBRARY_PATH` environment variables to `$CONDA_PREFIX/lib` fixes the problem.

All issue has been fixed, closed.