Repository: pytorch
Issue: C++ Extensions can't import c10d/reducer.hpp
Labels: oncall: distributed

### 🐛 Describe the bug

Hi, I was hoping to subclass the Reducer class and add custom functionality to how it buckets gradients and came across the C++ extensions functionality. I followed the guide here: https://pytorch.org/tutorials/advanced/cpp_extension.html#writing-a-c-extension, which compiles fine in my container. 

It however does not compile when I add `#include <c10d/reducer.hpp>` to the top. `#include <c10d/ProcessGroup.hpp>` compiles fine, but including the Reducer header file shows this error:

```
root@GCR-OPENPAI-22:/home/----/lltm# /usr/src/Python-3.8.0/python setup.py install
running install
running bdist_egg
running egg_info
writing lltm_cpp.egg-info/PKG-INFO
writing dependency_links to lltm_cpp.egg-info/dependency_links.txt
writing top-level names to lltm_cpp.egg-info/top_level.txt
reading manifest file 'lltm_cpp.egg-info/SOURCES.txt'
writing manifest file 'lltm_cpp.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'lltm_cpp' extension
Emitting ninja build file /home/----/lltm/build/temp.linux-x86_64-3.8/build.ninja...
Compiling objects...
Using envvar MAX_JOBS (32) as the number of workers...
[1/1] c++ -MMD -MF /home/----/lltm/build/temp.linux-x86_64-3.8/lltm.o.d -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/root/.local/lib/python3.8/site-packages/torch/include -I/root/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/root/.local/lib/python3.8/site-packages/torch/include/TH -I/root/.local/lib/python3.8/site-packages/torch/include/THC -I/usr/src/Python-3.8.0/Include -I/usr/src/Python-3.8.0 -c -c /home/----/lltm/lltm.cpp -o /home/----/lltm/build/temp.linux-x86_64-3.8/lltm.o -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1013"' -DTORCH_EXTENSION_NAME=lltm_cpp -D_GLIBCXX_USE_CXX11_ABI=1 -std=c++14
FAILED: /home/----/lltm/build/temp.linux-x86_64-3.8/lltm.o 
c++ -MMD -MF /home/----/lltm/build/temp.linux-x86_64-3.8/lltm.o.d -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/root/.local/lib/python3.8/site-packages/torch/include -I/root/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/root/.local/lib/python3.8/site-packages/torch/include/TH -I/root/.local/lib/python3.8/site-packages/torch/include/THC -I/usr/src/Python-3.8.0/Include -I/usr/src/Python-3.8.0 -c -c /home/----/lltm/lltm.cpp -o /home/----/lltm/build/temp.linux-x86_64-3.8/lltm.o -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1013"' -DTORCH_EXTENSION_NAME=lltm_cpp -D_GLIBCXX_USE_CXX11_ABI=1 -std=c++14
In file included from /home/----/lltm/lltm.cpp:8:
/root/.local/lib/python3.8/site-packages/torch/include/c10d/reducer.hpp:23:10: fatal error: torch/csrc/distributed/autograd/context/context.h: No such file or directory
   23 | #include <torch/csrc/distributed/autograd/context/context.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/root/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1896, in _run_ninja_build
    subprocess.run(
  File "/usr/src/Python-3.8.0/Lib/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v', '-j', '32']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "setup.py", line 4, in <module>
    setup(name='lltm_cpp',
  File "/usr/local/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/src/Python-3.8.0/Lib/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/src/Python-3.8.0/Lib/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/src/Python-3.8.0/Lib/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/usr/src/Python-3.8.0/Lib/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/src/Python-3.8.0/Lib/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 172, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 158, in call_command
    self.run_command(cmdname)
  File "/usr/src/Python-3.8.0/Lib/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/src/Python-3.8.0/Lib/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/usr/src/Python-3.8.0/Lib/distutils/command/install_lib.py", line 107, in build
    self.run_command('build_ext')
  File "/usr/src/Python-3.8.0/Lib/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/src/Python-3.8.0/Lib/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 84, in run
    _build_ext.run(self)
  File "/usr/src/Python-3.8.0/Lib/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/root/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 839, in build_extensions
    build_ext.build_extensions(self)
  File "/usr/src/Python-3.8.0/Lib/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/usr/src/Python-3.8.0/Lib/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 205, in build_extension
    _build_ext.build_extension(self, ext)
  File "/usr/src/Python-3.8.0/Lib/distutils/command/build_ext.py", line 528, in build_extension
    objects = self.compiler.compile(sources,
  File "/root/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 654, in unix_wrap_ninja_compile
    _write_ninja_file_and_compile_objects(
  File "/root/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1569, in _write_ninja_file_and_compile_objects
    _run_ninja_build(
  File "/root/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1912, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension
```

The files are as follows: 

reducer_subclass.cpp
```
#include <c10d/reducer.hpp>
```

setup.py
```
from setuptools import setup, Extension
from torch.utils import cpp_extension

setup(name='reducer_subclass_cpp',
      ext_modules=[cpp_extension.CppExtension('reducer_subclass_cpp', ['reducer_subclass.cpp'])],
      cmdclass={'build_ext': cpp_extension.BuildExtension})
```

And to get the error, run `python setup.py install`

This is run in a container with python3.8 installed from source (with very slight modification to its shared_memory module). 

Am I importing this the wrong way, or is it just not supported? If it's not supported, is there another way I can go about doing this? 

Thanks for the help!

### Versions

```
root@GCR-OPENPAI-22:/home/----# /usr/src/Python-3.8.0/python collect_env.py 
Collecting environment information...
PyTorch version: 1.13.0a0+git46a6a50
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 5.1.20531-cacfa990

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 14.0.0 (https://github.com/RadeonOpenCompute/llvm-project roc-5.1.1 22114 5cba46feb6af367b1cafaa183ec42dbfb8207b14)
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.0 (default, Sep 22 2022, 22:16:25)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.13.0-1031-azure-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: 
GPU models and configuration: 
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 5.1.20531
MIOpen runtime version: 2.16.0
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.3
[pip3] torch==1.13.0a0+git46a6a50
[pip3] torchvision==0.13.1
[conda] mkl                       2022.0.1           h06a4308_117  
[conda] mkl-include               2022.0.1           h06a4308_117  
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] torch                     1.8.1+96f20ce            pypi_0    pypi
[conda] torchvision               0.9.0a0+8fb5838          pypi_0    pypi
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu