Repository: transformers
Issue: A warning message showing that `MultiScaleDeformableAttention.so` is not found in `/root/.cache/torch_extensions` if `ninja` is installed with `transformers`
Labels: bug

### System Info

* `transformers`: `4.47.1`
* `torch`: `2.5.1`
* `timm`: `1.0.12`
* `ninja`: `1.11.1.3`
* `python`: `3.10.14`
* `pip`: `23.0.1`
* CUDA runtime installed by `torch`: `nvidia-cuda-runtime-cu12==12.4.127`
* OS (in container): Debian GNU/Linux 12 (bookworm)
* OS (native device): Windows 11 Enterprise 23H2 (`10.0.22631 Build 22631`)
* Docker version: `27.3.1, build ce12230`
* NVIDIA Driver: `565.57.02`


### Who can help?

I am asking help for [`DeformableDetrModel`](https://huggingface.co/docs/transformers/v4.47.1/en/model_doc/deformable_detr#transformers.DeformableDetrModel)
vision models: @amyeroberts, @qubvel

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Start a new docker container by
    ```sh
    docker run --gpus all -it --rm --shm-size=1g python:3.10-slim bash
    ```
2. Install dependencies
    ```sh
    pip install transformers[torch] requests pillow timm
    ```
3. Run the following script (copied from [the document](https://huggingface.co/docs/transformers/v4.47.1/en/model_doc/deformable_detr#transformers.DeformableDetrModel.forward.example)), it works fine and does not show any message.
    ```python
    from transformers import AutoImageProcessor, DeformableDetrModel
    from PIL import Image
    import requests
    
    url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    image = Image.open(requests.get(url, stream=True).raw)
    
    image_processor = AutoImageProcessor.from_pretrained("SenseTime/deformable-detr")
    model = DeformableDetrModel.from_pretrained("SenseTime/deformable-detr")
    
    inputs = image_processor(images=image, return_tensors="pt")
    
    outputs = model(**inputs)
    
    last_hidden_states = outputs.last_hidden_state
    list(last_hidden_states.shape)

    ```
4.  Install ninja:
    ```sh
    pip install ninja
    ```
5.  Run [the same script](https://huggingface.co/docs/transformers/v4.47.1/en/model_doc/deformable_detr#transformers.DeformableDetrModel.forward.example) again, this time, the following warning messages will show
    ```text
        
                                   !! WARNING !!
    
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    Your compiler (c++) is not compatible with the compiler Pytorch was
    built with for this platform, which is g++ on linux. Please
    use g++ to to compile your extension. Alternatively, you may
    compile PyTorch from source using c++, and then you can also use
    c++ to compile your extension.
    
    See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help
    with compiling PyTorch from source.
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    
                                  !! WARNING !!
    
      warnings.warn(WRONG_COMPILER_WARNING.format(
    Could not load the custom kernel for multi-scale deformable attention: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
    Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
    Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
    Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
    Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
    Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
    Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
    Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
    Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
    Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
    Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
    Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
    ```
    Certainly, `/root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/` is empty.

The issue happens only when both `ninja` and `transformers` are installed. I believe that the following issue may be related to this issue:

https://app.semanticdiff.com/gh/huggingface/transformers/pull/32834/overview


### Expected behavior

It seems that ninja will let `DeformableDetrModel` throw unexpected error messages (despite that the script still works). That's may be because I am using a container without any compiler or CUDA preinstalled (the CUDA run time is installed by `pip`).

I think there should be a check that automatically turn of the `ninja` related functionalities even if `ninja` is installed by `pip`, as long as the requirements like compiler version, CUDA path, or something, are not fulfilled.