Repository: pytorch
Issue: [torch.compile] Integers stored on nn.Modules as dynamic causing errors
Labels: high priority,triaged,actionable,oncall: pt2,module: dynamic shapes

### 🐛 Describe the bug

reporduce code
```python
from transformers import BloomModel, BloomConfig
import json
import torch

CONFIG = """
{
  "apply_residual_connection_post_layernorm": false,
  "architectures": [
    "BloomModel"
  ],
  "attention_dropout": 0.0,
  "attention_softmax_in_fp32": true,
  "bias_dropout_fusion": true,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_dropout": 0.0,
  "hidden_size": 1536,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "masked_softmax_fusion": true,
  "model_type": "bloom",
  "n_head": 16,
  "n_inner": null,
  "n_layer": 24,
  "offset_alibi": 100,
  "pad_token_id": 3,
  "pretraining_tp": 1,
  "seq_length": 2048,
  "skip_bias_add": true,
  "skip_bias_add_qkv": false,
  "slow_but_exact": false,
  "torch_dtype": "float32",
  "transformers_version": "4.29.2",
  "unk_token_id": 0,
  "use_cache": false,
  "vocab_size": 250880
}
"""

config = BloomConfig(**json.loads(CONFIG))
config.n_layer = 1
model = BloomModel(config).half().cuda().eval()

x = torch.randint(100, config.vocab_size, (1, 20)).cuda()

with torch.no_grad():
    ref_out = model(x)[0]

    compiled_fn = torch.compile(model, dynamic=True)
    out = compiled_fn(x)[0]
```
The above code is fine on 2.3, but it will throw an error on 2.4 and the main branch.
What is error looks like:
```
def triton_(in_ptr0, in_ptr1, out_ptr0, ks0, ks1, ks2, xnumel, XBLOCK : tl.constexpr):
    xoffset = tl.program_id(0) * XBLOCK
    xindex = xoffset + tl.arange(0, XBLOCK)[:]
    xmask = xindex < xnumel
    x1 = (xindex // ks2)
    x0 = xindex % ks2
    x2 = xindex
    tmp5 = tl.load(in_ptr0 + (x0), xmask, eviction_policy='evict_last')
    tmp8 = tl.load(in_ptr1 + (x0), xmask, eviction_policy='evict_last')
    tmp0 = (ks1.to(tl.float64)) ** ((-1)*((ks0.to(tl.float64)) ** (-1.00000000000000)))
                                          ^
AttributeError("'tensor' object has no attribute '__pow__'")

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
```
This seems to be a problem with inductor or triton, but it is actually because *self.num_heads* is dynamic.
This phenomenon occurred after this commit: https://github.com/pytorch/pytorch/issues/115711

The problem is that torch made all ints in nn.Module dynamic, but this is actually not necessary. I understand that there will be an automatic dynamic mechanism for this in the future, but for the moment if I can have a simple way to avoid the problems.


### Versions

```
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Alibaba Cloud Linux release 3 (Soaring Falcon)  (x86_64)
GCC version: (GCC) 10.2.1 20200825 (Alibaba 10.2.1-3.6 2.32)
Clang version: Could not collect
CMake version: version 3.26.5
Libc version: glibc-2.32

Python version: 3.10.14 (main, May  6 2024, 19:42:50) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.10.134-16.1.al8.x86_64-x86_64-with-glibc2.32
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A10
Nvidia driver version: 550.54.15
cuDNN version: Probably one of the following:
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn.so.8.9.7
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.9.7
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.9.7
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.9.7
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.9.7
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.9.7
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.9.7
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
架构：           x86_64
CPU 运行模式：   32-bit, 64-bit
字节序：         Little Endian
CPU:             48
在线 CPU 列表：  0-47
每个核的线程数： 2
每个座的核数：   24
座：             1
NUMA 节点：      1
厂商 ID：        GenuineIntel
BIOS Vendor ID:  Alibaba Cloud
CPU 系列：       6
型号：           106
型号名称：       Intel(R) Xeon(R) Platinum 8369B CPU @ 2.90GHz
BIOS Model name: pc-i440fx-2.1
步进：           6
CPU MHz：        2899.998
BogoMIPS：       5799.99
超管理器厂商：   KVM
虚拟化类型：     完全
L1d 缓存：       48K
L1i 缓存：       32K
L2 缓存：        1280K
L3 缓存：        49152K
NUMA 节点0 CPU： 0-47
标记：           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves wbnoinvd arat avx512vbmi avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid fsrm arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] optree==0.12.1
[pip3] torch==2.4.0
[pip3] triton==3.0.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] optree                    0.12.1                   pypi_0    pypi
[conda] torch                     2.4.0                    pypi_0    pypi
[conda] triton                    3.0.0                    pypi_0    pypi

```

cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @chauhang @penguinwu