Repository: pytorch
Issue: TypeError: finfo(): argument 'type' (position 1) must be torch.dtype, not HFProxy
Labels: triaged,module: fx

### 🐛 Describe the bug

I suspect that https://github.com/pytorch/pytorch/pull/84011 broke some Hugging Face models tracing this way:

```
-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/Users/pbelevich/PycharmProjects/tau/pippy/utils.py", line 134, in run_worker
    run_master(pp_ranks_per_dp_group[rank], args, *extra_args)
  File "/Users/pbelevich/PycharmProjects/tau/test/local_test_forward_hf_gpt2.py", line 59, in run_master
    gpt2_pipe = Pipe.from_tracing(gpt2, MULTI_USE_PARAM_CONFIG, tracer=PiPPyHFTracer(), concrete_args=concrete_args)
  File "/Users/pbelevich/PycharmProjects/tau/pippy/IR.py", line 775, in from_tracing
    graph = _pipeline_tracer.trace(mod, **kwargs)
  File "/Users/pbelevich/PycharmProjects/tau/pippy/hf/utils.py", line 259, in trace
    graph = super().trace(*args, **kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/transformers/utils/fx.py", line 994, in trace
    self.graph = super().trace(root, concrete_args=concrete_args)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 738, in trace
    (self.create_arg(fn(*args)),),
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 906, in forward
    outputs = block(
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 716, in module_call_wrapper
    return self.call_module(mod, forward, args, kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/transformers/utils/fx.py", line 901, in call_module
    return super().call_module(m, forward, args, kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 433, in call_module
    return forward(*args, **kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 709, in forward
    return _orig_module_call(mod, *args, **kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/pbelevich/PycharmProjects/tau/pippy/IR.py", line 822, in forward
    return self.mod(*args, **kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 716, in module_call_wrapper
    return self.call_module(mod, forward, args, kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/transformers/utils/fx.py", line 901, in call_module
    return super().call_module(m, forward, args, kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 433, in call_module
    return forward(*args, **kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 709, in forward
    return _orig_module_call(mod, *args, **kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 406, in forward
    attn_outputs = self.attn(
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 716, in module_call_wrapper
    return self.call_module(mod, forward, args, kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/transformers/utils/fx.py", line 901, in call_module
    return super().call_module(m, forward, args, kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 433, in call_module
    return forward(*args, **kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 709, in forward
    return _orig_module_call(mod, *args, **kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 347, in forward
    attn_output, attn_weights = self._attn(query, key, value, attention_mask, head_mask)
  File "/Users/pbelevich/miniconda3/envs/tau/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 209, in _attn
    mask_value = torch.finfo(attn_weights.dtype).min
TypeError: finfo(): argument 'type' (position 1) must be torch.dtype, not HFProxy
```

cc @ezyang @SherlockNoMad @soumith @kwen2501 

### Versions

the bug started occurring after 1.13.0dev20220825, I blame commit https://github.com/pytorch/pytorch/commit/8b8942b11464bbe042b731bc332d65297161353a