Repository: pytorch
Issue: [primTorch] `|` operator does not work with FakeTensor in _refs
Labels: feature,triaged,module: meta tensors,module: primTorch

### 🐛 Describe the bug

Change this line:
https://github.com/pytorch/pytorch/blob/9ca561cbd4ab2bee3279b998e3ea4d1f302ced7e/torch/_refs/__init__.py#L1187
to
```python
a | b
```

And you will see some failures in `nan_to_num`. Example failure:

```
======================================================================
ERROR: test_python_ref_meta__refs_nan_to_num_cpu_int16 (__main__.TestCommonCPU)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 377, in instantiated_test
    result = test(self, **param_kwargs)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 813, in test_wrapper
    return test(*args, **kwargs)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 1006, in only_fn
    return fn(self, *args, **kwargs)
  File "/home/gaoxiang/pytorch-prims/test/test_ops.py", line 380, in test_python_ref_meta
    meta_result = op(meta_sample.input, *meta_sample.args, **meta_sample.kwargs)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/testing/_internal/common_methods_invocations.py", line 952, in __call__
    return self.op(*args, **kwargs)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/_prims/wrappers.py", line 195, in _fn
    result = fn(*args, **kwargs)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/_prims/wrappers.py", line 114, in _fn
    result = fn(**bound.arguments)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/_refs/__init__.py", line 609, in nan_to_num
    is_neginf = bitwise_and(isinf(a), is_neg)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/_prims/wrappers.py", line 195, in _fn
    result = fn(*args, **kwargs)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/_prims/wrappers.py", line 265, in _fn
    return fn(*args, **kwargs)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/_prims/wrappers.py", line 114, in _fn
    result = fn(**bound.arguments)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/_refs/__init__.py", line 348, in _ref
    return prim(a)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/_refs/__init__.py", line 506, in isinf
    return logical_not(logical_or(isnan(a), isfinite(a)))
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/_prims/wrappers.py", line 195, in _fn
    result = fn(*args, **kwargs)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/_prims/wrappers.py", line 114, in _fn
    result = fn(**bound.arguments)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/_refs/__init__.py", line 754, in _ref
    return prim(a, b)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/_refs/__init__.py", line 1187, in _logical_or
    return a | b
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/utils/_python_dispatch.py", line 74, in wrapped
    return f(self, *args, **kwargs)
  File "/home/gaoxiang/.local/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 482, in __torch_dispatch__
    raise Exception(
Exception: Invoking operators with non-Fake Tensor inputs in FakeTensorMode is not yet supported. Please convert all Tensors to FakeTensors first. Found in aten.bitwise_or.Tensor

```

### Versions

```
Collecting environment information...
PyTorch version: 1.13.0a0+gitd8d8a48
Is debug build: True
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Arch Linux (x86_64)
GCC version: (GCC) 12.1.0
Clang version: 14.0.6
CMake version: version 3.23.2
Libc version: glibc-2.35

Python version: 3.10.5 (main, Jun  6 2022, 18:49:26) [GCC 12.1.0] (64-bit runtime)
Python platform: Linux-5.18.9-arch1-1-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.64
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 2080 Ti

Nvidia driver version: 515.57
cuDNN version: Probably one of the following:
/usr/lib/libcudnn.so.8.4.1
/usr/lib/libcudnn_adv_infer.so.8.4.1
/usr/lib/libcudnn_adv_train.so.8.4.1
/usr/lib/libcudnn_cnn_infer.so.8.4.1
/usr/lib/libcudnn_cnn_train.so.8.4.1
/usr/lib/libcudnn_ops_infer.so.8.4.1
/usr/lib/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.13.0a0+gitd8d8a48
[pip3] torch-ucc==1.0.0
[pip3] torchani==2.2
[pip3] torchvision==0.2.2.post3
[conda] Could not collect
```

cc @ezyang @mruberry @ngimel