Repository: pytorch
Issue: Testing InvokeAI 2.3.1.post1, using mps, with PyTorch nightly dev20230226 yields RuntimeError cross-device copies are not allowed!)
Labels: triaged,module: mps

### 🐛 Describe the bug

Testing InvokeAI 2.3.1.post1 with PyTorch nightly dev20230226, using mps, yields `RuntimeError: Attempting to copy from device mps:0 to device meta, but cross-device copies are not allowed!`

Comment: Posted this as [an issue](https://github.com/invoke-ai/InvokeAI/issues/2826) in InvokeAI repo. Moderator deferred this to pytorch issue.

To reproduce:

1. Install InvokeAI 2.3.1.post1
2. Install recommended models during configuration (option `r`.)
3. Run invoke.sh
4. Choose web interface (option 2: browser-based UI)
5. See console output.


Traceback:
```Traceback (most recent call last):
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/ldm/generate.py", line 889, in set_model
    model_data = cache.get_model(model_name)
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/ldm/invoke/model_manager.py", line 106, in get_model
    requested_model, width, height, hash = self._load_model(model_name)
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/ldm/invoke/model_manager.py", line 341, in _load_model
    model, width, height, model_hash = self._load_diffusers_model(mconfig)
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/ldm/invoke/model_manager.py", line 514, in _load_diffusers_model
    pipeline = StableDiffusionGeneratorPipeline.from_pretrained(
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 870, in from_pretrained
    loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2362, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 776, in __init__
    self.text_model = CLIPTextTransformer(config)
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 681, in __init__
    self.embeddings = CLIPTextEmbeddings(config)
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 209, in __init__
    self.token_embedding = nn.Embedding(config.vocab_size, embed_dim)
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 144, in __init__
    self.reset_parameters()
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 153, in reset_parameters
    init.normal_(self.weight)
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/torch/nn/init.py", line 155, in normal_
    return _no_grad_normal_(tensor, mean, std)
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/torch/nn/init.py", line 19, in _no_grad_normal_
    return tensor.normal_(mean, std)
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/torch/_refs/__init__.py", line 437, in _fn
    return fn(a, *args, out=a, **kwargs)
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/torch/_prims_common/wrappers.py", line 248, in _fn
    _safe_copy_out(copy_from=result, copy_to=out, exact_dtype=exact_dtype)  # type: ignore[arg-type]
  File "/Users/.../invokeai/.venv/lib/python3.10/site-packages/torch/_prims_common/wrappers.py", line 168, in _safe_copy_out
    raise RuntimeError(msg)
RuntimeError: Attempting to copy from device mps:0 to device meta, but cross-device copies are not allowed!```

### Versions

Collecting environment information...
PyTorch version: 2.0.0.dev20230226
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.202)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.4 (v3.10.4:9d38120e33, Mar 23 2022, 17:29:05) [Clang 13.0.0 (clang-1300.0.29.30)] (64-bit runtime)
Python platform: macOS-13.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] clip-anytorch==2.5.0
[pip3] numpy==1.23.5
[pip3] pytorch-lightning==1.7.7
[pip3] torch==2.0.0.dev20230226
[pip3] torch-fidelity==0.3.0
[pip3] torchaudio==2.0.0.dev20230226
[pip3] torchdiffeq==0.2.3
[pip3] torchmetrics==0.11.1
[pip3] torchsde==0.2.5
[pip3] torchvision==0.15.0.dev20230226
[conda] No relevant packages


cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev