Repository: pytorch
Issue: App crashes when I attempt to run it with an iOS xcframework that relies on the LibTorch-Lite CocoaPod
Labels: module: binaries,triaged,module: ios

### 🐛 Describe the bug

I posted this issue on [stackoverflow.com](https://stackoverflow.com/questions/74730081/app-crashes-when-i-attempt-to-run-it-with-an-ios-xcframework-that-relies-on-the), I am posting it here again.

1. MyLibrary: CocoaPod project that I am trying to install and run but failing. (https://github.com/steve-ham/MyLibrary) (Includes below 'MyFramework')
2. MyFramework: iOS framework with 'LibTorch-Lite' CocoaPod dependency. (https://github.com/steve-ham/MyFramework)

When I run 'MyLibrary App' (https://github.com/steve-ham/MyLibrary) I get below error:

```
    libc++abi: terminating with uncaught exception of type torch::jit::ErrorReport: 
    Unknown custom class type quantized.Conv2dPackedParamsBase. Please ensure it is registered.:
    quantized::conv1d(Tensor qx, __torch__.torch.classes.quantized.Conv2dPackedParamsBase packed_weight, float output_scale, int output_zero_point) -> Tensor
                                                                   ~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    
    terminating with uncaught exception of type torch::jit::ErrorReport: 
    Unknown custom class type quantized.Conv2dPackedParamsBase. Please ensure it is registered.:
    quantized::conv1d(Tensor qx, __torch__.torch.classes.quantized.Conv2dPackedParamsBase packed_weight, float output_scale, int output_zero_point) -> Tensor
                                                                   ~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
```

I used below code to create MyFramework.xcframework from MyFramework project (https://github.com/steve-ham/MyFramework)

```
    xcodebuild archive -workspace MyFramework.xcworkspace \
      -scheme MyFramework \
      -sdk iphoneos \
      -archivePath "./MyFramework.xcarchive" \
      BUILD_LIBRARY_FOR_DISTRIBUTION=YES \
      SKIP_INSTALL=NO
    
    xcodebuild archive -workspace MyFramework.xcworkspace \
      -scheme MyFramework \
      -sdk iphonesimulator \
      -archivePath "./MyFramework_sim.xcarchive" \
      BUILD_LIBRARY_FOR_DISTRIBUTION=YES \
      SKIP_INSTALL=NO \
      EXCLUDED_ARCHS=arm64
    
    xcodebuild -create-xcframework \
        -framework "MyFramework.xcarchive/Products/Library/Frameworks/MyFramework.framework" \
        -framework "MyFramework_sim.xcarchive/Products/Library/Frameworks/MyFramework.framework" \
        -output MyFramework.xcframework

```
I don't encounter this issue with other CocoaPods projects, this happens only with 'LibTorch-Lite'. Is there a fundamental limitation that prevents wrapping 'LibTorch-Lite' into an .xcframework for successful installation and execution in my 'MyLibrary' project? Thank you.

### Versions

PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: macOS 14.1.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: version 3.24.0
Libc version: N/A

Python version: 3.9.6 (default, May  7 2023, 23:32:44)  [Clang 14.0.3 (clang-1403.0.22.14.1)] (64-bit runtime)
Python platform: macOS-14.1.1-arm64-arm-64bit
Is CUDA available: N/A
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.25.2
[conda] Could not collect

cc @seemethere @malfet @osalpekar @atalman