Repository: pytorch
Issue: Domain Transformation APIs for LibTorch and LibTorch-Lite
Labels: module: cpp,triaged,module: vision,module: data

## 🚀 Feature
<!-- A clear and concise description of the feature proposal -->

**TL;DR**: Providing domain-specific transformation APIs will make it straightforward to pre-process and post-process the data in LibTorch `Tensor` format.

The LibTorch and LibTorch-Lite libraries are already great C++ front-ends for PyTorch on desktop and mobile devices. They provide great flexibility in deploying PyTorch models to edge devices. 

The goal of this proposal is to pitch for domain-specific transformation APIs (akin to [TorchAudio](https://pytorch.org/audio/stable/index.html), [TorchText](https://pytorch.org/text/stable/index.html), and [TorchVision](https://pytorch.org/vision/stable/index.html) libraries) with C++ frontend to use with LibTorch and LibTorch-Lite libraries. 

**Disclaimer**: I mainly build apps for iOS and was well experienced with PyTorch beforehand. So I'll try to motivate this proposal in that context. I hope Android developers will also be able to relate to me. This proposal is not meant to convey that Apple APIs are bad (they're not) but to convey that PyTorch-experienced engineer will be more comfortable with using known PyTorch API designs. This will also potentially increase the adoption of LibTorch and LibTorch-Lite libraries and make it easy to deploy latest research PyTorch models on edge devices.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->

Say, you're a deep learning engineer well experienced with PyTorch. And now you want to start building ML-powered apps in various domains (for example, object segmentation, machine translation, or text-to-speech) for iOS. You can go for CoreML, which is a great option! But at some point of your development you want your graph to be dynamic (CoreML misses this feature). You can't manipulate the graph on each execution. Moreover, you can't perform tensor operations from linear algebra like QR factorization and you might also lack some operation like fast Fourier transform. You can try to implement those with [MIL Ops](https://apple.github.io/coremltools/source/coremltools.converters.mil.mil.ops.defs.html) iff your complex PyTorch operation can be decomposed into currently existing MIL Ops! This is a very strong restriction! (I suffered trying to create FFT and finally came to LibTorch-Lite.)

So you find a fantastic C++ frontend library called LibTorch-Lite which you can interface to your Swift code via Objective-C wrapper. So you choose it! It looks almost similar to PyTorch API you're already used to! Now you can easily write tensor operations right in your app (even with Complex data types which you can't even in Metal Performance Shaders Graph yet)! You can also compute gradients and more. I think this is HUGE! 

## Pitch

<!-- A clear and concise description of what you want to happen. -->

Now let's talk about the missing pieces. You now want to modify the input data such as image, video, audio, or text before you feed it to your ML model. But you are not aware of Apple APIs for such tasks such as [Vision](https://developer.apple.com/documentation/vision), [Natural Language](https://developer.apple.com/documentation/naturallanguage), and [AVFoundation](https://developer.apple.com/documentation/avfoundation). They're good but the learning curve is high especially if you're not already an Apple developer. 

You wanna be able to use domain-specific transformations easily. Akin to how LibTorch-Lite makes it easy for PyTorch users to start building apps, I believe C++ frontends for PyTorch domain (only transformation) APIs will be very useful to developers in building apps quickly. For example, one may need to convert RGB to grayscale image, or convert spectrogram back to audio via Griffin-Lim algorithm, and so on. (We may exclude TorchText as it doesn't seem to have any transformation APIs) 

_**Providing domain-specific transformation APIs will make it straightforward to pre-process and post-process the data in LibTorch `Tensor` format.**_

## Alternatives

<!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->

I have considered using Vision and AVFoundation APIs but they lack a lot of data transformation features that PyTorch already provides. 

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->

N/A


cc @yf225 @glaringlee @fmassa @vfdev-5 @pmeier @VitalyFedyunin @ejguan @NivekT