[Huawei Developers] Jack Wu Shenzhen:
New media — like vlogs and short videos — are on fire, as they ride the wave of changes brought by mobile phones on how we socialize and keep entertained. Photo and video editing apps are therefore becoming more functional, utilizing AI-empowered functions for a range of situations.

Such an editing app is expected to have various functions and materials, which (logically) entails a long development period and high requirements. Creating such an app is a daunting task for developers, one that is both slow and demanding. Video Editor Kit, launched in HMS Core 6.0, helps developers build a smart, easy-to-use editing app with a range of capabilities including resource input, editing, rendering, and output, as well as material management. In addition to these common functions, the kit also offers advanced, AI-driven functions such as AI filter, track person, and color hair, allowing users to fully unleash their creativity.

View attachment 5610661​

These powerful functions must come from somewhere, and that somewhere is neural network models. A single model can exceed 10 MB. Together, those models can occupy considerable ROM and RAM space on a device. Therefore, another challenge for video editing app developers is to ensure that their app occupies as little space as possible.

To resolve this concern, Video Editor Kit turns to MindSpore Lite (a Huawei-developed AI engine) for inference of neural network models. MindSpore Lite is a fantastic solution to this because of its inference neural network models. It is loaded with unified APIs, supporting flexible model deployment from devices to edges and cloud. It is available on devices with the Ascend processor, GPU, CPU (whose architecture can be X86, ARM, and more) or other hardware units, and can run on operating systems such as HarmonyOS, Android, iOS, Windows, or others. In addition to models trained by MindSpore, MindSpore Lite can convert and infer third-party models from TensorFlow, TensorFlow Lite, Caffe, ONNX, and more.

View attachment 5610621

MindSpore Lite architecture​

MindSpore Lite provides high-performance and ultra-lightweight solutions to AI model inference. It delivers efficient kernel algorithms and assembly-level optimization, as well as heterogeneous scheduling of CPUs, GPUs, and NPUs. In this way, MindSpore Lite allows hardware to fully wield its computing power, to minimize the inference duration and power consumption. MindSpore Lite adopts post-training quantization (PTQ) to support model quantization compression. It requires no dataset and directly maps the weight data of the float type to the low-bit fixed-point data. For this reason, MindSpore Lite slashes the size of models, allowing them to be deployed in environments with limited resources.

View attachment 5610625

How the quantization technique works​

Weight quantization supports fixed bit quantization and mixed bit quantization. The former adopts bit-packing and supports quantization between 1 and 16, to satisfy different compression needs. Additionally, fixed bit quantization checks how data is distributed after quantization and then automatically chooses the proper policy for compression and encoding, to deliver the ideal compression effect.

View attachment 5610627

Fixed bit quantization​

The sensitivity of layers in a neural network to weight loss varies substantially. With this in mind, mixed bit quantization makes the mean-square error its optimization target. It automatically finds the bit most suitable for a layer, delivering a greater compression rate without compromising accuracy. As to the quantized model, mixed bit quantization adopts Finite State Entropy (FSE) to further compress the quantized weight data by using an entropy coding scheme. Mixed bit quantization consequently adeptly compresses models, increasing the model transmission rate while reducing model space occupancy.

View attachment 5610629

Mixed bit quantization​

To minimize the noise of quantization, mixed bit quantization uses the bias correction technique. It takes into account the inherent statistical features of weight data and calibrates it during dequantization. This ensures that weight data has the same expectation and variance before and after model quantization, for considerably higher model accuracy.

Video Editor Kit chooses mixed bit quantization for its AI models. As a result, the kit ensures its compressed models have both high accuracy and an average file size which is just 20% of that of the original models. The model of color hair, for example, saw its size drop from 20.86 MB to 3.76 MB. Video Editor Kit is thus a handy tool for developers, which is easy to deploy.

View attachment 5610631

Model quantization effect for Video Editor Kit capabilities (sourced from the test data of MindSpore)​

With model quantization and compression, Video Editor Kit allows more of its AI models to be deployed in an app, without occupying more ROM space, helping equip an app with smarter editing capabilities.

Just take a look at Petal Clip, Huawei's own dedicated editing app. After integrating Video Editor Kit, the app now offers intelligent editing functions such as AI filter and track person, and more capabilities driven by the kit will become available during later app updates. With those functions, Petal Clip makes video editing more fun and straightforward.

The high-performance, ultra-lightweight AI engine of MindSpore Lite is not just armed with powerful kernel algorithms, heterogeneous hardware scheduling, and model quantization compression, but also provides one-stop model training and inference capabilities that feature the device-cloud synergy. Together with MindSpore Lite, Video Editor Kit paves the road for developing a straightforward, intelligent editing mobile app.

To find more, please visit:

HUAWEI Developers> HMS Core

MindSpore

Open-source code for MindSpore