GPU Pooling
Published: 2023-11-14 08:27:31

The heterogeneous computing combination of CPU and GPU collocation is still the first choice for AI arithmetic. In the application of AI, GPU has shortcomings in elastic support, energy consumption and IT architecture construction.

CPU and GPU has huge difference-CPU is a necessity while GPU plays as a driver.

CPU runs all the time. But GPU, as a device attached to the computer, is only deployed when needed. Therefore, the key to efficiently utilizing GPU resources is to call them on demand, release them after use without worrying whether GPU resources are enough to use and where are they come from.
 
**GPU virtualization VS GPU pooling**
 
The industry has been exploring how to use GPU resources more optimally. However, most of the schemes do not solve the key problems.There are four stages: Even-slice virtualization, Elastic virtualization, remote deployment, and GPU pooling. 
 
**Stage 1. Even-slice Virtualization**: Divide the physical GPU into multiple virtual GPUs according to a fixed ratio, such as 1/2 or 1/4. Each virtual GPU has equal video memory, and the computing power is polled.

**Stage 2. Elastic Virtualization**: The goal is still to virtualize standalone GPU. But some technical means can be used to support the flexible segmentation of physical GPUs from computing power and video memory, to realize the custom sizes, and meeting the differentiated needs of AI applications.

**Stage 3. Remote Deployment**: An important technological breakthrough is the cross-node deployment of GPUs. AI applications can be deployed anywhere in the data center, regardless of whether the nodes have GPUs or not. At this stage, the scope of resource management extends from a single node to the entire data center interconnected by the network, which is a milestone in the evolution from GPU virtualization to GPU resource pooling.

**Stage 4. GPU Pooling**: The key is to call on demand, dynamically scale, and release when used up. With the pooling capability, AI applications can call GPUs of any size according to load requirements, and even aggregate GPUs of multiple physical nodes. After the container or virtual machine is created, the number and size of virtual GPUs can still be adjusted. When the AI application stops, GPU resources are immediately released and returned to the entire GPU resource pool, so that resources can be efficiently transferred and fully utilized. 

Through the above analysis, it can be seen that the traditional GPU virtualization technology (GPU slicing), is basically based on hardware, and can only virtualize the GPU on the local physical machine. But, GPU resource pooling is based on the entire data center which can not only support local GPU virtualization, but also break the physical boundary of single-machine resource scheduling, allowing users to transparently use GPU resources on any physical machine, flexibly deploy them on demand, and release them after use, thus greatly improving the utilization rate of expensive GPUs and business flexibility.

**OrionX GPU Pooling Solution:**

OrionX is a software of VirtAI Tech. Through the deployment of OrionX software on the physical server cluster, users can unify and schedule GPU resources of the entire data center through the graphical management interface or command line. 
Mainly functions of OrionX:

**- GPU resource pool construction**: Decouple business and GPU servers and overall manage and schedule of different models of GPUs, forming an enterprise-level GPU resource pool.

**- Decoupling CPU and GPU**: GPU pooling technology decouples CPUs from GPUs. It can deploy tasks to multiple remote CPU servers and calling unused GPU cards at the remote end through high-performance networks to achieve full utilization of resources.

**- Dynamic mounting/releasing of GPU resources**: For the GPU cards that have been added to the pool, it can realize the dynamic deploy of GPU resources from the pool during operation. And the GPU resources can be dynamically released to the pool after the end of the operation.

**- GUI monitoring interface**: Overall control through the highly developed web version of the scheduling manager interface, or control through the command line on the server node according to business needs.

Contact: BD@virtaitech.com
Official Website: https://virtaitech.com/en/index