DEPLOY_TEXT = f""" # 🚀 Deployment Tips A collection of powerful models is valuable, but ultimately, you need to be able to use them effectively. This tab is dedicated to providing guidance and code snippets for performing inference with leaderboard models on Intel platforms. Below is a table of open-source software options for inference, along with the supported Intel hardware platforms. A 🚀 indicates that inference with the associated software package is supported on the hardware. We hope this information helps you choose the best option for your specific use case. Happy building!
Inference Software Gaudi Xeon GPU Max Arc GPU Core Ultra
Optimum Habana 🚀
Intel Extension for PyTorch 🚀 🚀 🚀
Intel Extension for Transformers 🚀 🚀 🚀
OpenVINO 🚀 🚀 🚀 🚀
BigDL 🚀 🚀 🚀 🚀
NPU Acceleration Library 🚀
PyTorch 🚀 🚀 🚀
Tensorflow 🚀 🚀 🚀
# Intel® Gaudi® Accelerators The Intel Gaudi 2 accelerator is Intel's most capable deep learning chip. You can learn about Gaudi 2 [here](https://habana.ai/products/gaudi2/). Intel Gaudi Software supports PyTorch and DeepSpeed for accelerating LLM training and inference. The Intel Gaudi Software graph compiler will optimize the execution of the operations accumulated in the graph (e.g. operator fusion, data layout management, parallelization, pipelining and memory management, and graph-level optimizations). Optimum Habana provides covenient functionality for various tasks. Below is a command line snippet to run inference on Gaudi with meta-llama/Llama-2-7b-hf. 👍[Optimum Habana GitHub](https://github.com/huggingface/optimum-habana) The "run_generation.py" script below can be found [here on GitHub](https://github.com/huggingface/optimum-habana/tree/main/examples/text-generation) ```bash python run_generation.py \ --model_name_or_path meta-llama/Llama-2-7b-hf \ --use_hpu_graphs \ --use_kv_cache \ --max_new_tokens 100 \ --do_sample \ --batch_size 2 \ --prompt "Hello world" "How are you?" ```
# Intel® Max Series GPU The Intel® Data Center GPU Max Series is Intel's highest performing, highest density, general-purpose discrete GPU, which packs over 100 billion transistors into one package and contains up to 128 Xe Cores--Intel's foundational GPU compute building block. You can learn more about this GPU [here](https://www.intel.com/content/www/us/en/products/details/discrete-gpus/data-center-gpu/max-series.html). ### INT4 Inference (GPU) with Intel Extension for Transformers and Intel Extension for Python Intel® Extension for Transformers is an innovative toolkit designed to accelerate GenAI/LLM everywhere with the optimal performance of Transformer-based models on various Intel platforms, including Intel Gaudi2, Intel CPU, and Intel GPU. 👍 [Intel Extension for Transformers GitHub](https://github.com/intel/intel-extension-for-transformers) Intel® Extension for PyTorch* extends PyTorch* with up-to-date features optimizations for an extra performance boost on Intel hardware. Optimizations take advantage of Intel® Advanced Vector Extensions 512 (Intel® AVX-512) Vector Neural Network Instructions (VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel Xe Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs through the PyTorch* xpu device. 👍 [Intel Extension for PyTorch GitHub](https://github.com/intel/intel-extension-for-pytorch) ```python import intel_extension_for_pytorch as ipex from intel_extension_for_transformers.transformers.modeling import AutoModelForCausalLM from transformers import AutoTokenizer device_map = "xpu" model_name ="Qwen/Qwen-7B" tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) prompt = "When winter becomes spring, the flowers..." inputs = tokenizer(prompt, return_tensors="pt").input_ids.to(device_map) model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, device_map=device_map, load_in_4bit=True) model = ipex.optimize_transformers(model, inplace=True, dtype=torch.float16, woq=True, device=device_map) output = model.generate(inputs) ```
# Intel® Xeon® CPUs The Intel® Xeon® CPUs have the most built-in accelerators of any CPU on the market, including Advanced Matrix Extensions (AMX) to accelerate matrix multiplication in deep learning training and inference. Learn more about the Xeon CPUs [here](https://www.intel.com/content/www/us/en/products/details/processors/xeon.html). ### Optimum Intel and Intel Extension for PyTorch (no quantization) 🤗 Optimum Intel is the interface between the 🤗 Transformers and Diffusers libraries and the different tools and libraries provided by Intel to accelerate end-to-end pipelines on Intel architectures. 👍 [Optimum Intel GitHub](https://github.com/huggingface/optimum-intel) Requires installing/updating optimum `pip install --upgrade-strategy eager optimum[ipex]` ```python from optimum.intel import IPEXModelForCausalLM from transformers import AutoTokenizer, pipeline model = IPEXModelForCausalLM.from_pretrained(model_id) tokenizer = AutoTokenizer.from_pretrained(model_id) pipe = pipeline("text-generation", model=model, tokenizer=tokenizer) results = pipe("A fisherman at sea...") ``` ### Intel® Extension for PyTorch - Mixed Precision (fp32 and bf16) ```python import torch import intel_extension_for_pytorch as ipex import transformers model= transformers.AutoModelForCausalLM(model_name_or_path).eval() dtype = torch.float # or torch.bfloat16 model = ipex.llm.optimize(model, dtype=dtype) # generation inference loop with torch.inference_mode(): model.generate() ``` ### Intel® Extension for Transformers - INT4 Inference (CPU) ```python from transformers import AutoTokenizer from intel_extension_for_transformers.transformers import AutoModelForCausalLM model_name = "Intel/neural-chat-7b-v3-1" prompt = "When winter becomes spring, the flowers..." tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) inputs = tokenizer(prompt, return_tensors="pt").input_ids model = AutoModelForCausalLM.from_pretrained(model_name, load_in_4bit=True) outputs = model.generate(inputs) ```
# Intel® Core Ultra (NPUs and iGPUs) Intel® Core™ Ultra Processors are optimized for premium thin and powerful laptops, featuring 3D performance hybrid architecture, advanced AI capabilities, and available with built-in Intel® Arc™ GPU. Learn more about Intel Core Ultra [here](https://www.intel.com/content/www/us/en/products/details/processors/core-ultra.html). For now, there is support for smaller models like [TinyLama-1.1B](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0). ### Intel® NPU Acceleration Library The Intel® NPU Acceleration Library is a Python library designed to boost the efficiency of your applications by leveraging the power of the Intel Neural Processing Unit (NPU) to perform high-speed computations on compatible hardware. 👍 [Intel NPU Acceleration Library GitHub](https://github.com/intel/intel-npu-acceleration-library) ```python from transformers import AutoTokenizer, TextStreamer, AutoModelForCausalLM import intel_npu_acceleration_library import torch model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" model = AutoModelForCausalLM.from_pretrained(model_id, use_cache=True).eval() tokenizer = AutoTokenizer.from_pretrained(model_id, use_default_system_prompt=True) tokenizer.pad_token_id = tokenizer.eos_token_id streamer = TextStreamer(tokenizer, skip_special_tokens=True) print("Compile model for the NPU") model = intel_npu_acceleration_library.compile(model, dtype=torch.int8) query = input("Ask something: ") prefix = tokenizer(query, return_tensors="pt")["input_ids"] generation_kwargs = dict( input_ids=prefix, streamer=streamer, do_sample=True, top_k=50, top_p=0.9, max_new_tokens=512, ) print("Run inference") _ = model.generate(**generation_kwargs) ``` ### OpenVINO Tooling with Optimum Intel OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference. 👍 [OpenVINO GitHub](https://github.com/openvinotoolkit/openvino) ```python from optimum.intel import OVModelForCausalLM from transformers import AutoTokenizer, pipeline model_id = "helenai/gpt2-ov" model = OVModelForCausalLM.from_pretrained(model_id) tokenizer = AutoTokenizer.from_pretrained(model_id) pipe = pipeline("text-generation", model=model, tokenizer=tokenizer) pipe("In the spring, beautiful flowers bloom...") ```
# Intel® Arc GPUs You can learn more about Arc GPUs [here](https://www.intel.com/content/www/us/en/products/details/discrete-gpus/arc.html). Code snippets coming soon! """