README.md

---
title: README
emoji: 📚
colorFrom: blue
colorTo: green
sdk: static
pinned: false
---


# together we advance_AI

AI is increasingly pervasive across the modern world. 
It’s driving our smart technology in retail, cities, factories and healthcare, 
and transforming our digital homes. 
AMD offers advanced AI acceleration from data center to edge, 
enabling high performance and high efficiency to make the world smarter.

# Getting Started with Hugging Face Transformers



Looking for how to use the most common transformers on Hugging Face 
for inference workloads on select AMD Instinct™ accelerators and AMD Radeon™ GPUs using the AMD ROCm™ software? 
This base knowledge can be leveraged to start fine-tuning from a base model or even start developing your own model.
General Linux and ML experience is a required pre-requisite.

## 1. Confirm you have a supported AMD hardware platform

Is my [hardware supported](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html) with ROCm on Linux?

## 2. Install ROCm driver, libraries and tools

Follow the detailed [installation instructions](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/) for your Linux based platform.

## 3. Install Machine Learning Frameworks
Pip installation is an easy way to acquire all the required packages and is described in more detail below.

>If you prefer to use a container strategy, check out the pre-built images at 
[ROCm Docker Hub](https://hub.docker.com/u/rocm/) 
and [AMD Infinity Hub](https://www.amd.com/en/developer/resources/infinity-hub.html) 
after installing the required [dependancies](https://rocm.docs.amd.com/en/latest/deploy/docker.html).

### PyTorch
AMD ROCm is fully integrated into the mainline PyTorch ecosystem. Pip wheels are built and tested as part of the stable and nightly releases. 
Go to [pytorch.org](https://pytorch.org) and use the 'Install PyTorch' widget. 
Select 'Stable + Linux + Pip + Python + ROCm' to get the specific pip installation command.

An example command line (note the versioning of the whl file): 
  
> `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2`

### TensorFlow

AMD ROCm is upstreamed into the TensorFlow github repository. 
Pre-built wheels are hosted on [pipy.org](https://pypi.org/project/tensorflow-rocm/)

The latest version can be installed with this command: 
  
> `pip install tensorflow-rocm`

## 4. Use a Hugging Face Model
Now that you have the base requirements installed, get the latest transformer models.

> `pip install transformers`

This allows you to easily import any of the base models into your python application.
Here is an example using [GPT2](https://huggingface.co/gpt2) in PyTorch:

```python
from transformers import GPT2Tokenizer, GPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2Model.from_pretrained('gpt2')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```

All of the 200+ standard transformer models are regularly tested with our supported hardware platforms.
Note that this also implies that all derivatives of those core models should also function correctly. 
Let us know if you run into issues at our [ROCm Community page](https://github.com/RadeonOpenCompute/ROCm/discussions)

Here are a few of the more popular ones to get you started:
- [BERT](https://huggingface.co/bert-base-uncased)
- [BLOOM](https://huggingface.co/bigscience/bloom)
- [LLaMA](https://huggingface.co/huggyllama/llama-7b)
- [OPT](https://huggingface.co/facebook/opt-66b)
- [T5](https://huggingface.co/t5-base)

Click on the 'Use in Transformers' button to see the exact code to import a specific model into your Python application.

## 5. Optimum Support
For a deeper dive into using Hugging Face libraries on AMD GPUs, check out the [Optimum](https://huggingface.co/docs/optimum/main/en/amd/amdgpu/overview) page 
describing details on Flash Attention 2, GPTQ Quantization and ONNX Runtime integration.  

Details on getting started 
with Hugging Face models are available on the [Optimum page](https://huggingface.co/docs/optimum/main/en/amd/index) 

# Serving a model with TGI

Text Generation Inference (a.k.a “TGI”) provides an end-to-end solution to deploy large language models for inference at scale. 
TGI is already usable in production on AMD Instinct™ GPUs through the docker image `ghcr.io/huggingface/text-generation-inference:latest-rocm`. 
Make sure to refer to the [documentation](https://huggingface.co/docs/text-generation-inference/supported_models#supported-hardware) 
concerning the support and any limitations. 

# Benchmarking

The [Optimum-Benchmark](https://github.com/huggingface/optimum-benchmark) is available as a utility to easily benchmark the performance of transformers on AMD GPUs, 
across normal and distributed settings, with various supported optimizations and quantization schemes.     

# Useful Links and Blogs

- Detailed Llama-3 results [Run TGI on AMD Instinct MI300X](https://huggingface.co/blog/huggingface-amd-mi300)
- Detailed Llama-2 results show casing the [Optimum benchmark on AMD Instinct MI250](https://huggingface.co/blog/huggingface-and-optimum-amd)
- Check out our blog titled [Run a Chatgpt-like Chatbot on a Single GPU with ROCm](https://huggingface.co/blog/chatbot-amd-gpu)
- Complete ROCm [Documentation](https://rocm.docs.amd.com/en/latest/) for installation and usage
- Extended training content and connect with the development community at the [Developer Hub](https://www.amd.com/en/developer/resources/rocm-hub.html)