Instructions to use inception42/jais-30b-chat-v3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inception42/jais-30b-chat-v3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="inception42/jais-30b-chat-v3", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("inception42/jais-30b-chat-v3", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inception42/jais-30b-chat-v3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inception42/jais-30b-chat-v3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inception42/jais-30b-chat-v3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/inception42/jais-30b-chat-v3

SGLang

How to use inception42/jais-30b-chat-v3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inception42/jais-30b-chat-v3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inception42/jais-30b-chat-v3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inception42/jais-30b-chat-v3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inception42/jais-30b-chat-v3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use inception42/jais-30b-chat-v3 with Docker Model Runner:
```
docker model run hf.co/inception42/jais-30b-chat-v3
```

I need a help to overcome Cuda out of memory

#13

by HanaRasheed - opened Dec 31, 2024

Discussion

HanaRasheed

Dec 31, 2024

•

edited Dec 31, 2024

Hello, I am trying to fine tune JAIS 13B Chat version model using instruction tuning to perform a classification task on Arabic dataset , I applied quantization and PEFT, but I am getting cuda of memory when I reach trainer.train() step, can you help me in this matter ? I am using 3*RTX A6000 resource

alielfilali01

Inception42 org Jan 2, 2025

Hello @HanaRasheed , Tnx for opening this discussion.
Well, it would have been more practical to open it in the model you are facing the issue with (which 13B variant ?) but to answer your message, generally, it is hard to point out where could be went wrong ! Can you provide more context, which quantization format, code you are running, format of the dataset in case you can't provide the dataset. Also, i would suggest to try out LLMTools and see of it works.
Anyway, I would be glad to help figure it out ounce you provide more context of the error.
On a seperate note, i would suggest to use inceptionai/jais-family-13b or inceptionai/jais-adapted-13b as they are more recent and better in terms of benchmarks results.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment