Instructions to use ornith-ai/Ornith-1.0-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ornith-ai/Ornith-1.0-9B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ornith-ai/Ornith-1.0-9B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ornith-ai/Ornith-1.0-9B")
model = AutoModelForMultimodalLM.from_pretrained("ornith-ai/Ornith-1.0-9B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ornith-ai/Ornith-1.0-9B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ornith-ai/Ornith-1.0-9B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ornith-ai/Ornith-1.0-9B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ornith-ai/Ornith-1.0-9B

SGLang

How to use ornith-ai/Ornith-1.0-9B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ornith-ai/Ornith-1.0-9B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ornith-ai/Ornith-1.0-9B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ornith-ai/Ornith-1.0-9B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ornith-ai/Ornith-1.0-9B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ornith-ai/Ornith-1.0-9B with Docker Model Runner:
```
docker model run hf.co/ornith-ai/Ornith-1.0-9B
```

Whitespace and Backspace

#11

by LittleVoidGames - opened Jul 5

Discussion

LittleVoidGames

Jul 5

No questions, just noting some observations. Maybe it's my llama c++ build here, but the 9B model seems to forget to include whitespaces between words occasionally.

Another interesting behavior is that I've watched this model start outputting something, seem to change its mind, and then somehow delete the last couple of tokens and output something else. Not sure if that's from Qwen itself or the Ornith training, but it's fascinating to see happen in real time.

FlameF0X

1 day ago

•

edited 1 day ago

No questions, just noting some observations. Maybe it's my llama c++ build here, but the 9B model seems to forget to include whitespaces between words occasionally.

Another interesting behavior is that I've watched this model start outputting something, seem to change its mind, and then somehow delete the last couple of tokens and output something else. Not sure if that's from Qwen itself or the Ornith training, but it's fascinating to see happen in real time.

So, about the white space i have no answer. But for the backspace I have a answer, there is a paper about this, i think it was called SequenceMatch.

The IL framework also allows us to incorporate backtracking by introducing a backspace action into the generation process. This further mitigates the compounding error problem by allowing the model to revert a sampled token if it takes the sequence OOD. -- The abstraction of the paper
Its a thinkg that involes how Ornith was trained since backspace tokens are special tokens.

I might be wrong or something.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment