Instructions to use hbpkillerX/lift-fp8-dynamic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hbpkillerX/lift-fp8-dynamic with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="hbpkillerX/lift-fp8-dynamic")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("hbpkillerX/lift-fp8-dynamic")
model = AutoModelForMultimodalLM.from_pretrained("hbpkillerX/lift-fp8-dynamic", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use hbpkillerX/lift-fp8-dynamic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hbpkillerX/lift-fp8-dynamic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hbpkillerX/lift-fp8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/hbpkillerX/lift-fp8-dynamic

SGLang

How to use hbpkillerX/lift-fp8-dynamic with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hbpkillerX/lift-fp8-dynamic" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hbpkillerX/lift-fp8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hbpkillerX/lift-fp8-dynamic" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hbpkillerX/lift-fp8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use hbpkillerX/lift-fp8-dynamic with Docker Model Runner:
```
docker model run hf.co/hbpkillerX/lift-fp8-dynamic
```

Lift FP8 Dynamic

Mixed-precision FP8 derivative of datalab-to/lift, source revision 221e0fcecfb3f557ca123e5a6e3452d459c25d90. Lift extracts structured JSON from document images.

Quantization

Created using LLM Compressor 0.12.0.1 on one NVIDIA L40S, with PyTorch 2.12.0, Transformers 5.10.1, and compressed-tensors 0.17.1. The FP8_DYNAMIC recipe uses static per-channel FP8 weights and dynamic per-token FP8 activations. No calibration data was used.

Targets: Linear; exclusions: lm_head, re:.*visual.*, re:.*linear_attn.*. The vision encoder, linear attention, embeddings and other excluded parameters retain BF16 precision. KV-cache quantization is not included. There are 128 FP8 tensors. Weight files total 13,520,203,008 bytes. The original processor, tokenizer, and chat template are included. Only parameters loaded by the Transformers model class are saved; auxiliary source checkpoint tensors not loaded by that class are not preserved.

Validation status

Serialized FP8 tensor dtypes and quantization metadata have been verified. vLLM inference compatibility and document extraction accuracy have not yet been validated for this checkpoint. No quality or speed equivalence is claimed. See quantization-report.json for conversion measurements and package versions.

Reproduction

See modal_quantize.py for the Modal conversion job and recipe.yaml for the recipe.

License and attribution

This derivative retains Lift's upstream modified OpenRAIL-M model license and usage restrictions. Quantization does not change those terms. Read the original model card, the included license files, and the upstream repository before use. Lift is developed by Datalab and uses the Qwen3.5 architecture.

Downloads last month: 38

Safetensors

Model size

9B params

Tensor type

BF16

F8_E4M3

Model tree for hbpkillerX/lift-fp8-dynamic

Base model

datalab-to/lift

Quantized

(15)

this model