Improve installation + code snippets

by Xenova HF staff - opened 4 days ago

base: refs/heads/main

←

from: refs/pr/2

Discussion Files changed

+29

-57

Xenova

Hugging Quants org 4 days ago

No description provided.

Improve installation + code snippetse4beb03d

pcuenq

Hugging Quants org 4 days ago

Looks good to me! Should the dtype be bfloat16, or not when using AWQ?

alvarobartt

Hugging Quants org 4 days ago

LGTM, thanks @Xenova ! And @pcuenq apparently for AWQ using float16 is recommended, if you set the torch_dtype=torch.bfloat16 the following message appears We suggest you to set torch_dtype=torch.float16 for better efficiency with AWQ., this is why we're using torch.float16 with AWQ :D

alvarobartt changed pull request status to merged 4 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment