mobiuslabsgmbh
/

Llama-3.1-70b-instruct_4bitgs64_hqq

Text Generation

Model card Files Files and versions Community

mobicham commited on Aug 14, 2024

Commit

7fd9eec

·

verified ·

1 Parent(s): 247e309

correct decoding time

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -19,8 +19,8 @@ This is an <a href="https://github.com/mobiusml/hqq/">HQQ</a> all 4-bit (group-s
 ## Model Decoding Speed
 | Models            | fp16| HQQ 4-bit/gs-64|
 |:-------------------:|:--------:|:----------------:|
-| Decoding - short seq (tokens/sec)| 10.5  (tokens/sec)**  | 10.7 (tokens/sec)* |
-| Decoding - long  seq (tokens/sec)|  9.5 (tokens/sec)**  | 9.7 (tokens/sec)*|
 **: 2xA100 80GB<br>
 *:  1xA100 80GB

 ## Model Decoding Speed
 | Models            | fp16| HQQ 4-bit/gs-64|
 |:-------------------:|:--------:|:----------------:|
+| Decoding - short seq (tokens/sec)| 10.5  (tokens/sec)**  | 23 (tokens/sec)* |
+| Decoding - long  seq (tokens/sec)|  9.5 (tokens/sec)**  | 19 (tokens/sec)*|
 **: 2xA100 80GB<br>
 *:  1xA100 80GB