microsoft
/

Mistral-7B-v0.1-onnx

Model card Files Files and versions Community

petermcaughan commited on Dec 11, 2023

Commit

7353d4f

•

1 Parent(s): 646bb29

Update README.md

Files changed (1) hide show

README.md +8 -8

README.md CHANGED Viewed

@@ -34,14 +34,14 @@ Below is average latency of generating a token using a prompt of varying size us
 | Prompt Length      | Batch Size | PyTorch 2.1 torch.compile    | ONNX Runtime CUDA |
 |-------------|------------|----------------|-------------------|
-| 16      | 1          | N/A            | N/A           |
-| 256      | 1          | N/A            | N/A       |
-| 1024     | 1          | N/A        | N/A           |
-| 2048     | 1          | N/A       | N/A         |
-| 16      | 4          | N/A            | N/A           |
-| 256      | 4          | N/A            | N/A          |
-| 1024     | 4          | N/A        | N/A           |
-| 2048     | 4          | N/A       | N/A          |
 ## Usage Example

 | Prompt Length      | Batch Size | PyTorch 2.1 torch.compile    | ONNX Runtime CUDA |
 |-------------|------------|----------------|-------------------|
+| 32      | 1          | 32.58ms            | 12.08ms           |
+| 256      | 1          | 54.54ms            | 23.20ms       |
+| 1024     | 1          | 100.6ms        | 77.49ms         |
+| 2048     | 1          | 236.8ms       | 144.99ms         |
+| 32      | 4          | 63.71ms           | 15.32ms           |
+| 256      | 4          | 86.74ms            | 75.94ms         |
+| 1024     | 4          | 380.2ms        | 273.9ms           |
+| 2048     | 4          | N/A       | 554.5ms          |
 ## Usage Example