microsoft
/

falcon-7B-onnx

Model card Files Files and versions Community

petermcaughan commited on Dec 11, 2023

Commit

cceea9d

•

1 Parent(s): a44ba12

Update README.md

Files changed (1) hide show

README.md +8 -8

README.md CHANGED Viewed

@@ -35,14 +35,14 @@ Below is average latency of generating a token using a prompt of varying size us
 | Prompt Length      | Batch Size | PyTorch 2.1 torch.compile    | ONNX Runtime CUDA |
 |-------------|------------|----------------|-------------------|
-| 16      | 1          | N/A            | N/A           |
-| 256      | 1          | N/A            | N/A       |
-| 1024     | 1          | N/A        | N/A           |
-| 2048     | 1          | N/A       | N/A         |
-| 16      | 4          | N/A            | N/A           |
-| 256      | 4          | N/A            | N/A          |
-| 1024     | 4          | N/A        | N/A           |
-| 2048     | 4          | N/A       | N/A          |
 ## Usage Example

 | Prompt Length      | Batch Size | PyTorch 2.1 torch.compile    | ONNX Runtime CUDA |
 |-------------|------------|----------------|-------------------|
+| 32      | 1          | 53.64ms            | 15.68ms           |
+| 256      | 1          | 59.55ms            | 26.05ms       |
+| 1024     | 1          | 89.82ms        | 99.05ms          |
+| 2048     | 1          | 208.0ms      | 227.0ms         |
+| 32      | 4          | 70.8ms            | 19.62ms           |
+| 256      | 4          | 78.6ms            | 81.29ms       |
+| 1024     | 4          | 373.7ms        | 369.6ms           |
+| 2048     | 4          | N/A       | 879.2ms          |
 ## Usage Example