update readme
Browse files
README.md
CHANGED
|
@@ -56,6 +56,15 @@ widely considered to be the father of modern computer science and artificial int
|
|
| 56 |
|
| 57 |
This requires a GPU with 12GB memory.
|
| 58 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 59 |
```python
|
| 60 |
import torch
|
| 61 |
import transformers
|
|
|
|
| 56 |
|
| 57 |
This requires a GPU with 12GB memory.
|
| 58 |
|
| 59 |
+
To run inference with int8, please ensure you have installed accelerate and bitandbytes. You can install them with the following command:
|
| 60 |
+
|
| 61 |
+
```bash
|
| 62 |
+
pip install accelerate
|
| 63 |
+
pip install bitsandbytes
|
| 64 |
+
```
|
| 65 |
+
|
| 66 |
+
Then you can run inference with int8 as follows:
|
| 67 |
+
|
| 68 |
```python
|
| 69 |
import torch
|
| 70 |
import transformers
|