add usage
Browse files
README.md
CHANGED
@@ -32,7 +32,22 @@ We would like to acknowledge the open-source frameworks [llm.c](https://github.c
|
|
32 |
|
33 |
The model can be used in two primary ways:
|
34 |
1. **With Hugging Face’s Transformers Library**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
35 |
2. **With llama.cpp**
|
|
|
|
|
|
|
|
|
36 |
|
37 |
## Disclaimer
|
38 |
|
|
|
32 |
|
33 |
The model can be used in two primary ways:
|
34 |
1. **With Hugging Face’s Transformers Library**
|
35 |
+
```python
|
36 |
+
from transformers import pipeline
|
37 |
+
import torch
|
38 |
+
|
39 |
+
path = "tinyllm/124M-0.0"
|
40 |
+
prompt = "The sea is blue but it's his red sea"
|
41 |
+
|
42 |
+
generator = pipeline("text-generation", model=path,max_new_tokens = 30, repetition_penalty=1.3, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto")
|
43 |
+
print(generator(prompt)[0]['generated_text'])
|
44 |
+
```
|
45 |
+
|
46 |
2. **With llama.cpp**
|
47 |
+
Generate a GGUF model file using this [tool](https://github.com/ggerganov/llama.cpp/blob/master/convert_hf_to_gguf.py) and use the generated GGUF file for inferencing.
|
48 |
+
```python
|
49 |
+
python3 convert_hf_to_gguf.py models/mymodel/
|
50 |
+
```
|
51 |
|
52 |
## Disclaimer
|
53 |
|