Text Generation
Transformers
llm-rs
ggml
Inference Endpoints
File size: 7,180 Bytes
1f0e430
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
---
datasets:
- bigscience/xP3
license: bigscience-bloom-rail-1.0
language:
- ak
- ar
- as
- bm
- bn
- ca
- code
- en
- es
- eu
- fon
- fr
- gu
- hi
- id
- ig
- ki
- kn
- lg
- ln
- ml
- mr
- ne
- nso
- ny
- or
- pa
- pt
- rn
- rw
- sn
- st
- sw
- ta
- te
- tn
- ts
- tum
- tw
- ur
- vi
- wo
- xh
- yo
- zh
- zu
programming_language: 
- C
- C++
- C#
- Go
- Java
- JavaScript
- Lua
- PHP
- Python
- Ruby
- Rust
- Scala
- TypeScript
tags:
- llm-rs
- ggml
pipeline_tag: text-generation
---

# GGML covnerted Models of [BigScience](https://huggingface.co/bigscience)'s Bloom models

## Description

> We present BLOOMZ & mT0, a family of models capable of following human instructions in dozens of languages zero-shot. We finetune BLOOM & mT5 pretrained multilingual language models on our crosslingual task mixture (xP3) and find the resulting models capable of crosslingual generalization to unseen tasks & languages.

- **Repository:** [bigscience-workshop/xmtf](https://github.com/bigscience-workshop/xmtf)
- **Paper:** [Crosslingual Generalization through Multitask Finetuning](https://arxiv.org/abs/2211.01786)
- **Point of Contact:** [Niklas Muennighoff](mailto:niklas@hf.co)
- **Languages:** Refer to [bloom](https://huggingface.co/bigscience/bloom) for pretraining & [xP3](https://huggingface.co/datasets/bigscience/xP3) for finetuning language proportions. It understands both pretraining & finetuning languages.

### Intended use

We recommend using the model to perform tasks expressed in natural language. For example, given the prompt "*Translate to English: Je t’aime.*", the model will most likely answer "*I love you.*". Some prompt ideas from our paper: 
- 一个传奇的开端,一个不灭的神话,这不仅仅是一部电影,而是作为一个走进新时代的标签,永远彪炳史册。你认为这句话的立场是赞扬、中立还是批评?
- Suggest at least five related search terms to "Mạng neural nhân tạo".
- Write a fairy tale about a troll saving a princess from a dangerous dragon. The fairy tale is a masterpiece that has achieved praise worldwide and its moral is "Heroes Come in All Shapes and Sizes". Story (in Spanish):
- Explain in a sentence in Telugu what is backpropagation in neural networks.

## Converted Models
| Name                                                                                                            | Based on                                                                | Type   | Container   | GGML Version   |
|:----------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:-------|:------------|:---------------|
| [bloomz-1b1-f16.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b1-f16.bin)               | [bigscience/bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)   | F16    | GGML        | V3             |
| [bloomz-1b1-q4_0.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b1-q4_0.bin)             | [bigscience/bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)   | Q4_0   | GGML        | V3             |
| [bloomz-1b1-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b1-q4_0-ggjt.bin)   | [bigscience/bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)   | Q4_0   | GGJT        | V3             |
| [bloomz-1b1-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b1-q5_1-ggjt.bin)   | [bigscience/bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)   | Q5_1   | GGJT        | V3             |
| [bloomz-1b7-f16.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b7-f16.bin)               | [bigscience/bloomz-1b7](https://huggingface.co/bigscience/bloomz-1b7)   | F16    | GGML        | V3             |
| [bloomz-1b7-q4_0.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b7-q4_0.bin)             | [bigscience/bloomz-1b7](https://huggingface.co/bigscience/bloomz-1b7)   | Q4_0   | GGML        | V3             |
| [bloomz-1b7-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b7-q4_0-ggjt.bin)   | [bigscience/bloomz-1b7](https://huggingface.co/bigscience/bloomz-1b7)   | Q4_0   | GGJT        | V3             |
| [bloomz-1b7-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b7-q5_1-ggjt.bin)   | [bigscience/bloomz-1b7](https://huggingface.co/bigscience/bloomz-1b7)   | Q5_1   | GGJT        | V3             |
| [bloomz-3b-f16.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-3b-f16.bin)                 | [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b)     | F16    | GGML        | V3             |
| [bloomz-3b-q4_0.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-3b-q4_0.bin)               | [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b)     | Q4_0   | GGML        | V3             |
| [bloomz-3b-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-3b-q4_0-ggjt.bin)     | [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b)     | Q4_0   | GGJT        | V3             |
| [bloomz-3b-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-3b-q5_1-ggjt.bin)     | [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b)     | Q5_1   | GGJT        | V3             |
| [bloomz-560m-f16.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-560m-f16.bin)             | [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) | F16    | GGML        | V3             |
| [bloomz-560m-q4_0.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-560m-q4_0.bin)           | [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) | Q4_0   | GGML        | V3             |
| [bloomz-560m-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-560m-q4_0-ggjt.bin) | [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) | Q4_0   | GGJT        | V3             |
| [bloomz-560m-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-560m-q5_1-ggjt.bin) | [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) | Q5_1   | GGJT        | V3             |

## Usage

### Python via [llm-rs](https://github.com/LLukas22/llm-rs-python):

#### Installation
Via pip: `pip install llm-rs`

#### Run inference
```python
from llm_rs import AutoModel

#Load the model, define any model you like from the list above as the `model_file`
model = AutoModel.from_pretrained("rustformers/bloomz-ggml",model_file="bloomz-3b-q4_0-ggjt.bin")

#Generate
print(model.generate("The meaning of life is"))
```

### Rust via [Rustformers/llm](https://github.com/rustformers/llm): 

#### Installation
```
git clone --recurse-submodules https://github.com/rustformers/llm.git
cd llm
cargo build --release
```

#### Run inference
```
cargo run --release -- bloom infer -m path/to/model.bin  -p "Tell me how cool the Rust programming language is:"
```