Update README.md
Browse files
README.md
CHANGED
@@ -1,199 +1,207 @@
|
|
1 |
---
|
2 |
-
|
3 |
-
|
|
|
|
|
|
|
|
|
|
|
4 |
---
|
5 |
|
6 |
-
# Model Card for Model ID
|
7 |
-
|
8 |
-
<!-- Provide a quick summary of what the model is/does. -->
|
9 |
-
|
10 |
-
|
11 |
-
|
12 |
## Model Details
|
13 |
|
14 |
-
|
15 |
-
|
16 |
-
<!-- Provide a longer summary of what this model is. -->
|
17 |
-
|
18 |
-
This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
|
19 |
-
|
20 |
-
- **Developed by:** [More Information Needed]
|
21 |
-
- **Funded by [optional]:** [More Information Needed]
|
22 |
-
- **Shared by [optional]:** [More Information Needed]
|
23 |
-
- **Model type:** [More Information Needed]
|
24 |
-
- **Language(s) (NLP):** [More Information Needed]
|
25 |
-
- **License:** [More Information Needed]
|
26 |
-
- **Finetuned from model [optional]:** [More Information Needed]
|
27 |
-
|
28 |
-
### Model Sources [optional]
|
29 |
-
|
30 |
-
<!-- Provide the basic links for the model. -->
|
31 |
-
|
32 |
-
- **Repository:** [More Information Needed]
|
33 |
-
- **Paper [optional]:** [More Information Needed]
|
34 |
-
- **Demo [optional]:** [More Information Needed]
|
35 |
|
36 |
-
|
37 |
|
38 |
-
|
39 |
|
40 |
-
|
41 |
|
42 |
-
|
43 |
|
44 |
-
|
45 |
|
46 |
-
|
47 |
|
48 |
-
|
49 |
|
50 |
-
|
51 |
|
52 |
-
|
53 |
-
|
54 |
-
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
|
55 |
-
|
56 |
-
[More Information Needed]
|
57 |
-
|
58 |
-
## Bias, Risks, and Limitations
|
59 |
-
|
60 |
-
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
|
61 |
-
|
62 |
-
[More Information Needed]
|
63 |
-
|
64 |
-
### Recommendations
|
65 |
-
|
66 |
-
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
|
67 |
-
|
68 |
-
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
|
69 |
|
70 |
## How to Get Started with the Model
|
71 |
|
72 |
-
|
73 |
-
|
74 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
75 |
|
76 |
-
|
|
|
77 |
|
78 |
-
|
|
|
79 |
|
80 |
-
|
|
|
|
|
|
|
|
|
|
|
81 |
|
82 |
-
|
|
|
|
|
83 |
|
84 |
-
|
85 |
|
86 |
-
|
|
|
|
|
87 |
|
88 |
-
|
89 |
|
90 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
91 |
|
|
|
92 |
|
93 |
-
|
94 |
|
95 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
96 |
|
97 |
-
|
|
|
98 |
|
99 |
-
|
|
|
|
|
|
|
100 |
|
101 |
-
|
|
|
102 |
|
103 |
-
|
104 |
|
105 |
-
|
106 |
|
107 |
-
|
|
|
|
|
108 |
|
109 |
-
|
110 |
|
111 |
-
|
112 |
|
113 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
114 |
|
115 |
-
#### Factors
|
116 |
|
117 |
-
|
118 |
|
119 |
-
|
120 |
|
121 |
-
|
|
|
122 |
|
123 |
-
|
124 |
|
125 |
-
|
|
|
126 |
|
127 |
-
|
|
|
128 |
|
129 |
-
|
130 |
|
131 |
-
|
|
|
132 |
|
133 |
|
|
|
134 |
|
135 |
-
|
|
|
|
|
|
|
136 |
|
137 |
-
|
|
|
138 |
|
139 |
-
|
140 |
|
141 |
-
|
|
|
142 |
|
143 |
-
|
144 |
|
145 |
-
|
|
|
146 |
|
147 |
-
- **Hardware Type:** [More Information Needed]
|
148 |
-
- **Hours used:** [More Information Needed]
|
149 |
-
- **Cloud Provider:** [More Information Needed]
|
150 |
-
- **Compute Region:** [More Information Needed]
|
151 |
-
- **Carbon Emitted:** [More Information Needed]
|
152 |
|
153 |
-
##
|
154 |
-
|
155 |
-
### Model Architecture and Objective
|
156 |
-
|
157 |
-
[More Information Needed]
|
158 |
-
|
159 |
-
### Compute Infrastructure
|
160 |
-
|
161 |
-
[More Information Needed]
|
162 |
-
|
163 |
-
#### Hardware
|
164 |
-
|
165 |
-
[More Information Needed]
|
166 |
-
|
167 |
-
#### Software
|
168 |
-
|
169 |
-
[More Information Needed]
|
170 |
-
|
171 |
-
## Citation [optional]
|
172 |
-
|
173 |
-
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
|
174 |
|
175 |
**BibTeX:**
|
176 |
|
177 |
-
|
178 |
-
|
179 |
-
|
180 |
-
|
181 |
-
|
182 |
-
|
183 |
-
|
184 |
-
|
185 |
-
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
|
186 |
-
|
187 |
-
[More Information Needed]
|
188 |
-
|
189 |
-
## More Information [optional]
|
190 |
-
|
191 |
-
[More Information Needed]
|
192 |
-
|
193 |
-
## Model Card Authors [optional]
|
194 |
|
195 |
-
|
196 |
|
197 |
-
|
198 |
|
199 |
-
[
|
|
|
1 |
---
|
2 |
+
datasets:
|
3 |
+
- homebrewltd/instruction-speech-whispervq-v2
|
4 |
+
language:
|
5 |
+
- en
|
6 |
+
license: apache-2.0
|
7 |
+
tags:
|
8 |
+
- sound language model
|
9 |
---
|
10 |
|
|
|
|
|
|
|
|
|
|
|
|
|
11 |
## Model Details
|
12 |
|
13 |
+
We have developed and released the family [llama3s](https://huggingface.co/collections/homebrew-research/llama3-s-669df2139f0576abc6eb7405). This family is natively understanding audio and text input.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
14 |
|
15 |
+
We expand the Semantic tokens experiment with WhisperVQ as a tokenizer for audio files from [homebrewltd/llama3.1-s-base-v0.2](https://huggingface.co/homebrewltd/llama3.1-s-base-v0.2) with nearly 1B tokens from [Instruction Speech WhisperVQ v2](https://huggingface.co/datasets/homebrewltd/instruction-speech-whispervq-v2) dataset.
|
16 |
|
17 |
+
**Model developers** Homebrew Research.
|
18 |
|
19 |
+
**Input** Text and sound.
|
20 |
|
21 |
+
**Output** Text.
|
22 |
|
23 |
+
**Model Architecture** Llama-3.
|
24 |
|
25 |
+
**Language(s):** English.
|
26 |
|
27 |
+
## Intended Use
|
28 |
|
29 |
+
**Intended Use Cases** This family is primarily intended for research applications. This version aims to further improve the LLM on sound understanding capabilities.
|
30 |
|
31 |
+
**Out-of-scope** The use of llama3-s in any manner that violates applicable laws or regulations is strictly prohibited.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
32 |
|
33 |
## How to Get Started with the Model
|
34 |
|
35 |
+
Try this model using [Google Colab Notebook](https://colab.research.google.com/drive/18IiwN0AzBZaox5o0iidXqWD1xKq11XbZ?usp=sharing).
|
36 |
+
|
37 |
+
First, we need to convert the audio file to sound tokens
|
38 |
+
|
39 |
+
```python
|
40 |
+
device = "cuda" if torch.cuda.is_available() else "cpu"
|
41 |
+
if not os.path.exists("whisper-vq-stoks-medium-en+pl-fixed.model"):
|
42 |
+
hf_hub_download(
|
43 |
+
repo_id="jan-hq/WhisperVQ",
|
44 |
+
filename="whisper-vq-stoks-medium-en+pl-fixed.model",
|
45 |
+
local_dir=".",
|
46 |
+
)
|
47 |
+
vq_model = RQBottleneckTransformer.load_model(
|
48 |
+
"whisper-vq-stoks-medium-en+pl-fixed.model"
|
49 |
+
).to(device)
|
50 |
+
def audio_to_sound_tokens(audio_path, target_bandwidth=1.5, device=device):
|
51 |
+
vq_model.ensure_whisper(device)
|
52 |
+
|
53 |
+
wav, sr = torchaudio.load(audio_path)
|
54 |
+
if sr != 16000:
|
55 |
+
wav = torchaudio.functional.resample(wav, sr, 16000)
|
56 |
+
with torch.no_grad():
|
57 |
+
codes = vq_model.encode_audio(wav.to(device))
|
58 |
+
codes = codes[0].cpu().tolist()
|
59 |
|
60 |
+
result = ''.join(f'<|sound_{num:04d}|>' for num in codes)
|
61 |
+
return f'<|sound_start|>{result}<|sound_end|>'
|
62 |
|
63 |
+
def audio_to_sound_tokens_transcript(audio_path, target_bandwidth=1.5, device=device):
|
64 |
+
vq_model.ensure_whisper(device)
|
65 |
|
66 |
+
wav, sr = torchaudio.load(audio_path)
|
67 |
+
if sr != 16000:
|
68 |
+
wav = torchaudio.functional.resample(wav, sr, 16000)
|
69 |
+
with torch.no_grad():
|
70 |
+
codes = vq_model.encode_audio(wav.to(device))
|
71 |
+
codes = codes[0].cpu().tolist()
|
72 |
|
73 |
+
result = ''.join(f'<|sound_{num:04d}|>' for num in codes)
|
74 |
+
return f'<|reserved_special_token_69|><|sound_start|>{result}<|sound_end|>'
|
75 |
+
```
|
76 |
|
77 |
+
Then, we can inference the model the same as any other LLM.
|
78 |
|
79 |
+
```python
|
80 |
+
def setup_pipeline(model_path, use_4bit=False, use_8bit=False):
|
81 |
+
tokenizer = AutoTokenizer.from_pretrained(model_path)
|
82 |
|
83 |
+
model_kwargs = {"device_map": "auto"}
|
84 |
|
85 |
+
if use_4bit:
|
86 |
+
model_kwargs["quantization_config"] = BitsAndBytesConfig(
|
87 |
+
load_in_4bit=True,
|
88 |
+
bnb_4bit_compute_dtype=torch.bfloat16,
|
89 |
+
bnb_4bit_use_double_quant=True,
|
90 |
+
bnb_4bit_quant_type="nf4",
|
91 |
+
)
|
92 |
+
elif use_8bit:
|
93 |
+
model_kwargs["quantization_config"] = BitsAndBytesConfig(
|
94 |
+
load_in_8bit=True,
|
95 |
+
bnb_8bit_compute_dtype=torch.bfloat16,
|
96 |
+
bnb_8bit_use_double_quant=True,
|
97 |
+
)
|
98 |
+
else:
|
99 |
+
model_kwargs["torch_dtype"] = torch.bfloat16
|
100 |
|
101 |
+
model = AutoModelForCausalLM.from_pretrained(model_path, **model_kwargs)
|
102 |
|
103 |
+
return pipeline("text-generation", model=model, tokenizer=tokenizer)
|
104 |
|
105 |
+
def generate_text(pipe, messages, max_new_tokens=64, temperature=0.0, do_sample=False):
|
106 |
+
generation_args = {
|
107 |
+
"max_new_tokens": max_new_tokens,
|
108 |
+
"return_full_text": False,
|
109 |
+
"temperature": temperature,
|
110 |
+
"do_sample": do_sample,
|
111 |
+
}
|
112 |
|
113 |
+
output = pipe(messages, **generation_args)
|
114 |
+
return output[0]['generated_text']
|
115 |
|
116 |
+
# Usage
|
117 |
+
llm_path = "homebrewltd/llama3.1-s-instruct-v0.2"
|
118 |
+
pipe = setup_pipeline(llm_path, use_8bit=True)
|
119 |
+
```
|
120 |
|
121 |
+
## Training process
|
122 |
+
**Training Metrics Image**: Below is a snapshot of the training loss curve visualized.
|
123 |
|
124 |
+
![training_](https://cdn-uploads.huggingface.co/production/uploads/65713d70f56f9538679e5a56/pQ8y9GoSvtv42MgkKRDt0.png)
|
125 |
|
126 |
+
### Hardware
|
127 |
|
128 |
+
**GPU Configuration**: Cluster of 8x NVIDIA H100-SXM-80GB.
|
129 |
+
**GPU Usage**:
|
130 |
+
- **Continual Training**: 12 hours.
|
131 |
|
132 |
+
### Training Arguments
|
133 |
|
134 |
+
We utilize [torchtune](https://github.com/pytorch/torchtune) library for the latest FSDP2 training code implementation.
|
135 |
|
136 |
+
| Parameter | Instruction Fine-tuning |
|
137 |
+
|----------------------------|-------------------------|
|
138 |
+
| **Epoch** | 1 |
|
139 |
+
| **Global batch size** | 128 |
|
140 |
+
| **Learning Rate** | 7e-5 |
|
141 |
+
| **Learning Scheduler** | Cosine with warmup |
|
142 |
+
| **Optimizer** | Adam torch fused |
|
143 |
+
| **Warmup Ratio** | 0.01 |
|
144 |
+
| **Weight Decay** | 0.005 |
|
145 |
+
| **Max Sequence Length** | 4096 |
|
146 |
|
|
|
147 |
|
148 |
+
## Examples
|
149 |
|
150 |
+
1. Good example:
|
151 |
|
152 |
+
<details>
|
153 |
+
<summary>Click to toggle Example 1</summary>
|
154 |
|
155 |
+
```
|
156 |
|
157 |
+
```
|
158 |
+
</details>
|
159 |
|
160 |
+
<details>
|
161 |
+
<summary>Click to toggle Example 2</summary>
|
162 |
|
163 |
+
```
|
164 |
|
165 |
+
```
|
166 |
+
</details>
|
167 |
|
168 |
|
169 |
+
2. Misunderstanding example:
|
170 |
|
171 |
+
<details>
|
172 |
+
<summary>Click to toggle Example 3</summary>
|
173 |
+
|
174 |
+
```
|
175 |
|
176 |
+
```
|
177 |
+
</details>
|
178 |
|
179 |
+
3. Off-tracked example:
|
180 |
|
181 |
+
<details>
|
182 |
+
<summary>Click to toggle Example 4</summary>
|
183 |
|
184 |
+
```
|
185 |
|
186 |
+
```
|
187 |
+
</details>
|
188 |
|
|
|
|
|
|
|
|
|
|
|
189 |
|
190 |
+
## Citation Information
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
191 |
|
192 |
**BibTeX:**
|
193 |
|
194 |
+
```
|
195 |
+
@article{Llama3-S: Sound Instruction Language Model 2024,
|
196 |
+
title={Llama3-S},
|
197 |
+
author={Homebrew Research},
|
198 |
+
year=2024,
|
199 |
+
month=August},
|
200 |
+
url={https://huggingface.co/homebrewltd/llama3.1-s-2024-08-20}
|
201 |
+
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
202 |
|
203 |
+
## Acknowledgement
|
204 |
|
205 |
+
- **[WhisperSpeech](https://github.com/collabora/WhisperSpeech)**
|
206 |
|
207 |
+
- **[Meta-Llama-3.1-8B-Instruct ](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)**
|