Update README.md

README.md

@@ -1,199 +1,274 @@
 ---
-library_name: transformers
-tags: []
+datasets:
+- jan-hq/instruction-speech-v1.5
+language:
+- en
+license: apache-2.0
+tags:
+- sound language model
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
-
 ## Model Details
 
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
+We have developed and released the family [Jan-Llama3s](https://huggingface.co/collections/jan-hq/jan-llama3-668e4dad446c8736208dca4f). This family is natively understanding audio and text input.
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+We continue to expand our last checkpoint [jan-hq/Jan-Llama3s-0708](https://huggingface.co/jan-hq/Jan-Llama3s-0708) with 1.3B tokens from [Instruction Speech v1.5](https://huggingface.co/datasets/jan-hq/instruction-speech-v1) dataset.
 
-[More Information Needed]
+**Model developers** Homebrew Research.
 
-### Out-of-Scope Use
+**Input** Text and sound.
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+**Output** Text.
 
-[More Information Needed]
+**Model Architecture** Llama-3.
 
-## Bias, Risks, and Limitations
+**Language(s):** English.
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+## Intended Use
 
-[More Information Needed]
+**Intended Use Cases** This family is primarily intended for research applications. This version aims to further improve the LLM on sound understanding capabilities.
 
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+**Out-of-scope** The use of Jan-Llama3s in any manner that violates applicable laws or regulations is strictly prohibited.
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
+First, we need to convert the audio file to sound tokens
+
+```python
+import torch
+import torchaudio
+from encodec import EncodecModel
+from encodec.utils import convert_audio
+
+def audio_to_sound_tokens(audio_path, target_bandwidth=1.5, device="cuda"):
+    # Initialize Encodec
+    model = EncodecModel.encodec_model_24khz()
+    model.set_target_bandwidth(target_bandwidth)
+    model.to(device)
+
+    # Load and preprocess audio
+    wav, sr = torchaudio.load(audio_path)
+    wav = convert_audio(wav, sr, model.sample_rate, model.channels)
+    wav = wav.unsqueeze(0).to(device)
+
+    # Encode audio
+    with torch.no_grad():
+        encoded_frames = model.encode(wav)
+    codes = torch.cat([encoded[0] for encoded in encoded_frames], dim=-1)
+
+    # Flatten codes
+    audio_code1, audio_code2 = codes[0][0], codes[0][1]
+    flatten_tokens = torch.stack((audio_code1, audio_code2), dim=1).flatten().tolist()
+
+    # Convert to sound tokens
+    result = ''.join(f'<|sound_{num:04d}|>' for num in flatten_tokens)
+    return f'<|sound_start|>{result}<|sound_end|>'
+
+# Usage
+sound_tokens = audio_to_sound_tokens("/path/to/your/audio/file")
+```
+
+Then, we can inference the model the same as any other LLM.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, pipeline
+
+def setup_pipeline(model_path, use_4bit=True):
+    tokenizer = AutoTokenizer.from_pretrained(model_path)
+
+    model_kwargs = {"device_map": "auto"}
+
+    if use_4bit:
+        model_kwargs["quantization_config"] = BitsAndBytesConfig(
+            load_in_4bit=True,
+            bnb_4bit_compute_dtype=torch.bfloat16,
+            bnb_4bit_use_double_quant=True,
+            bnb_4bit_quant_type="nf4",
+        )
+    else:
+        model_kwargs["torch_dtype"] = torch.bfloat16
+
+    model = AutoModelForCausalLM.from_pretrained(model_path, **model_kwargs)
+
+    return pipeline("text-generation", model=model, tokenizer=tokenizer)
+
+def generate_text(pipe, messages, max_new_tokens=64, temperature=0.0, do_sample=False):
+    generation_args = {
+        "max_new_tokens": max_new_tokens,
+        "return_full_text": False,
+        "temperature": temperature,
+        "do_sample": do_sample,
+    }
+
+    output = pipe(messages, **generation_args)
+    return output[0]['generated_text']
+
+# Usage
+llm_path = "jan-hq/Jan-Llama3s-0719"
+pipe = setup_pipeline(llm_path, use_4bit=True)
+messages = [
+    {"role": "user", "content": sound_tokens},
+]
+generated_text = generate_text(pipe, messages)
+print(generated_text)
+```
+
+## Training process
+**Training Metrics Image**: Below is a snapshot of the training loss curve visualized.
+
+![train_loss_curve](https://cdn-uploads.huggingface.co/production/uploads/65713d70f56f9538679e5a56/JYUOqGs1c-7vXtJt06KH5.png)
+
+### Hardware
+
+**GPU Configuration**: Cluster of 8x NVIDIA H100-SXM-80GB.
+**GPU Usage**:
+  - **Continual Training**: 14 hours.
+
+### Training Arguments
+
+| Parameter                  | Continual Training      | 
+|----------------------------|-------------------------|
+| **Epoch**                  | 1                       | 
+| **Global batch size**      | 128                     | 
+| **Learning Rate**          | 1.5e-4                  | 
+| **Learning Scheduler**     | Cosine with warmup      | 
+| **Optimizer**              | Adam torch fused        | 
+| **Warmup Ratio**           | 0.05                    | 
+| **Weight Decay**           | 0.01                    |
+| **beta1**                  | 0.9                     | 
+| **beta2**                  | 0.98                    | 
+| **epsilon**                | 1e-6                    | 
+| **Gradient Cliping**       | 1.0                     | 
+
+###
+ Accelerate FSDP Config
+
+```
+compute_environment: LOCAL_MACHINE
+debug: false
+distributed_type: FSDP
+downcast_bf16: 'no'
+enable_cpu_affinity: true
+fsdp_config:
+  fsdp_activation_checkpointing: true
+  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
+  fsdp_backward_prefetch: BACKWARD_PRE
+  fsdp_cpu_ram_efficient_loading: true
+  fsdp_forward_prefetch: false
+  fsdp_offload_params: false
+  fsdp_sharding_strategy: FULL_SHARD
+  fsdp_state_dict_type: SHARDED_STATE_DICT
+  fsdp_sync_module_states: true
+  fsdp_use_orig_params: false
+machine_rank: 0
+main_training_function: main
+mixed_precision: bf16
+num_machines: 1
+num_processes: 8
+rdzv_backend: static
+same_network: true
+tpu_env: []
+tpu_use_cluster: false
+tpu_use_sudo: false
+use_cpu: false
+```
+
+## Examples
+
+1. Good example:
+<details>
+<summary>Click to toggle Example 1`</summary>
+
+```
+# Question:
+Is the Paris Climate Change agreement going to help slow pollution?
+
+# Expected Output:
+Yes, the Paris Climate Agreement is designed to help slow pollution and combat climate change on a global scale. The main goal of the agreement is to limit global warming to well below 2 degrees Celsius, preferably to 1.5 degrees Celsius, compared to pre-industrial levels. To achieve this, countries that have signed the agreement commit to reducing their greenhouse gas emissions.
+
+# Model's Output:
+Yes, the Paris Climate Agreement is a global commitment to limit global warming to well below 2 degrees Celsius above pre-industrial levels and pursue efforts to limit it to 1.5 degrees Celsius. The agreement was signed by 195 countries and entered into force on November 4, 2016. 
+```
+
+<summary>Click to toggle Example 2`</summary>
+
+```
+# Question:
+Is the Paris Climate Change agreement going to help slow pollution?
+
+# Expected Output:
+Yes, the Paris Climate Agreement is designed to help slow pollution and combat climate change on a global scale. The main goal of the agreement is to limit global warming to well below 2 degrees Celsius, preferably to 1.5 degrees Celsius, compared to pre-industrial levels. To achieve this, countries that have signed the agreement commit to reducing their greenhouse gas emissions.
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+# Model's Output:
+Yes, the Paris Climate Agreement is a global commitment to limit global warming to well below 2 degrees Celsius above pre-industrial levels and pursue efforts to limit it to 1.5 degrees Celsius. The agreement was signed by 195 countries and entered into force on November 4, 2016. 
+```
+</details>
+
+
+2. Misunderstanding example:
+```
+# Question:
+Why is gasoline a liquid?
 
-[More Information Needed]
+# Expected Output:
+Gasoline is a liquid because of the nature of its molecules and the forces acting between them. It is made up of hydrocarbons, which are molecules composed of hydrogen and carbon atoms. These molecules are held together by covalent bonds, which are quite strong. 
 
-## Evaluation
+The forces acting between the molecules of gasoline are known as Van der Waals forces, which are relatively weak intermolecular forces. These forces are strong enough to keep the molecules together in a liquid form under normal conditions (room temperature and pressure), but not strong enough to keep them in a solid form. 
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+When the temperature drops significantly, gasoline can become a solid, and when it's heated, it becomes a gas. But under most conditions we encounter in daily life, these forces make gasoline a liquid.
 
-### Testing Data, Factors & Metrics
+# Model's Output:
+В случае, когда вы хотите перевести фразу "The first step is to identify the language of the text." на русский язык, вы можете использовать следующую фразу: "Первый шаг - определить язык текста."
+```
+
+3. Off-tracked example:
+```
+# Quesion:
+Answer the following question: -  The roots of the growing plants start to break up the rock - The plant acids dissolve the rock - The rock is broken down into smaller pieces - Erosion begins.  What might be the first step of the process?
+Answer:
+
+# Expected Output:
+The first step of the process might be: The roots of the growing plants start to break up the rock.
+
+# Model's Output:
+Step 1: Identify the key elements in the premise and hypothesis.
 
-#### Testing Data
+Premise: "A man in a blue shirt and a red hat is holding a sign."
+Key elements: man, blue shirt, red hat, holding, sign
 
-<!-- This should link to a Dataset Card if possible. -->
+Hypothesis: "A man is holding a sign."
+Key elements: man, holding, sign
 
-[More Information Needed]
+Step 2: Compare the key elements in the premise and hypothesis.
 
-#### Factors
+In the premise, we have a man wearing a blue shirt and a red hat, while in the hypothesis, we only have a man. The premise also mentions that the man is holding a sign, which is also mentioned in the hypothesis.
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+Step 3: Determine if the hypothesis is entailed by the premise.
 
-[More Information Needed]
+Since the hypothesis states that a man is holding a sign, and the premise confirms that a man is holding a sign, we can conclude that the hypothesis is entailed by the premise. The additional information about the man's clothing in the premise does not contradict or negate the
+```
 
-#### Metrics
+Despite being undertrained, the model demonstrates an emerging grasp of sound-text semantics.
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
 
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+## Citation Information
 
 **BibTeX:**
 
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
+```
+@article{Llama-3-Sound: Sound Instruction LLM 2024,
+  title={Llama-3-Sound},
+  author={Homebrew Research},
+  year=2024,
+  month=July},
+  url={https://huggingface.co/jan-hq/Jan-Llama3-0708}
+```
 
-## Model Card Authors [optional]
+## Acknowledgement
 
-[More Information Needed]
+- **[WhisperSpeech](https://github.com/collabora/WhisperSpeech)**
 
-## Model Card Contact
+- **[Encodec](https://github.com/facebookresearch/encodec)**
 
-[More Information Needed]
+- **[Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)**