Update README.md

README.md

@@ -0,0 +1,254 @@
+---
+base_model: mistralai/Mixtral-8x22B-Instruct-v0.1
+model_creator: mistralai
+quantized_by: jartine
+license: apache-2.0
+prompt_template: |
+  [INST] {{prompt}} [/INST]
+tags:
+- llamafile
+language:
+- en
+---
+
+# Mixtral 8x22B Instruct v0.1 - llamafile
+
+This repository contains executable weights (which we call
+[llamafiles](https://github.com/Mozilla-Ocho/llamafile)) that run on
+Linux, MacOS, Windows, FreeBSD, OpenBSD, and NetBSD for AMD64 and ARM64.
+
+- Model creator: [Mistral AI](https://mistral.ai/)
+- Original model: [mistralai/Mixtral-8x22B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1)
+
+## Quickstart
+
+Assuming your system has at least 128GB of RAM, you can try running the
+following command which download, concatenate, and execute the model.
+
+```
+( curl -L https://huggingface.co/jartine/Mixtral-8x22B-Instruct-v0.1-llamafile/resolve/main/Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile.cat0
+  curl -L https://huggingface.co/jartine/Mixtral-8x22B-Instruct-v0.1-llamafile/resolve/main/Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile.cat1
+) > Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile
+chmod +x Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile
+./Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile --help   # view manual
+./Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile          # launch web gui + oai api
+./Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile -p ...   # cli interface (scriptable)
+```
+
+Alternatively, you may download an official `llamafile` executable from
+Mozilla Ocho on GitHub, in which case you can use the Mixtral llamafiles
+as a simple weights data file.
+
+```
+llamafile -m Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile ...
+```
+
+For further information, please see the [llamafile
+README](https://github.com/mozilla-ocho/llamafile/).
+
+Having **trouble?** See the ["Gotchas"
+section](https://github.com/mozilla-ocho/llamafile/?tab=readme-ov-file#gotchas-and-troubleshooting)
+of the README.
+
+## Prompting
+
+Prompt template:
+
+```
+[INST] {{prompt}} [/INST]
+```
+
+Command template:
+
+```
+./Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile -p "[INST]{{prompt}}[/INST]"
+```
+
+## About llamafile
+
+llamafile is a new format introduced by Mozilla Ocho on Nov 20th 2023.
+It uses Cosmopolitan Libc to turn LLM weights into runnable llama.cpp
+binaries that run on the stock installs of six OSes for both ARM64 and
+AMD64.
+
+In addition to being executables, llamafiles are also zip archives. Each
+llamafile contains a GGUF file, which you can extract using the `unzip`
+command. If you want to change or add files to your llamafiles, then the
+`zipalign` command (distributed on the llamafile github) should be used
+instead of the traditional `zip` command.
+
+## About Upload Limits
+
+Files which exceed the Hugging Face 50GB upload limit have a .cat𝑋
+extension. You need to use the `cat` command locally to turn them back
+into a single file, using the same order.
+
+## About Quantization Formats (General Advice)
+
+Your choice of quantization format depends on three things:
+
+1. Will it fit in RAM or VRAM?
+2. Is your use case reading (e.g. summarization) or writing (e.g. chatbot)?
+3. llamafiles bigger than 4.30 GB are hard to run on Windows (see [gotchas](https://github.com/mozilla-ocho/llamafile/?tab=readme-ov-file#gotchas-and-troubleshooting))
+
+Good quants for writing (prediction speed) are Q5\_K\_M, and Q4\_0. Text
+generation is bounded by memory speed, so smaller quants help, but they
+cause the LLM to hallucinate more. However that doesn't mean they can't
+think correctly. A highly degraded quant like `Q2_K` may not make a
+great encyclopedia, but it's still capable of logical reasoning and
+the emergent capabilities LLMs exhibit.
+
+Good quants for reading (evaluation speed) are BF16, F16, Q8\_0, and
+Q4\_0 (ordered from fastest to slowest). Prompt evaluation is bounded by
+flop count, which means perf can be improved through software
+engineering alone, e.g. BLAS algorithms, in which case quantization
+starts hurting more than it helps, since it competes for CPU resources
+and makes it harder for the compiler to parallelize instructions. You
+want to ideally use the simplest smallest floating point format that's
+natively implemented by your hardware. In most cases, that's BF16 or
+FP16. However, llamafile is able to still offer respectable tinyBLAS
+speedups for llama.cpp's simplest quants: Q8\_0 and Q4\_0.
+
+## Hardware Choices (Mixtral 8x22B Specific)
+
+This model is very large. Even at Q2 quantization, it's still well-over
+twice as large the highest tier NVIDIA gaming GPUs. llamafile supports
+splitting models over multiple GPUs (for NVIDIA only currently) if you
+have such a system. The easiest way to have one, if you don't, is to pay
+a few bucks an hour to rent a 4x RTX 4090 rig off vast.ai.
+
+Mac Studio is a good option for running this model locally. An M2 Ultra
+desktop from Apple is affordable and has 128GB of unified RAM+VRAM. If
+you have one, then llamafile will use your Metal GPU. Try starting out
+with the `Q4_0` quantization level.
+
+Another good option for running large, large language models locally and
+fully under your control is to just use CPU inference. We developed new
+tensor multiplication kernels on the llamafile project specifically to
+speed up "mixture of experts" LLMs like Mixtral. On a AMD Threadripper
+Pro 7995WX with 256GB of 5200 MT/s RAM, llamafile v0.8 runs Mixtral
+8x22B Q4\_0 on Linux at 98 tokens per second for evaluation, and it
+predicts 9.44 tokens per second.
+
+---
+
+# Model Card for Mixtral-8x22B-Instruct-v0.1
+The Mixtral-8x22B-Instruct-v0.1 Large Language Model (LLM) is an instruct fine-tuned version of the [Mixtral-8x22B-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-v0.1).
+
+## Run the model 
+```python
+from transformers import AutoModelForCausalLM
+from mistral_common.protocol.instruct.messages import (
+    AssistantMessage,
+    UserMessage,
+)
+from mistral_common.protocol.instruct.tool_calls import (
+    Tool,
+    Function,
+)
+from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
+from mistral_common.tokens.instruct.normalize import ChatCompletionRequest
+
+device = "cuda" # the device to load the model onto
+
+tokenizer_v3 = MistralTokenizer.v3()
+
+mistral_query = ChatCompletionRequest(
+    tools=[
+        Tool(
+            function=Function(
+                name="get_current_weather",
+                description="Get the current weather",
+                parameters={
+                    "type": "object",
+                    "properties": {
+                        "location": {
+                            "type": "string",
+                            "description": "The city and state, e.g. San Francisco, CA",
+                        },
+                        "format": {
+                            "type": "string",
+                            "enum": ["celsius", "fahrenheit"],
+                            "description": "The temperature unit to use. Infer this from the users location.",
+                        },
+                    },
+                    "required": ["location", "format"],
+                },
+            )
+        )
+    ],
+    messages=[
+        UserMessage(content="What's the weather like today in Paris"),
+    ],
+    model="test",
+)
+
+encodeds = tokenizer_v3.encode_chat_completion(mistral_query).tokens
+model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x22B-Instruct-v0.1")
+model_inputs = encodeds.to(device)
+model.to(device)
+
+generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
+sp_tokenizer = tokenizer_v3.instruct_tokenizer.tokenizer
+decoded = sp_tokenizer.decode(generated_ids[0])
+print(decoded)
+```
+
+# Instruct tokenizer
+The HuggingFace tokenizer included in this release should match our own. To compare: 
+`pip install mistral-common`
+
+```py
+from mistral_common.protocol.instruct.messages import (
+    AssistantMessage,
+    UserMessage,
+)
+from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
+from mistral_common.tokens.instruct.normalize import ChatCompletionRequest
+
+from transformers import AutoTokenizer
+
+tokenizer_v3 = MistralTokenizer.v3()
+
+mistral_query = ChatCompletionRequest(
+    messages=[
+        UserMessage(content="How many experts ?"),
+        AssistantMessage(content="8"),
+        UserMessage(content="How big ?"),
+        AssistantMessage(content="22B"),
+        UserMessage(content="Noice 🎉 !"),
+    ],
+    model="test",
+)
+hf_messages = mistral_query.model_dump()['messages']
+
+tokenized_mistral = tokenizer_v3.encode_chat_completion(mistral_query).tokens
+
+tokenizer_hf = AutoTokenizer.from_pretrained('mistralai/Mixtral-8x22B-Instruct-v0.1')
+tokenized_hf = tokenizer_hf.apply_chat_template(hf_messages, tokenize=True)
+
+assert tokenized_hf == tokenized_mistral
+```
+
+# Function calling and special tokens
+This tokenizer includes more special tokens, related to function calling : 
+- [TOOL_CALLS]
+- [AVAILABLE_TOOLS]
+- [/AVAILABLE_TOOLS]
+- [TOOL_RESULTS]
+- [/TOOL_RESULTS]
+
+If you want to use this model with function calling, please be sure to apply it similarly to what is done in our [SentencePieceTokenizerV3](https://github.com/mistralai/mistral-common/blob/main/src/mistral_common/tokens/tokenizers/sentencepiece.py#L299).
+
+# The Mistral AI Team
+Albert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Antoine Roux,
+Arthur Mensch, Audrey Herblin-Stoop, Baptiste Bout, Baudouin de Monicault,
+Blanche Savary, Bam4d, Caroline Feldman, Devendra Singh Chaplot,
+Diego de las Casas, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger,
+Gianna Lengyel, Guillaume Bour, Guillaume Lample, Harizo Rajaona,
+Jean-Malo Delignon, Jia Li, Justus Murke, Louis Martin, Louis Ternon,
+Lucile Saulnier, Lélio Renard Lavaud, Margaret Jennings, Marie Pellat,
+Marie Torelli, Marie-Anne Lachaux, Nicolas Schuhl, Patrick von Platen,
+Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao,
+Thibaut Lavril, Timothée Lacroix, Théophile Gervet, Thomas Wang,
+Valera Nemychnikova, William El Sayed, William Marshall