--- license: apache-2.0 tags: - merge --- # openmixtral-6x7b-merged_v2 openmixtral-6x7b-merged_v2 is a merge of the following models: ## 🧩 Configuration ```yaml base_model: mlabonne/Marcoro14-7B-slerp experts: - source_model: openchat/openchat-3.5-1210 positive_prompts: - "chat" - "assistant" - "tell me" - "explain" - source_model: Weyaxi/Einstein-v4-7B positive_prompts: - "physics" - "biology" - "chemistry" - "science" - source_model: BioMistral/BioMistral-7B positive_prompts: - "medical" - "pubmed" - "healthcare" - "health" - source_model: beowolx/CodeNinja-1.0-OpenChat-7B positive_prompts: - "code" - "python" - "javascript" - "programming" - "algorithm" - source_model: maywell/PiVoT-0.1-Starling-LM-RP positive_prompts: - "storywriting" - "write" - "scene" - "story" - "character" - source_model: WizardLM/WizardMath-7B-V1.1 positive_prompts: - "reason" - "math" - "mathematics" - "solve" - "count" tokenizer_source: union ``` ## 💻 Usage ```python # install llamacpp see here: https://github.com/ggerganov/llama.cpp # or other GGUF tool like llamacpp-python: https://github.com/abetlen/llama-cpp-python MODEL_REPO="openmixtral-6x7b-merged_v2-GGUF" MODEL_NAME="openmixtral-6x7b-merged_v2" method="Q4_K_M" prompt="why the sky is blue" qtype = f"{MODEL_REPO}/{MODEL_NAME.lower()}.{method.upper()}.gguf" !./llama.cpp/main -m {qtype} -n 128 --color -ngl 0 -p "{prompt}" ``` Log Result ``` Log start main: build = 2382 (621e86b3) main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu main: seed = 1710306347 ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes ggml_init_cublas: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes llama_model_loader: loaded meta data with 25 key-value pairs and 803 tensors from openmixtral-6x7b-merged_v2-GGUF/openmixtral-6x7b-merged_v2.Q4_K_M.gguf (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = . llama_model_loader: - kv 2: llama.context_length u32 = 32768 llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 llama_model_loader: - kv 4: llama.block_count u32 = 32 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 9: llama.expert_count u32 = 6 llama_model_loader: - kv 10: llama.expert_used_count u32 = 2 llama_model_loader: - kv 11: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 12: llama.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 13: general.file_type u32 = 15 llama_model_loader: - kv 14: tokenizer.ggml.model str = llama llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32000] = ["", "~~", "~~", "<0x00>", "<... llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 20: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 1 llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 24: general.quantization_version u32 = 2 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type f16: 32 tensors llama_model_loader: - type q4_K: 593 tensors llama_model_loader: - type q6_K: 113 tensors llm_load_vocab: special tokens definition check successful ( 259/32000 ). llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32000 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 6 llm_load_print_meta: n_expert_used = 2 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 35.43 B llm_load_print_meta: model size = 19.96 GiB (4.84 BPW) llm_load_print_meta: general.name = . llm_load_print_meta: BOS token = 1 '~~' llm_load_print_meta: EOS token = 2 '~~' llm_load_print_meta: UNK token = 0 '' llm_load_print_meta: PAD token = 1 '' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_tensors: ggml ctx size = 0.31 MiB llm_load_tensors: offloading 0 repeating layers to GPU llm_load_tensors: offloaded 0/33 layers to GPU llm_load_tensors: CPU buffer size = 20441.87 MiB .................................................................................................... llama_new_context_with_model: n_ctx = 512 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA_Host KV buffer size = 64.00 MiB llama_new_context_with_model: KV self size = 64.00 MiB, K (f16): 32.00 MiB, V (f16): 32.00 MiB llama_new_context_with_model: CUDA_Host input buffer size = 10.01 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 114.52 MiB llama_new_context_with_model: graph splits (measure): 1 system_info: n_threads = 12 / 24 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | sampling: repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000 top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800 mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000 sampling order: CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature generate: n_ctx = 512, n_batch = 512, n_predict = 128, n_keep = 1 why the sky is blue # Why The Sky is Blue: Rayleigh Scattering ## Introduction to Rayleigh Scattering Have you ever wondered why the sky appears blue during a clear day? You might think that it's because there are more blue molecules in the Earth's atmosphere, or maybe that the sun emits blue light. However, neither of these explanations is accurate. The true reason behind the blue color of the sky has to do with a phenomenon called Rayleigh scattering. Rayleigh scattering is a fundamental process in physics and atmospheric science that occurs when sunlight interacts with mole llama_print_timings: load time = 1605.68 ms llama_print_timings: sample time = 14.83 ms / 128 runs ( 0.12 ms per token, 8632.32 tokens per second) llama_print_timings: prompt eval time = 312.86 ms / 6 tokens ( 52.14 ms per token, 19.18 tokens per second) llama_print_timings: eval time = 21083.26 ms / 127 runs ( 166.01 ms per token, 6.02 tokens per second) llama_print_timings: total time = 21439.97 ms / 133 tokens Log end ``` ## Quantized Model Response can you spot the difference? ``` #### User Prompt why the sky is blue #### Q2_K Response The sky appears blue because of a phenomenon called Rayleigh scattering. This process occurs when sunlight (which is made up of many different colors or wavelengths of light) interacts with particles in the atmosphere like air molecules, water vapor and dust. The shorter wavelengths of light, such as violet and blue, are more easily scattered than longer wavelengths like red and yellow, due to their smaller size compared to the wavelengths they're interacting with. When we look up at the sky, most of these scattered shorter-wavelength lights (violet, blue) are #### Q3_K_M Response In the late 19th century, an English scientist named Lord Rayleigh and a Scottish scientist named William Thomson (later Lord Kelvin) embarked on a quest to solve one of the most enduring mysteries in science: why is the sky blue? Their findings, which are now widely accepted, can be attributed to a combination of physics principles and molecular behavior. The color we perceive in the sky is actually the result of how sunlight interacts with various gases and particles in our atmosphere. Sunlight is composed of multiple colors, or wavelengths, ranging from red at one end #### Q4_K_M Response The reason why we see the sky as blue is due to a natural phenomenon known as Rayleigh scattering. When sunlight enters the Earth’s atmosphere, it is made up of numerous colors or wavelengths, including blue, green, yellow, orange, and red. Rayleigh scattering occurs when the shorter-wavelength light, such as blue and violet colors, interacts with the molecules (mainly nitrogen and oxygen) in the atmosphere. These tiny particles scatter these short-wave lengths more effectively than longer ones like red or yellow, which results in the sky appearing blue to our eyes. #### Q5_K_M Response the blue color of the sky, a phenomenon that has been observed and admired since ancient times, can be explained through the principles of physics and atmospheric science. Sunlight, which appears white to us, is actually made up of many colors of light, each with a different wavelength. These colors include red, orange, yellow, green, blue, indigo, and violet; together, they are known as the visible light spectrum. When sunlight enters Earth's atmosphere, it encounters various gases, such as nitrogen and oxygen, which scatter the shorter wavelengths of light. #### Q6_K Response In order to understand why the sky appears blue, we must first understand how light behaves and interacts with air molecules. Visible light is a form of electromagnetic radiation that spans wavelengths between approximately 400 nanometers (violet) and 700 nanometers (red). When sunlight reaches Earth's atmosphere, it encounters various gases and particles that scatter or absorb different wavelengths of light. The scattering of light is due to the interaction between electromagnetic waves and free-moving electrons within air molecules, particularly nitrogen and oxygen...etc #### Q8_0 Response I’m just curious, it seems like a lot of people know this but I can’t recall ever being taught. Why exactly is the sky blue? I know it reflects sunlight and so forth but how does that lead to the blue color? The reason the sky appears blue has to do with a combination of Rayleigh scattering and Earth's atmosphere. Here's a brief explanation: 1. Sunlight consists of multiple colors (wavelengths) including red, orange, yellow, green, blue, indigo, and violet. These different colors are represented by their respective wavelengths in...etc ```