Update README.md

README.md

@@ -103,56 +103,154 @@ Refer to the Provided Files table below to see what files use which methods, and
 | [tc-instruct-dpo.Q8_0.gguf](https://huggingface.co/pek111/TC-instruct-DPO-GGUF/blob/main/tc-instruct-dpo.Q8_0.gguf) | Q8_0 | 8 | 7.19 GB | very large, extremely low quality loss - not recommended     |
 | [tc-instruct-dpo.QF16.gguf](https://huggingface.co/pek111/TC-instruct-DPO-GGUF/blob/main/tc-instruct-dpo.Q8_0.gguf) | QF16 | 16 | 13.53 GB | largest, lowest quality loss - highly not recommended |
 
-# Inference Code
+## How to download GGUF files
 
-Here is example code using HuggingFace Transformers to inference the model (note: in 4bit, it will require around 5GB of VRAM)
+**Note for manual downloaders:** You almost never want to clone the entire repo! Multiple different quantisation formats are provided, and most users only want to pick and download a single file.
 
-Note: To use function calling, you should see the github repo above.
+The following clients/libraries will automatically download models for you, providing a list of available models to choose from:
 
-```python
-# Requires pytorch, transformers, bitsandbytes, sentencepiece, protobuf, and flash-attn packages
+- LM Studio
+- LoLLMS Web UI
+- Faraday.dev
 
-import torch
-from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig, GenerationConfig
-import time
+### In `text-generation-webui`
 
-base_model_id = "tanamettpk/TC-instruct-DPO"
+Under Download Model, you can enter the model repo: TheBloke/Llama-2-13B-GGUF and below it, a specific filename to download, such as: llama-2-13b.q4_K_M.gguf.
 
-input_text = """
+Then click Download.
+
+### On the command line, including multiple files at once
+
+I recommend using the `huggingface-hub` Python library:
+
+```shell
+pip3 install huggingface-hub>=0.17.1
+```
+
+Then you can download any individual model file to the current directory, at high speed, with a command like this:
+
+```shell
+huggingface-cli download pek111/TC-instruct-DPO-GGUF tc-instruct-dpo.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
+```
+
+<details>
+  <summary>More advanced huggingface-cli download usage</summary>
+
+
+You can also download multiple files at once with a pattern:
+
+```shell
+huggingface-cli download pek111/TC-instruct-DPO-GGUF --local-dir . --local-dir-use-symlinks False --include='*Q4_K*gguf'
+```
+
+For more documentation on downloading with `huggingface-cli`, please see: [HF -> Hub Python Library -> Download files -> Download from the CLI](https://huggingface.co/docs/huggingface_hub/guides/download#download-from-the-cli).
+
+To accelerate downloads on fast connections (1Gbit/s or higher), install `hf_transfer`:
+
+```shell
+pip3 install hf_transfer
+```
+
+And set environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`:
+
+```shell
+HUGGINGFACE_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download TheBloke/Llama-2-13B-GGUF llama-2-13b.q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
+```
+
+Windows CLI users: Use `set HUGGINGFACE_HUB_ENABLE_HF_TRANSFER=1` or `$env:HUGGINGFACE_HUB_ENABLE_HF_TRANSFER=1` before running the download command.
+</details>
+
+## Example `llama.cpp` command
+
+Make sure you are using `llama.cpp` from commit [d0cee0d36d5be95a0d9088b674dbb27354107221](https://github.com/ggerganov/llama.cpp/commit/d0cee0d36d5be95a0d9088b674dbb27354107221) or later.
+
+```shell
+./main -ngl 32 -m tc-instruct-dpo.Q4_K_M.gguf --color -c 4096 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "{prompt}"
+```
+
+Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
+
+Change `-c 4096` to the desired sequence length. For extended sequence models - eg 8K, 16K, 32K - the necessary RoPE scaling parameters are read from the GGUF file and set by llama.cpp automatically.
+
+If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
+
+For other parameters and how to use them, please refer to [the llama.cpp documentation](https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md)
+
+## How to run in `text-generation-webui`
+
+Further instructions here: [text-generation-webui/docs/llama.cpp.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp.md).
+
+## How to run from Python code
+
+You can use GGUF models from Python using the [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) or [ctransformers](https://github.com/marella/ctransformers) libraries.
+
+### How to load this model from Python using ctransformers
+
+#### First install the package
+
+```shell
+# Base llama-cpp-python with no GPU acceleration
+pip install llama-cpp-python
+# With NVidia CUDA acceleration
+CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python
+# Or with OpenBLAS acceleration
+CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS" pip install llama-cpp-python
+# Or with CLBLast acceleration
+CMAKE_ARGS="-DLLAMA_CLBLAST=on" pip install llama-cpp-python
+# Or with AMD ROCm GPU acceleration (Linux only)
+CMAKE_ARGS="-DLLAMA_HIPBLAS=on" pip install llama-cpp-python
+# Or with Metal GPU acceleration for macOS systems only
+CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python
+
+# In windows, to set the variables CMAKE_ARGS in PowerShell, follow this format; eg for Nvidia CUDA:
+$env:CMAKE_ARGS = "-DLLAMA_CUDA=on"
+pip install llama_cpp_python --verbose
+# If BLAS = 0 try installing with these command instead (Windows + CUDA)
+set CMAKE_ARGS="-DLLAMA_CUDA=on"
+set FORCE_CMAKE=1
+$env:CMAKE_ARGS = "-DLLAMA_CUDA=on"
+$env:FORCE_CMAKE = 1
+python -m pip install llama_cpp_python>=0.2.26 --verbose --force-reinstall --no-cache-dir
+```
+
+#### Simple example code to load one of these GGUF models
+
+```python
+import llama_cpp
+
+llm_cpp = llama_cpp.Llama(
+    model_path="tc-instruct-dpo.Q4_K_M.gguf",  # Path to the model
+    n_threads=10,  # CPU cores
+    n_batch=512,  # Should be between 1 and n_ctx, consider the amount of VRAM in your GPU.
+    n_gpu_layers=35,  # Change this value based on your model and your GPU VRAM pool.
+    n_ctx=4096,  # Max context length
+)
+
+prompt = """
 ### Instruction:
 ด่าฉันด้วยคำหยาบคายหน่อย
 
 ### Response:
 """
 
-model = AutoModelForCausalLM.from_pretrained(
-    base_model_id,
-    low_cpu_mem_usage=True,
-    return_dict=True,
-    device_map={"": 0},
+response = llm_cpp(
+    prompt=prompt,
+    max_tokens=1024,
+    temperature=0.5,
+    top_k=1,
+    repeat_penalty=1.1,
+    echo=True
 )
-tokenizer = AutoTokenizer.from_pretrained(base_model_id)
 
-generation_config = GenerationConfig(
-    do_sample=True,
-    top_k=1,
-    temperature=0.5,
-    max_new_tokens=300,
-    repetition_penalty=1.1,
-    pad_token_id=tokenizer.eos_token_id)
+print(response)
+```
 
-# Tokenize input
-inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
+## How to use with LangChain
 
-# Generate outputs
-st_time = time.time()
-outputs = model.generate(**inputs, generation_config=generation_config)
+Here's guides on using llama-cpp-python or ctransformers with LangChain:
 
-# Decode and print response
-response = tokenizer.decode(outputs[0], skip_special_tokens=True)
-print(f"Response time: {time.time() - st_time} seconds")
-print(response)
-```
+* [LangChain + llama-cpp-python](https://python.langchain.com/docs/integrations/llms/llamacpp)
+* [LangChain + ctransformers](https://python.langchain.com/docs/integrations/providers/ctransformers)
 
 # Original model card: tanamettpk's TC Instruct DPO - Typhoon 7B