shaowenchen
Update README.md
a172f59
metadata
inference: false
language:
  - zh
  - en
license: other
model_creator: Photolens
model_link: https://huggingface.co/Photolens/llama-2-7b-langchain-chat
model_name: llama-2-7b-langchain-chat
model_type: llama
pipeline_tag: text-generation
quantized_by: shaowenchen
tasks:
  - text2text-generation
tags:
  - meta
  - gguf
  - llama
  - llama-2
  - chinese
  - 7b

Provided files

Name Quant method Size
llama-2-7b-langchain-chat.Q2_K.gguf Q2_K 2.6 GB
llama-2-7b-langchain-chat.Q3_K.gguf Q3_K 3.1 GB
llama-2-7b-langchain-chat.Q3_K_L.gguf Q3_K_L 3.3 GB
llama-2-7b-langchain-chat.Q3_K_S.gguf Q3_K_S 2.7 GB
llama-2-7b-langchain-chat.Q4_0.gguf Q4_0 3.6 GB
llama-2-7b-langchain-chat.Q4_1.gguf Q4_1 3.9 GB
llama-2-7b-langchain-chat.Q4_K.gguf Q4_K 3.8 GB
llama-2-7b-langchain-chat.Q4_K_S.gguf Q4_K_S 3.6 GB
llama-2-7b-langchain-chat.Q5_0.gguf Q5_0 4.3 GB
llama-2-7b-langchain-chat.Q5_1.gguf Q5_1 4.7 GB
llama-2-7b-langchain-chat.Q5_K.gguf Q5_K 4.5 GB
llama-2-7b-langchain-chat.Q5_K_S.gguf Q5_K_S 4.3 GB
llama-2-7b-langchain-chat.Q6_K.gguf Q6_K 5.1 GB
llama-2-7b-langchain-chat.Q8_0.gguf Q8_0 6.7 GB
llama-2-7b-langchain-chat.gguf full 13 GB

Usage:

docker run --rm -it -p 8000:8000 -v /path/to/models:/models -e MODEL=/models/gguf-model-name.gguf hubimage/llama-cpp-python:latest

and you can view http://localhost:8000/docs to see the swagger UI.

Provided images

Name Quant method Size
shaowenchen/llama-2-7b-langchain-chat-gguf:Q2_K Q2_K 6.72 GB
shaowenchen/llama-2-7b-langchain-chat-gguf:Q3_K Q3_K 7.64 GB
shaowenchen/llama-2-7b-langchain-chat-gguf:Q3_K_L Q3_K_L 8.27 GB
shaowenchen/llama-2-7b-langchain-chat-gguf:Q3_K_S Q3_K_S 6.97 GB
shaowenchen/llama-2-7b-langchain-chat-gguf:Q4_0 Q4_0 8.55 GB
shaowenchen/llama-2-7b-langchain-chat-gguf:Q4_1 Q4_1 9.41 GB
shaowenchen/llama-2-7b-langchain-chat-gguf:Q4_K Q4_K 9.17 GB
shaowenchen/llama-2-7b-langchain-chat-gguf:Q4_K_S Q4_K_S 8.72 GB
shaowenchen/llama-2-7b-langchain-chat-gguf:Q5_0 Q5_0 10.4 GB
shaowenchen/llama-2-7b-langchain-chat-gguf:Q5_K Q5_K 10.6 GB
shaowenchen/llama-2-7b-langchain-chat-gguf:Q5_K_S Q5_K_S 10.4 GB

Usage:

docker run --rm -p 8000:8000 shaowenchen/llama-2-7b-langchain-chat-gguf:Q2_K

and you can view http://localhost:8000/docs to see the swagger UI.