llama-2-ko-70b-GPTQ / README.md
kuotient's picture
Update README.md
1aaab80
metadata
base_model: https://huggingface.co/beomi/llama-2-ko-70b
inference: false
language:
  - en
  - ko
model_name: Llama 2 7B Chat
model_type: llama
pipeline_tag: text-generation
quantized_by: kuotient
tags:
  - facebook
  - meta
  - pytorch
  - llama
  - llama-2
  - kollama
  - llama-2-ko
  - gptq
license: cc-by-nc-sa-4.0

Llama-2-Ko-70b-GPTQ

  • ๋ชจ๋ธ ์ œ์ž‘์ž: beomi
  • ์›๋ณธ ๋ชจ๋ธ: Llama-2-ko-70b

Description

์ด ๋ ˆํฌ๋Š” Llama-2-ko-70b์˜ GPTQ ๋ชจ๋ธ ํŒŒ์ผ์„ ํฌํ•จํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค.

Provided files and GPTQ parameters

ํ•˜๋“œ์›จ์–ด์™€ ์š”๊ตฌ์‚ฌํ•ญ์— ๊ฐ€์žฅ ์ ํ•ฉํ•œ ์–‘์žํ™” ๋งค๊ฐœ๋ณ€์ˆ˜๋ฅผ ์„ ํƒํ•  ์ˆ˜ ์žˆ๋„๋ก ์—ฌ๋Ÿฌ ๊ฐ€์ง€(๊ณง) ์–‘์žํ™” ๋งค๊ฐœ๋ณ€์ˆ˜๊ฐ€ ์ œ๊ณต๋ฉ๋‹ˆ๋‹ค. ๊ฐ ์–‘์žํ™”๋Š” ๋‹ค๋ฅธ ๋ธŒ๋žœ์น˜์— ์žˆ์Šต๋‹ˆ๋‹ค. ๋ชจ๋“  GPTQ ์–‘์žํ™”๋Š” AutoGPTQ๋กœ ๋งŒ๋“ค์–ด์กŒ์Šต๋‹ˆ๋‹ค.

GPTQ ํŒŒ๋ผ๋ฏธํ„ฐ ์ •๋ณด
  • Bits: ์–‘์žํ™”๋œ ๋ชจ๋ธ์˜ ๋น„ํŠธ ํฌ๊ธฐ์ž…๋‹ˆ๋‹ค.
  • GS: GPTQ ๊ทธ๋ฃน ์‚ฌ์ด์ฆˆ. ์ˆซ์ž๊ฐ€ ๋†’์„์ˆ˜๋ก VRAM์„ ๋œ ์‚ฌ์šฉํ•˜์ง€๋งŒ ์–‘์žํ™” ์ •ํ™•๋„๊ฐ€ ๋‚ฎ์•„์ง‘๋‹ˆ๋‹ค. "None"์€ ๊ฐ€๋Šฅํ•œ ๊ฐ€์žฅ ๋‚ฎ์€ ๊ฐ’์ž…๋‹ˆ๋‹ค.
  • Act Order: True or False. desc_act์œผ๋กœ๋„ ์•Œ๋ ค์ ธ ์žˆ์Šต๋‹ˆ๋‹ค. ์ฐธ์ด๋ฉด ์–‘์žํ™” ์ •ํ™•๋„๊ฐ€ ํ–ฅ์ƒ๋ฉ๋‹ˆ๋‹ค.
  • Damp %: ์ƒ˜ํ”Œ์ด ์ •๋Ÿ‰ํ™”๋ฅผ ์œ„ํ•ด ์ฒ˜๋ฆฌ๋˜๋Š” ๋ฐฉ์‹์— ์˜ํ–ฅ์„ ์ฃผ๋Š” GPTQ ๋งค๊ฐœ๋ณ€์ˆ˜์ž…๋‹ˆ๋‹ค. 0.01์ด ๊ธฐ๋ณธ๊ฐ’์ด์ง€๋งŒ 0.1์„ ์‚ฌ์šฉํ•˜๋ฉด ์ •ํ™•๋„๊ฐ€ ์•ฝ๊ฐ„ ํ–ฅ์ƒ๋ฉ๋‹ˆ๋‹ค.
  • GPTQ dataset: ์ •๋Ÿ‰ํ™”์— ์‚ฌ์šฉ๋˜๋Š” ๋ฐ์ดํ„ฐ ์„ธํŠธ์ž…๋‹ˆ๋‹ค. ๋ชจ๋ธ ํ•™์Šต์— ๋” ์ ํ•ฉํ•œ ๋ฐ์ดํ„ฐ ์„ธํŠธ๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด ์ •๋Ÿ‰ํ™” ์ •ํ™•๋„๊ฐ€ ํ–ฅ์ƒ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. GPTQ ๋ฐ์ดํ„ฐ ์„ธํŠธ๋Š” ๋ชจ๋ธ ํ•™์Šต์— ์‚ฌ์šฉ๋œ ๋ฐ์ดํ„ฐ ์„ธํŠธ์™€ ๋™์ผํ•˜์ง€ ์•Š์œผ๋ฏ€๋กœ ํ•™์Šต ๋ฐ์ดํ„ฐ ์„ธํŠธ์— ๋Œ€ํ•œ ์ž์„ธํ•œ ๋‚ด์šฉ์€ ์›๋ณธ ๋ชจ๋ธ repo๋ฅผ ์ฐธ์กฐํ•˜์„ธ์š”.
  • Sequence Length: ์ •๋Ÿ‰ํ™”์— ์‚ฌ์šฉ๋œ ๋ฐ์ดํ„ฐ ์„ธํŠธ ์‹œํ€€์Šค์˜ ๊ธธ์ด์ž…๋‹ˆ๋‹ค. ์ด์ƒ์ ์œผ๋กœ๋Š” ๋ชจ๋ธ ์‹œํ€€์Šค ๊ธธ์ด์™€ ๋™์ผํ•ฉ๋‹ˆ๋‹ค. ์ผ๋ถ€ ๋งค์šฐ ๊ธด ์‹œํ€€์Šค ๋ชจ๋ธ(16+K)์˜ ๊ฒฝ์šฐ ๋” ์งง์€ ์‹œํ€€์Šค ๊ธธ์ด๋ฅผ ์‚ฌ์šฉํ•ด์•ผ ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์‹œํ€€์Šค ๊ธธ์ด๊ฐ€ ์งง๋‹ค๊ณ  ํ•ด์„œ ์–‘์žํ™”๋œ ๋ชจ๋ธ์˜ ์‹œํ€€์Šค ๊ธธ์ด๊ฐ€ ์ œํ•œ๋˜๋Š” ๊ฒƒ์€ ์•„๋‹™๋‹ˆ๋‹ค. ์ด๋Š” ๊ธด ์ถ”๋ก  ์‹œํ€€์Šค์˜ ์–‘์žํ™” ์ •ํ™•๋„์—๋งŒ ์˜ํ–ฅ์„ ๋ฏธ์นฉ๋‹ˆ๋‹ค.
  • ExLlama Compatibility: Exllama๋กœ ์ด ํŒŒ์ผ์„ ๋กœ๋“œํ•  ์ˆ˜ ์žˆ๋Š”์ง€์˜ ์—ฌ๋ถ€์ด๋ฉฐ, ํ˜„์žฌ 4๋น„ํŠธ์˜ ๋ผ๋งˆ ๋ชจ๋ธ๋งŒ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค.
Branch Bits GS Act Order Damp % GPTQ Dataset Seq Len Size ExLlama Desc
main 4 None Yes 0.1 wikitext 4096 35.8 GB Yes 4-bit, Act Order ํฌํ•จ. VRAM ์‚ฌ์šฉ๋Ÿ‰์„ ์ค„์ด๊ธฐ ์œ„ํ•œ group size -1.

Original model card: Llama 2 ko 70b

๐Ÿšง Note: this repo is under construction ๐Ÿšง

Llama-2-Ko ๐Ÿฆ™๐Ÿ‡ฐ๐Ÿ‡ท

Llama-2-Ko serves as an advanced iteration of Llama 2, benefiting from an expanded vocabulary and the inclusion of a Korean corpus in its further pretraining. Just like its predecessor, Llama-2-Ko operates within the broad range of generative text models that stretch from 7 billion to 70 billion parameters. This repository focuses on the 70B pretrained version, which is tailored to fit the Hugging Face Transformers format. For access to the other models, feel free to consult the index provided below.

Model Details

Model Developers Junbum Lee (Beomi)

Variations Llama-2-Ko will come in a range of parameter sizes โ€” 7B, 13B, and 70B โ€” as well as pretrained and fine-tuned variations.

Input Models input text only.

Output Models generate text only.

Usage

Use with 8bit inference

  • Requires > 74GB vram (compatible with 4x RTX 3090/4090 or 1x A100/H100 80G or 2x RTX 6000 ada/A6000 48G)
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
model_8bit = AutoModelForCausalLM.from_pretrained(
    "beomi/llama-2-ko-70b", 
    load_in_8bit=True,
    device_map="auto",
)
tk = AutoTokenizer.from_pretrained('beomi/llama-2-ko-70b')
pipe = pipeline('text-generation', model=model_8bit, tokenizer=tk)
def gen(x):
    gended = pipe(f"### Title: {x}\n\n### Contents:",  # Since it this model is NOT finetuned with Instruction dataset, it is NOT optimal prompt.
        max_new_tokens=300,
        top_p=0.95,
        do_sample=True,
    )[0]['generated_text']
    print(len(gended))
    print(gended)

Use with bf16 inference

  • Requires > 150GB vram (compatible with 8x RTX 3090/4090 or 2x A100/H100 80G or 4x RTX 6000 ada/A6000 48G)
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
model = AutoModelForCausalLM.from_pretrained(
    "beomi/llama-2-ko-70b", 
    device_map="auto",
)
tk = AutoTokenizer.from_pretrained('beomi/llama-2-ko-70b')
pipe = pipeline('text-generation', model=model, tokenizer=tk)
def gen(x):
    gended = pipe(f"### Title: {x}\n\n### Contents:",  # Since it this model is NOT finetuned with Instruction dataset, it is NOT optimal prompt.
        max_new_tokens=300,
        top_p=0.95,
        do_sample=True,
    )[0]['generated_text']
    print(len(gended))
    print(gended)

Model Architecture

Llama-2-Ko is an auto-regressive language model that uses an optimized transformer architecture based on Llama-2.

Training Data Params Content Length GQA Tokens LR
Llama-2-Ko 70B A new mix of Korean online data 70B 4k โœ… >20B 1e-5
*Plan to train upto 300B tokens
Vocab Expansion
Model Name Vocabulary Size Description
--- --- ---
Original Llama-2 32000 Sentencepiece BPE
Expanded Llama-2-Ko 46592 Sentencepiece BPE. Added Korean vocab and merges
*Note: Llama-2-Ko 70B uses 46592 not 46336(7B), will update new 7B model soon.

Tokenizing "์•ˆ๋…•ํ•˜์„ธ์š”, ์˜ค๋Š˜์€ ๋‚ ์”จ๊ฐ€ ์ข‹๋„ค์š”. ใ…Žใ…Ž"

Model Tokens
Llama-2 ['โ–', '์•ˆ', '<0xEB>', '<0x85>', '<0x95>', 'ํ•˜', '์„ธ', '์š”', ',', 'โ–', '์˜ค', '<0xEB>', '<0x8A>', '<0x98>', '์€', 'โ–', '<0xEB>', '<0x82>', '<0xA0>', '์”จ', '๊ฐ€', 'โ–', '<0xEC>', '<0xA2>', '<0x8B>', '<0xEB>', '<0x84>', '<0xA4>', '์š”', '.', 'โ–', '<0xE3>', '<0x85>', '<0x8E>', '<0xE3>', '<0x85>', '<0x8E>']
Llama-2-Ko *70B ['โ–์•ˆ๋…•', 'ํ•˜์„ธ์š”', ',', 'โ–์˜ค๋Š˜์€', 'โ–๋‚ ', '์”จ๊ฐ€', 'โ–์ข‹๋„ค์š”', '.', 'โ–', 'ใ…Ž', 'ใ…Ž']
Tokenizing "Llama 2: Open Foundation and Fine-Tuned Chat Models"
Model Tokens
--- ---
Llama-2 ['โ–L', 'l', 'ama', 'โ–', '2', ':', 'โ–Open', 'โ–Foundation', 'โ–and', 'โ–Fine', '-', 'T', 'un', 'ed', 'โ–Ch', 'at', 'โ–Mod', 'els']
Llama-2-Ko 70B ['โ–L', 'l', 'ama', 'โ–', '2', ':', 'โ–Open', 'โ–Foundation', 'โ–and', 'โ–Fine', '-', 'T', 'un', 'ed', 'โ–Ch', 'at', 'โ–Mod', 'els']

Model Benchmark

LM Eval Harness - Korean (polyglot branch)

TBD

Note for oobabooga/text-generation-webui

Remove ValueError at load_tokenizer function(line 109 or near), in modules/models.py.

diff --git a/modules/models.py b/modules/models.py
index 232d5fa..de5b7a0 100644
--- a/modules/models.py
+++ b/modules/models.py
@@ -106,7 +106,7 @@ def load_tokenizer(model_name, model):
                 trust_remote_code=shared.args.trust_remote_code,
                 use_fast=False
             )
-        except ValueError:
+        except:
             tokenizer = AutoTokenizer.from_pretrained(
                 path_to_model,
                 trust_remote_code=shared.args.trust_remote_code,

Since Llama-2-Ko uses FastTokenizer provided by HF tokenizers NOT sentencepiece package, it is required to use use_fast=True option when initialize tokenizer. Apple Sillicon does not support BF16 computing, use CPU instead. (BF16 is supported when using NVIDIA GPU)

LICENSE

Citation

@misc {l._junbum_2023,
    author       = { {L. Junbum} },
    title        = { llama-2-ko-70b },
    year         = 2023,
    url          = { https://huggingface.co/beomi/llama-2-ko-70b },
    doi          = { 10.57967/hf/1130 },
    publisher    = { Hugging Face }
}