yokoe/llm-jp-3-13b-fintuned_wo_unsloth

モデル概要

  • ベースモデル: llm-jp/llm-jp-3-13b
  • データセット: ichikara-instruction

推論方法

下記のコードと同じディレクトリにelyza-tasks-100-TV_0.jsonl を配置した状態でコードを実行する。

!pip install -U bitsandbytes
!pip install -U transformers
!pip install -U accelerate
!pip install -U datasets
!pip install -U peft


import json
import os
from pathlib import Path

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
)
from peft import PeftModel
from tqdm import tqdm


bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    'llm-jp/llm-jp-3-13b',
    quantization_config=bnb_config,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
    'llm-jp/llm-jp-3-13b',
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(
    model,
    'yokoe/llm-jp-3-13b-fintuned_wo_unsloth',
)

# 推論対象データのロード
loaded_data = []
with open('./elyza-tasks-100-TV_0.jsonl', 'r') as f:
    item = ""
    for line in f:
      line = line.strip()
      item += line
      if item.endswith("}"):
        loaded_data.append(json.loads(item))
        item = ""

# 推論
results = []
for i, data in enumerate(tqdm(loaded_data)):

    input = data["input"]

    prompt = f"""### 指示
    {input}
    ### 回答
    """

    tokenized_input = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt").to(model.device)
    attention_mask = torch.ones_like(tokenized_input)
    with torch.no_grad():
      outputs = model.generate(
          tokenized_input,
          attention_mask=attention_mask,
          max_new_tokens=1024,
          do_sample=False,
          repetition_penalty=1.2,
          pad_token_id=tokenizer.eos_token_id
      )[0]
    output = tokenizer.decode(outputs[tokenized_input.size(1):], skip_special_tokens=True)
    results.append({"task_id": data["task_id"], "input": input, "output": output})

with open('./elyza-tasks-100-TV_0_preds.jsonl', 'w', encoding='utf-8') as f:
    for result in results:
        json.dump(result, f, ensure_ascii=False)  # ensure_ascii=False for handling non-ASCII characters
        f.write('\n')
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The model has no pipeline_tag.