Instructions to use minsup562/KorMedQwen with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use minsup562/KorMedQwen with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="minsup562/KorMedQwen", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("minsup562/KorMedQwen", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use minsup562/KorMedQwen with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "minsup562/KorMedQwen"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "minsup562/KorMedQwen",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/minsup562/KorMedQwen

SGLang

How to use minsup562/KorMedQwen with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "minsup562/KorMedQwen" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "minsup562/KorMedQwen",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "minsup562/KorMedQwen" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "minsup562/KorMedQwen",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use minsup562/KorMedQwen with Docker Model Runner:
```
docker model run hf.co/minsup562/KorMedQwen
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Intro

This is a self-created project to finetune Qwen 7B chat model to Korean Med QA dataset (3104) Since the dataset is quite small, it was mixed with kollm-conversation dataset (5000) Used total of 8104 datapoints.

Compared to original Qwen model, KorMedQwen performs about 2% better. Models were evaluated using 200 test data. Original Qwen model got 19.5% correct while KorMedQwen got 21% correct.

Personally I think if we perform continuous pretraining on medical data then finetune with a lot more Korean Med QA data, it will perform a lot better.

QuickStart

from transformers import AutoModelForCausalLM, AutoTokenizer

# Note: The default behavior now has injection attack prevention off.
tokenizer = AutoTokenizer.from_pretrained("minsup562/KorMedQwen", trust_remote_code=True)

# use bf16
# model = AutoModelForCausalLM.from_pretrained("minsup562/KorMedQwen", device_map="auto", trust_remote_code=True, bf16=True).eval()
# use fp16
# model = AutoModelForCausalLM.from_pretrained("minsup562/KorMedQwen", device_map="auto", trust_remote_code=True, fp16=True).eval()
# use cpu only
# model = AutoModelForCausalLM.from_pretrained("minsup562/KorMedQwen", device_map="cpu", trust_remote_code=True).eval()
# use auto mode, automatically select precision based on the device.
model = AutoModelForCausalLM.from_pretrained("minsup562/KorMedQwen", device_map="auto", trust_remote_code=True).eval()

query = """다음 질문과 보기들을 보고 가장 알맞은 보기를 골라서 설명없이 답만 말해.
  14세 여학생이 소화불량과 복부팽만으로 어머니와 함께 병원에 왔다. 대학생인 남자친구와 성관계를 규칙적으로 하고 있으며 월경이 3개월째 없다고 한다.
  소변 임신반응검사 결과는 양성이다. 여학생은 진료실 밖에 있는 어머니에게 임신 사실은 절대 말하지 말고 그냥 장염이라고 말해 달라고 부탁을 한다.
  임신 사실 고지에 대한 조치는?
  A: 즉시 어머니를 불러 고지함 B: 나중에 어머니에게 전화로 고지함
  C: 일단, 어머니와 상의하도록 학생을 설득함 D: 학생의 부탁대로 장염이라고 말해줌
  E: 남자친구와 다시 방문하도록 권유함"""

response, history = model.chat(tokenizer, query, history=None)
print(response)

Downloads last month: 17

Safetensors

Model size

8B params

Tensor type

BF16

Model tree for minsup562/KorMedQwen

Quantizations

1 model