oysu/Seonu

$ source .etc/Script/activate

BERT

자체 데이터로 Bert Training 하기 RoBERTa/BERT/DistilBERT and masked language modeling pretrainging

아래 경로에 model 생성됨 D:\tmp\test-mlm

python run_mlm.py \
    --model_name_or_path roberta-base \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --do_train \
    --do_eval \
    --output_dir /tmp/test-mlm

ref

• pytorch
• [HuggingFace 튜토리얼] 1. Quick Tour - Pipeline & AutoClass
• BERT
• python virtualenv
• CUDA/cuDNN
• CUDA Tookit 12.1(latest
• cuDDN

repo

• Seonu