transformers4token 4.20.0
State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
License: Apache
Requires Python: >=3.7.0
Keywords: NLP vision speech deep learning transformer pytorch tensorflow BERT GPT-2 Wav2Vec2 ViT

# bert蒸馏

## （非必须）将自有数据上预训练完的TF模型转化为Torch模型

首先要保证自己的环境版本是对的，因为我们修改的了源码。所以一些代码是在本地执行的，如果版本不对，本地执行的文件从远端的import可能会报错
cd /data/xiaoyichao/projects/DistilBERT
conda activate py38
pip install -r requirements.txt

使用原生模型测试 
    chinese_L-12_H-768_A-12_pytorch 这个文件夹需要新建

    export BERT_BASE_DIR=/data/search_opt_model/chinese_L-12_H-768_A-12
    export BERT_BASE_TORCH_DIR=/data/search_opt_model/chinese_L-12_H-768_A-12_pytorch

    cd /data/xiaoyichao/projects/DistilBERT/src/transformers4token/models/bert/
    conda activate py38

    python convert_bert_original_tf_checkpoint_to_pytorch.py \
    --tf_checkpoint_path="$BERT_BASE_DIR/bert_model.ckpt" \
    --bert_config_file="$BERT_BASE_DIR/bert_config.json" \
    --pytorch_dump_path="$BERT_BASE_TORCH_DIR/pytorch_model.bin"


使用自己预训练的模型，将自有数据上预训练完的TF模型转化为Torch模型，注意，这个参数tf_checkpoint_path的写法是不一样的
    chinese_L-12_H-768_A-12_pretrain_pytorch  这个文件夹需要新建

    export BERT_BASE_DIR=/data/search_opt_model/topk_opt/pretraining_12layer_output
    export BERT_BASE_TORCH_DIR=/data/search_opt_model/chinese_L-12_H-768_A-12_pretrain_pytorch

    cd /data/xiaoyichao/projects/DistilBERT/src/transformers4token/models/bert/
    conda activate py38

    python convert_bert_original_tf_checkpoint_to_pytorch.py \
    --tf_checkpoint_path="$BERT_BASE_DIR/" \
    --bert_config_file="$BERT_BASE_DIR/bert_config.json" \
    --pytorch_dump_path="$BERT_BASE_TORCH_DIR/pytorch_model.bin"


如果在BERT_BASE_TORCH_DIR/ 路径下没有config.json，就手动新增config.json 文件，内容如下（来源https://huggingface.co/bert-base-chinese/blob/main/config.json）：

    cd $BERT_BASE_TORCH_DIR/
    vim config.json
    输入英文冒号后，输入
    set paste
    回车，设置为粘贴模式
    粘贴的内容如下
{
"architectures": [
    "BertForMaskedLM"
],
"attention_probs_dropout_prob": 0.1,
"directionality": "bidi",
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"pooler_fc_size": 768,
"pooler_num_attention_heads": 12,
"pooler_num_fc_layers": 3,
"pooler_size_per_head": 128,
"pooler_type": "first_token_transform",
"type_vocab_size": 2,
"vocab_size": 21128
}

在BERT_BASE_TORCH_DIR/ 路径下新增/
vocab.txt 文件, 从chinese_L-12_H-768_A-12 的TF文件中复制过来即可/

    cp $BERT_BASE_DIR/vocab.txt $BERT_BASE_TORCH_DIR/

## 合并数据

conda activate py38
cd /data/xiaoyichao/projects/DistilBERT/examples/research_projects/distillation/scripts
python merge_txt.py

## 准备数据
### 将数据预处理为二进制

conda activate py38
cd /data/xiaoyichao/projects/DistilBERT/examples/research_projects/distillation

nohup python -u scripts/binarized_data.py \
    --file_path /data/search_opt_model/topk_opt/merged_pretraining_data.txt \
    --tokenizer_type bert \
    --tokenizer_name bert-base-chinese \
    --dump_file data/binarized_text > binarized_data.out 2>&1 &

### Preprocessing script before training the distilled model.
cd /data/xiaoyichao/projects/DistilBERT/examples/research_projects/distillation
python scripts/token_counts.py \
    --data_file ./data/binarized_text.bert-base-chinese.pickle \
    --token_counts_dump data/token_counts.bert-base-chinese.pickle \
    --vocab_size 21128

这个执行的很快

## 初始化学生模型的参数（如果处理过了，后边可以不处理） 在训练之前，转化bert模型，将m（m=12或24）层的bert转化成n（n<m）层的bert 

conda activate py38
cd /data/xiaoyichao/projects/DistilBERT/examples/research_projects/distillation/scripts

使用自己的模型
python extract_distilbert.py \
--dump_checkpoint /data/search_opt_model/topk_opt/distilbert/serialization_dir/tf_bert-base-uncased_20220928.pth \
--model_name $BERT_BASE_TORCH_DIR


其中 dump_checkpoint  就是保存的模型文件的路径 与 训练阶段的参数student_pretrained_weights 对应， 从老师模型中初始化的参数，更容易被学习的充分。


## 对蒸馏模型使用bert_config参数进行修改

vim /data/search_opt_model/topk_opt/distilbert/training_configs/distilbert-base-chinese.json

新增
"type_vocab_size": 8, 

## 训练

conda activate py38

cd /data/xiaoyichao/projects/DistilBERT/examples/research_projects/distillation

export BERT_BASE_TORCH_DIR=/data/search_opt_model/chinese_L-12_H-768_A-12_pretrain_pytorch
export OUTPUT_TORCH_DIR=/data/search_opt_model/topk_opt/distilbert/distilbert_torch


nohup python -u train.py \
    --student_type distilbert \
    --student_config /data/search_opt_model/topk_opt/distilbert/training_configs/distilbert-base-chinese.json \
    --teacher_type bert \
    --teacher_name bert-base-chinese  \
    --teacher_path $BERT_BASE_TORCH_DIR \
    --alpha_ce 5.0 --alpha_mlm 2.0 --alpha_cos 1.0 --alpha_clm 0.0 --mlm \
    --freeze_pos_embs \
    --batch_size 10 \
    --gradient_accumulation_steps 50 \
    --dump_path  $OUTPUT_TORCH_DIR \
    --data_file /data/search_opt_model/topk_opt/distilbert/data/binarized_text.bert-base-chinese.pickle \
    --token_counts /data/search_opt_model/topk_opt/distilbert/data/token_counts.bert-base-chinese.pickle \
    --student_pretrained_weights /data/search_opt_model/topk_opt/distilbert/serialization_dir/tf_bert-base-uncased_20220928.pth \
    --force > train.out 2>&1 &

tail -f train.out

batch_size=10 and gradient_accumulation_steps=50  的时候，显存消耗12G 
batch_size=50 and gradient_accumulation_steps=50  的时候，显存消耗30G 

这个数据有自己的预处理
07/14/2022 17:21:19 - INFO - utils - PID: 26970 -  Loading data from data/binarized_text.bert-base-chinese.pickle
07/14/2022 17:21:37 - INFO - utils - PID: 26970 -  Loading token counts from data/token_counts.bert-base-chinese.pickle (already pre-computed)
07/14/2022 17:22:07 - INFO - utils - PID: 26970 -  Splitting 1228 too long sequences.
07/14/2022 17:22:13 - INFO - utils - PID: 26003 -  Remove 5871110 too short (<=11 tokens) sequences.
07/14/2022 17:22:28 - INFO - utils - PID: 26003 -  Remove 3745 sequences with a high level of unknown tokens (50%).
07/14/2022 17:22:32 - INFO - utils - PID: 26003 -  8288747 sequences
07/14/2022 17:22:32 - INFO - utils - PID: 26003 -  Data loader created.

## 检查蒸馏出来的模型文件夹里的 bert_config的type_vocab_size 复制 vocab.txt

cp /data/search_opt_model/chinese_L-12_H-768_A-12_pretrain_pytorch/vocab.txt $OUTPUT_TORCH_DIR

cat $OUTPUT_TORCH_DIR/config.json ，没有这个文件就新建


检查这两个参数
"type_vocab_size": 8, 
"n_layers": 2,

如果在$OUTPUT_TORCH_DIR/ 路径下没有config.json，就手动新增config.json 文件，内容如下（来源https://huggingface.co/bert-base-chinese/blob/main/config.json）：

    cd $BERT_BASE_TORCH_DIR/
    vim config.json
    输入英文冒号后，输入
    set paste
    回车，设置为粘贴模式
    粘贴的内容如下
{
"architectures": [
    "BertForMaskedLM"
],
"attention_probs_dropout_prob": 0.1,
"directionality": "bidi",
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 12,
"num_hidden_layers": 2,
"pad_token_id": 0,
"pooler_fc_size": 768,
"pooler_num_attention_heads": 12,
"pooler_num_fc_layers": 3,
"pooler_size_per_head": 128,
"pooler_type": "first_token_transform",
"type_vocab_size": 8,
"vocab_size": 21128
}



## torch模型转化为TF模型
conda activate py38

export distilbert_tf_DIR=/data/search_opt_model/topk_opt/distilbert/distilbert_tf

python /data/xiaoyichao/projects/DistilBERT/examples/research_projects/distillation/convert_torch2tf.py \
        --model_name  bert_model \
        --pytorch_model_path  $OUTPUT_TORCH_DIR/pytorch_model.bin \
        --tf_cache_dir  $distilbert_tf_DIR


其实model_name 叫 bert_model不是很准确，但是为了后续的使用上，直接替换谷歌的Bert模型，所以叫这个名字，其实叫distilebert4hhz更合适，这个只是ckpt文件的命名时候使用  

## 复制TF版本的模型里的bert_config.json(torch 里边叫 config.json) 和 vocab.txt 文件到转化好的TF的BERT模型,注意修改bert_config里的"type_vocab_size": 8, 

cp /data/search_opt_model/chinese_L-12_H-768_A-12/vocab.txt $distilbert_tf_DIR
cp /data/search_opt_model/chinese_L-12_H-768_A-12/bert_config.json $distilbert_tf_DIR

cp /data/search_opt_model/chinese_L-12_H-768_A-12_pretrain_pytorch/vocab.txt $OUTPUT_TORCH_DIR
cp /data/search_opt_model/chinese_L-12_H-768_A-12_pretrain_pytorch/config.json $OUTPUT_TORCH_DIR

vim  $distilbert_tf_DIR/bert_config.json
bert_config.json中
"type_vocab_size": 8, 
"n_layers": 2,

把模型文件夹复制到/data/search_opt_model，其他模型使用的时候用这个路径/data/search_opt_model/distilbert_tf 的，防止自己做实验的时候影响其他项目的使用。
cp -r $distilbert_tf_DIR  /data/search_opt_model