unilm/adalm/incr_bpe/README.md

# Incremental BPE builder

<strong>Modified, simplified version of text_encoder_build_subword.py and its dependencies included in [tensor2tensor library](https://github.com/tensorflow/tensor2tensor), making its output fits to [google research's open-sourced BERT project](https://github.com/google-research/bert).</strong>

## Requirement
The environment I made this project in consists of :
- python 3.6
- tensorflow 1.11 

## Basic usage

#### Build domain-specific vocabulary automatically

If you want to build a proper size of vocabulary for specific domain using the incremental algorithm in our paper, you can do as following:

```bash
python vocab_extend.py \
	--corpus {file for the domain corpus} \
	--raw_vocab {bert_raw_vocab_file} \
	--output_file {he output file of the final vocabulary} \
	--interval {vocab size interval} \
	--threshold {threshold for P(D)}

# Example using sample data
python vocab_extend.py --corpus test_data/chem.txt \
	--raw_vocab test_data/vocab.txt \
	--output_file test_data/chem.vocab \
	--interval 1000 --threshold 1

```

If you simply want to get a specific size of vocab, you can run the following 

```
python subword_builder.py \
--corpus_filepattern {corpus_for_vocab} \
--raw_vocab {bert_raw_vocab_file} \
--output_filename {name_of_vocab} \
--vocab_size {final vocab size} \
--do_lower_case
```