Create README.md

README.md

@@ -0,0 +1,146 @@
+# WudaoSailing
+
+WudaoSailing is a package for pretraining chinese Language Model and finetune tasks. Now it supports GLM, Bert, T5, Cogview and Roberta models.
+
+
+
+
+## Get Started
+### Docker Image
+We prepare two docker images based on CUDA 10.2 and CUDA 11.2. You can build images from the docker file [docs/docker/cuda102.dockerfile](docs/docker/cuda102.dcokerfile) or pull the pre-built images from Docker Hub and run with docker v19.03+
+  ```shell
+  nvidia-docker run -id  --hostname=V100  --network=host\
+  --ipc=host --shm-size=16gb --name=deepspeed-cuda \  
+  -e NVIDIA_VISIBLE_DEVICES=0,1,2,3 \
+  -v /DATA/disk1/docker/containers/:/data deepspeed/cuda102:lastest
+  ```
+  or replace `cuda102` with `cuda112`.
+
+  ```shell
+    docker build -f cuda102.dockerfile  -t deepspeed/cuda102 .
+  ```
+
+### Clone this repo
+  ```shell
+  git clone https://github.com/wangguojim/WudaoSailing.git
+  cd WudaoSailing
+  pip install -r requirements.txt
+  ```
+
+## GLM
+
+We show some examples based on GLM model.
+
+### finetuene
+We provide scripts for finetuning GLM on some downstream tasks.
+
+#### SuperGLUE
+
+- Download the [SuperGlue](https://super.gluebenchmark.com/tasks) data and check the experiment setup in 
+  [examples/glm/scripts/ds_finetune_superglue.sh](xamples/glm/scripts/ds_finetune_superglue.sh). Note that `DATA_ROOT, CHECKPOINT_PATH, SAVE_PATH` 
+  need to be changed to your local path. You may also change the `batch-size` and `nproc_per_node` according to your 
+  available hardware.
+
+- Run the following script for text similarity finetune task (use the afqmc dataset as an example)
+
+```
+cd examples/glm/ 
+bash scripts/ds_finetune_superglue.sh\
+     config/model_blocklm_large_chinese.sh\
+     config_tasks/task_afqmc.sh
+```
+
+
+- Run the following script for text classification finetune task  (use the thunews and thunews dataset as an example)
+
+```
+cd examples/glm/ 
+bash scripts/ds_finetune_superglue.sh\
+     config/model_blocklm_large_chinese.sh\
+     config_tasks/task_tnews.sh
+```
+
+- Run the following script  for causal inference finetune task (use the COPA dataset as an example)
+
+```
+cd examples/glm/ 
+bash scripts/ds_finetune_superglue.sh\
+     config/model_blocklm_large_chinese.sh\
+     config_tasks/task_copa.sh
+```
+  
+- To apply GLM to a new NLU dataset with cloze-filling finetuning, implement a `DataProcessor` in
+  [examples/glm/tasks/superglue/dataset.py](examples/glm/tasks/superglue/dataset.py) for data loading and add a `PVP` in 
+  [examples/glm/tasks/superglue/pvp.py](examples/glm/tasks/superglue/pvp.py) for the cloze question. More details can be found 
+  [here](examples/glm/tasks/superglue/README.md).
+
+
+
+#### Blank Filling (Interactive)
+* Change `CHECKPOINT_PATH` to your local path. Run the following script
+```
+bash config/generate_block.sh\
+     config/model_blocklm_large_chinese.sh
+```
+##### Example1 (Entity Prediction):
+
+Context: 凯旋门位于意大利米兰市古城堡旁。1807年为纪念[MASK]而建，门高25米，顶上矗立两武士青铜古兵车铸像。
+
+GLM:拿破仑军队攻克米兰城
+
+##### Example2 (Sentence Prediction)
+Context: 工业互联网（Industrial Internet）是新一代信息通信技术与工业经济深度融合的新型基础设施、应用模式和工业生态，通过对人、机、物、系统等的全面连接，构建起覆盖全产业链、全价值链的全新制造和服务体系，为工业乃至产业数字化、网络化、智能化发展提供了实现途径，是第四次工业革命的重要基石。[sMASK]它以网络为基础、平台为中枢、数据为要素、安全为保障，既是工业数字化、网络化、智能化转型的基础设施，也是互联网、大数据、人工智能与实体经济深度融合的应用模式，同时也是一种新业态、新产业，将重塑企业形态、供应链和产业链。当前，工业互联网融合应用向国民经济重点行业广泛拓展，形成平台化设计、智能化制造、网络化协同、个性化定制、服务化延伸、数字化管理六大新模式，赋能、赋智、赋值作用不断显现，有力的促进了实体经济提质、增效、降本、绿色、安全发展。
+
+GLM: 工业互联网是制造业技术、管理、模式的重大变革,是推动互联网、大数据、人工智能和实体经济深度融合的重要载体,是建设制造强国和网络强国的重要基础。
+
+##### Example3 (Long Text Generation)
+Context: 问题:高斯所在的国家有什么汽车品牌?答案:[gMASK]
+
+GLM:答案:[gMASK]<|startofpiece|>德国奔驰、德国大众、别克、沃尔沃、斯柯达、本田、雪铁龙.
+
+
+### Ptuning
+Run the following script to integrate p-tuning with GLM:
+```shell
+cd algutils/ptuning/ 
+bash finetune_zy.sh
+```
+
+### Pretrain
+Run the following script to pre-train the GLM-Large model
+```shell
+cd examples/glm/ 
+bash scripts/ds_pretrain_nvidia.sh config/ds_block_large.sh
+```
+
+The script [examples/glm/config/ds_pretrain_nvidia.sh](examples/glm/config/ds_pretrain_nvidia.sh) launches the training program with DeepSpeed. You should change `NUM_WORKERS` and `NUM_GPUS_PER_WORKER` to the number of workers and the number of gpus per worker. Also change `HOST_FILE_PATH` to the path to an OpenMPI-style hostfile. More details about DeepSpeed launcher can be found [here](https://www.deepspeed.ai/getting-started/#resource-configuration-multi-node).
+
+The file [examples/glm/config/ds_block_large.sh](examples/glm/config/ds_block_large.sh) defines the hyperparameters for pretraining. Most of the arguments are fairly self-explanatory. Specifically, `--train-data` can be multiple keywords defined in `NAMED_CORPORA` in [data_utils/corpora.py](data_utils/corpora.py). The hyperparameters of the optimizer are defined in the corresponding json file under `config`. The semantics of the json file can be found [here](https://www.deepspeed.ai/docs/config-json).
+
+
+
+## Bert
+
+We show some examples based on GLM model.
+
+### Pretrain
+Run the following script to pre-train the Bert model
+```shell
+cd examples/bert/ 
+python quick_start.py
+```
+
+## CogView
+### Pretrain
+Run the following script to pre-train the cogview model
+```shell
+cd examples/cogview/ 
+bash config/pretrain_multiple_nodes.sh
+```
+
+### inference
+Run the following script to test the ability of  text2image
+```shell
+cd examples/cogview/ 
+bash config/text2image_cogview.sh
+```