support chatglm2&3
State: closed | Pull request: yes
Labels: name: NLP

name: common

# What does this PR do ?

Support ChatGLM2&ChatGLM3

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

### Model conversion from huggingface to nemo
```
python ./scripts/nlp_language_modeling/convert_hf_chatglm_to_nemo.py --in-file '/mount/data/chatglm3-6b' --out-file '/mount/nemo_models/chatglm3.nemo'
```
### Inference
```
python ./examples/nlp/language_modeling/megatron_gpt_eval.py \
    --config-name=megatron_chatglm_inference \
    gpt_model_file=${CHATGLM_NEMO_MODEL} \
    trainer.devices=1 \
    trainer.num_nodes=1 \
    tensor_model_parallel_size=1 \
    pipeline_model_parallel_size=1 
```
### Pretrain
```
 python3 -u examples/nlp/language_modeling/megatron_gpt_pretraining.py \
  --config-name=megatron_chatglm_config \
  trainer.devices=8 \
  trainer.num_nodes=1 \
  trainer.val_check_interval=100 \
  trainer.precision=bf16 \
 +model.data.data_prefix=${TRAIN_DS} \
  trainer.max_steps=1000 \
  trainer.max_epochs=1 \
  model.megatron_amp_O2=True \
  model.sequence_parallel=True \
  model.tensor_model_parallel_size=2 \
  model.pipeline_model_parallel_size=1 \
  model.global_batch_size=128 \
  model.micro_batch_size=1 \
  model.override_vocab_size=65024 \
  model.use_cpu_initialization=False
```
### SFT
```
python ./examples/nlp/language_modeling/tuning/megatron_gpt_sft.py \
    trainer.devices=8 \
    trainer.val_check_interval=200 \
    model.restore_from_path=${CHATGLM_NEMO_MODEL} \
    model.tensor_model_parallel_size=${TP_SIZE} \
    model.pipeline_model_parallel_size=${PP_SIZE} \
    model.answer_only_loss=True \
    model.global_batch_size=128 \
    model.micro_batch_size=1 \
    model.data.train_ds.file_names=${TRAIN_DS} \
    model.data.train_ds.concat_sampling_probabilities=[1.0] \
    model.data.validation_ds.file_names=${VALID_DS} \
    model.data.validation_ds.global_batch_size=128\
    model.data.validation_ds.micro_batch_size=1 \
    model.megatron_amp_O2=True \
    exp_manager.exp_dir=exp \
    exp_manager.name=chatglm_test \
    trainer.precision=bf16
```
### PEFT
```
./examples/nlp/language_modeling/tuning/megatron_gpt_peft_tuning.py \
  trainer.precision=bf16 \
  trainer.devices=8 \
  trainer.num_nodes=1 \
  trainer.max_steps=200 \
  trainer.val_check_interval=20 \
  ++trainer.limit_val_batches=10 \
  trainer.gradient_clip_val=1.0 \
  model.megatron_amp_O2=True \
  ++model.mcore_gpt=True \
  model.restore_from_path=${nemo_model} \
  model.peft.peft_scheme=${scheme} \
  model.data.train_ds.file_names=["${TRAIN_DS}"] \
  model.data.train_ds.concat_sampling_probabilities=[1.0] \
  model.data.train_ds.label_key='output' \
  model.data.train_ds.num_workers=0 \
  model.data.validation_ds.file_names=["${VALID_DS}"] \
  model.data.validation_ds.label_key='output' \
  model.data.validation_ds.num_workers=0 \
  model.tensor_model_parallel_size=${TP_SIZE} \
  model.pipeline_model_parallel_size=${PP_SIZE} \
  model.global_batch_size=${GBS} \
  model.micro_batch_size=1
```
# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)