Add normalization steps
Browse files
README.md
CHANGED
|
@@ -29,5 +29,12 @@ python train_tokenizer.py --dataset_name oscar --dataset_config_name unshuffled_
|
|
| 29 |
python create_config.py --name_or_path gpt2-medium --params '{"vocab_size": 42000}'
|
| 30 |
```
|
| 31 |
|
|
|
|
| 32 |
|
|
|
|
| 33 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
python create_config.py --name_or_path gpt2-medium --params '{"vocab_size": 42000}'
|
| 30 |
```
|
| 31 |
|
| 32 |
+
### Normalization steps
|
| 33 |
|
| 34 |
+
Steps:
|
| 35 |
|
| 36 |
+
- [ ] Remove stretched words such as ســــــــــلام
|
| 37 |
+
- [ ] Remove links, user-mentioning (such as @jane_doe)
|
| 38 |
+
- [ ] Remove Telegram, Instagram advertisements, or posts (whole record)
|
| 39 |
+
- [ ] Remove advertisement records
|
| 40 |
+
- [ ] Remove separated words (or the whole record) which are showed up as an individual record, while they are just the tags at the end of the post (such as بلاب ... بلاب ... ورزشی، خبری، سیاسی، اجتماعی، خانوده)
|