abxhr commited on
Commit
b923f75
1 Parent(s): c4fc0ca

Update pages

Browse files
Files changed (3) hide show
  1. backend/home.py +1 -137
  2. backend/qa.py +6 -11
  3. backend/sa.py +1 -41
backend/home.py CHANGED
@@ -7,150 +7,14 @@ def write():
7
  """
8
  # Arabic Natural Language Processing
9
 
10
- ![visitors](https://visitor-badge.glitch.me/badge?page_id=wissamantoun.arabicnlpapp)
11
-
12
-
13
- In this HuggingFace space you will be able to test the different Arabic NLP models that my colleges at [AUB MIND Lab](https://sites.aub.edu.lb/mindlab/) have built, with some other applications.
14
-
15
- Check the **Navigation bar** to access the apps:
16
- - Arabic Text Preprocessor: Test how text imput is treated by our preprocessor
17
- - Arabic Language Generation: Generate Arabic text using our AraGPT2 language models
18
- - Arabic Sentiment Analysis: Test the senitment analysis model that won the [Arabic Senitment Analysis competition @ KAUST](https://www.kaggle.com/c/arabic-sentiment-analysis-2021-kaust)
19
- - Arabic Question Answering: Test our AraELECTRA QA capabilities
20
  """
21
  )
22
  st.markdown("#")
23
- col1, col2, col3 = st.columns(3)
24
-
25
- col1.write("## **AraBERT**")
26
- col1.image("images/arabert_logo.png", width=200)
27
 
28
- col2.write("## **AraGPT2**")
29
- col2.image("images/AraGPT2.png", width=200)
30
-
31
- col3.write("## **AraElectra**")
32
- col3.image("images/AraELECTRA.png", width=200)
33
 
34
  st.markdown(
35
  """
36
 
37
- You can find the more details in the source code and paper linked in our repository on GitHub [repo](https://github.com/aub-mind/arabert).
38
-
39
- ## Dataset
40
-
41
- The pretraining data used for the new **AraBERT** model is also used for **AraGPT2 and AraELECTRA**.
42
-
43
- The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars (before applying Farasa Segmentation)
44
-
45
- Our large models were train a TPUv3-128 provided by TFRC.
46
-
47
- For the new dataset we added the unshuffled OSCAR corpus, after we thoroughly filter it, to the previous dataset used in AraBERTv1 but with out the websites that we previously crawled:
48
- - OSCAR unshuffled and filtered.
49
- - [Arabic Wikipedia dump](https://archive.org/details/arwiki-20190201) from 2020/09/01
50
- - [The 1.5B words Arabic Corpus](https://www.semanticscholar.org/paper/1.5-billion-words-Arabic-Corpus-El-Khair/f3eeef4afb81223df96575adadf808fe7fe440b4)
51
- - [The OSIAN Corpus](https://www.aclweb.org/anthology/W19-4619)
52
- - Assafir news articles. Huge thank you for Assafir for the data
53
-
54
- ## Models
55
-
56
- Model | HuggingFace Model Name | Size (MB/Params)| Pre-Segmentation | Hardware | Sequence Length | Batch Size | Num of Steps | Total Time (in Days) |
57
- ---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:
58
- AraBERTv0.2-base | [bert-base-arabertv02](https://huggingface.co/aubmindlab/bert-base-arabertv02) | 543MB / 136M | No | TPUv3-8 | 128 /512 | 2560/384 | 1M/ 2M | 36 |
59
- AraBERTv0.2-large| [bert-large-arabertv02](https://huggingface.co/aubmindlab/bert-large-arabertv02) | 1.38G / 371M | No | TPUv3-128 | 128 /512 | 13440 / 2056 | 250K / 300K | 7 |
60
- AraBERTv2-base| [bert-base-arabertv2](https://huggingface.co/aubmindlab/bert-base-arabertv2) | 543MB / 136M | Yes | TPUv3-8 |128 /512 | 2560 / 384 | 1M / 2M | 36 |
61
- AraBERTv2-large| [bert-large-arabertv2](https://huggingface.co/aubmindlab/bert-large-arabertv2) | 1.38G / 371M | Yes | TPUv3-128 |128 /512 | 13440 / 2056| 250K / 300K | 7 |
62
- AraBERTv0.1-base| [bert-base-arabertv01](https://huggingface.co/aubmindlab/bert-base-arabertv01) | 543MB / 136M | No | TPUv2-8 |128 /512 |128 / 512 | 900K / 300K| 4 |
63
- AraBERTv1-base| [bert-base-arabert](https://huggingface.co/aubmindlab/bert-base-arabert) | 543MB / 136M | Yes | TPUv2-8 |128 /512 |128 / 512 | 900K / 300K| 4 |
64
- AraGPT2-base | [aragpt2-base](https://huggingface.co/aubmindlab/aragpt2-base) | 527MB/135M | No | TPUv3-128 | 1024 | 1792 | 125K | 1.5 |
65
- AraGPT2-medium | [aragpt2-medium](https://huggingface.co/aubmindlab/aragpt2-medium) | 1.38G/370M | No |TPUv3-8 | 1024 | 80 | 1M | 15 |
66
- AraGPT2-large | [aragpt2-large](https://huggingface.co/aubmindlab/aragpt2-large) | 2.98GB/792M | No |TPUv3-128 | 1024 | 256 | 220k | 3 |
67
- AraGPT2-mega | [aragpt2-mega](https://huggingface.co/aubmindlab/aragpt2-mega) | 5.5GB/1.46B |No |TPUv3-128 | 1024 | 256 | 800K | 9 |
68
- AraELECTRA-base-generator | [araelectra-base-generator](https://huggingface.co/aubmindlab/araelectra-base-generator) | 227MB/60M | No | TPUv3-8 | 512 | 256 | 2M | 24
69
- AraELECTRA-base-discriminator | [araelectra-base-discriminator](https://huggingface.co/aubmindlab/araelectra-base-discriminator) | 516MB/135M | No | TPUv3-8 | 512 | 256 | 2M | 24
70
- AraBERTv0.2-Twitter-base| [bert-base-arabertv02-twitter](https://huggingface.co/aubmindlab/bert-base-arabertv02-twitter) | 543MB / 136M | No | V100 | *64* | - | - | - |
71
- AraBERTv0.2-Twitter-large| [bert-large-arabertv02-twitter](https://huggingface.co/aubmindlab/bert-large-arabertv02-twitter) | 1.38G / 371M | No | V100 | *64* | - | - | - |
72
-
73
- All models are available in the `HuggingFace` model page under the [aubmindlab](https://huggingface.co/aubmindlab/) name. Checkpoints are available in PyTorch, TF2 and TF1 formats.
74
-
75
- # Preprocessing
76
-
77
- You can test the Arabic Preprocessing pipeline in the Arabic Text Preprocessing page.
78
-
79
- It is recommended to apply our preprocessing function before training/testing on any dataset.
80
- **Install farasapy to segment text for AraBERT v1 & v2 `pip install farasapy`**
81
-
82
- ```python
83
- from arabert.preprocess import ArabertPreprocessor
84
-
85
- model_name = "aubmindlab/bert-base-arabertv2"
86
- arabert_prep = ArabertPreprocessor(model_name=model_name)
87
-
88
- text = "ولن نبالغ إذا قلنا: إن 'هاتف' أو 'كمبيوتر المكتب' في زمننا هذا ضروري"
89
- arabert_prep.preprocess(text)
90
- >>>"و+ لن نبالغ إذا قل +نا : إن ' هاتف ' أو ' كمبيوتر ال+ مكتب ' في زمن +نا هذا ضروري"
91
- ```
92
-
93
- You can also use the `unpreprocess()` function to reverse the preprocessing changes, by fixing the spacing around non alphabetical characters, and also de-segmenting if the model selected need pre-segmentation. We highly recommend unprocessing generated content of `AraGPT2` model, to make it look more natural.
94
- ```python
95
- output_text = "و+ لن نبالغ إذا قل +نا : إن ' هاتف ' أو ' كمبيوتر ال+ مكتب ' في زمن +نا هذا ضروري"
96
- arabert_prep.unpreprocess(output_text)
97
- >>>"ولن نبالغ إذا قلنا: إن 'هاتف' أو 'كمبيوتر المكتب' في زمننا هذا ضروري"
98
- ```
99
-
100
- # If you used this model please cite us as :
101
-
102
- ## AraBERT
103
- Google Scholar has our Bibtex wrong (missing name), use this instead
104
- ```
105
- @inproceedings{antoun2020arabert,
106
- title={AraBERT: Transformer-based Model for Arabic Language Understanding},
107
- author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
108
- booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
109
- pages={9}
110
- }
111
- ```
112
- ## AraGPT2
113
- ```
114
- @inproceedings{antoun-etal-2021-aragpt2,
115
- title = "{A}ra{GPT}2: Pre-Trained Transformer for {A}rabic Language Generation",
116
- author = "Antoun, Wissam and
117
- Baly, Fady and
118
- Hajj, Hazem",
119
- booktitle = "Proceedings of the Sixth Arabic Natural Language Processing Workshop",
120
- month = apr,
121
- year = "2021",
122
- address = "Kyiv, Ukraine (Virtual)",
123
- publisher = "Association for Computational Linguistics",
124
- url = "https://www.aclweb.org/anthology/2021.wanlp-1.21",
125
- pages = "196--207",
126
- }
127
- ```
128
-
129
- ## AraELECTRA
130
- ```
131
- @inproceedings{antoun-etal-2021-araelectra,
132
- title = "{A}ra{ELECTRA}: Pre-Training Text Discriminators for {A}rabic Language Understanding",
133
- author = "Antoun, Wissam and
134
- Baly, Fady and
135
- Hajj, Hazem",
136
- booktitle = "Proceedings of the Sixth Arabic Natural Language Processing Workshop",
137
- month = apr,
138
- year = "2021",
139
- address = "Kyiv, Ukraine (Virtual)",
140
- publisher = "Association for Computational Linguistics",
141
- url = "https://www.aclweb.org/anthology/2021.wanlp-1.20",
142
- pages = "191--195",
143
- }
144
- ```
145
-
146
-
147
- # Acknowledgments
148
- Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn't have done it without this program, and to the [AUB MIND Lab](https://sites.aub.edu.lb/mindlab/) Members for the continous support. Also thanks to [Yakshof](https://www.yakshof.com/#/) and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.
149
-
150
- # Contacts
151
- **Wissam Antoun**: [Linkedin](https://www.linkedin.com/in/wissam-antoun-622142b4/) | [Twitter](https://twitter.com/wissam_antoun) | [Github](https://github.com/WissamAntoun) | wfa07 (AT) mail (DOT) aub (DOT) edu | wissam.antoun (AT) gmail (DOT) com
152
-
153
- **Fady Baly**: [Linkedin](https://www.linkedin.com/in/fadybaly/) | [Twitter](https://twitter.com/fadybaly) | [Github](https://github.com/fadybaly) | fgb06 (AT) mail (DOT) aub (DOT) edu | baly.fady (AT) gmail (DOT) com
154
-
155
  """
156
  )
 
7
  """
8
  # Arabic Natural Language Processing
9
 
10
+ Design project for **Arabic Natural Language Processing**, by [**Abshar Mohammed Aslam**](https://github.com/abxhr).
 
 
 
 
 
 
 
 
 
11
  """
12
  )
13
  st.markdown("#")
 
 
 
 
14
 
 
 
 
 
 
15
 
16
  st.markdown(
17
  """
18
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
  """
20
  )
backend/qa.py CHANGED
@@ -8,8 +8,7 @@ def write():
8
  _, col1, _ = st.columns(3)
9
 
10
  with col1:
11
- st.image("images/is2alni_logo.png", width=200)
12
- st.title("إسألني أي شيء")
13
 
14
  st.markdown(
15
  """
@@ -22,10 +21,6 @@ def write():
22
  unsafe_allow_html=True,
23
  )
24
 
25
- st.sidebar.header("Info")
26
- st.sidebar.image("images/AraELECTRA.png", width=150)
27
- st.sidebar.write("Powered by [AraELECTRA](https://github.com/aub-mind/arabert)")
28
-
29
  st.sidebar.write("\n")
30
  n_answers = st.sidebar.slider(
31
  "Max. number of answers", min_value=1, max_value=10, value=2, step=1
@@ -35,16 +30,16 @@ def write():
35
  if "؟" not in question:
36
  question += "؟"
37
 
38
- run_query = st.button("أجب")
39
  if run_query:
40
  # https://discuss.streamlit.io/t/showing-a-gif-while-st-spinner-runs/5084
41
- with st.spinner("... جاري البحث "):
42
  results_dict = get_qa_answers(question)
43
 
44
  if len(results_dict) > 0:
45
- st.write("## :الأجابات هي")
46
  for result in results_dict["results"][:n_answers]:
47
  annotate_answer(result)
48
- f"[**المصدر**](<{result['link']}>)"
49
  else:
50
- st.write("## 😞 ليس لدي جواب")
 
8
  _, col1, _ = st.columns(3)
9
 
10
  with col1:
11
+ st.title("Ask any question!")
 
12
 
13
  st.markdown(
14
  """
 
21
  unsafe_allow_html=True,
22
  )
23
 
 
 
 
 
24
  st.sidebar.write("\n")
25
  n_answers = st.sidebar.slider(
26
  "Max. number of answers", min_value=1, max_value=10, value=2, step=1
 
30
  if "؟" not in question:
31
  question += "؟"
32
 
33
+ run_query = st.button("Find answers")
34
  if run_query:
35
  # https://discuss.streamlit.io/t/showing-a-gif-while-st-spinner-runs/5084
36
+ with st.spinner("Searching..."):
37
  results_dict = get_qa_answers(question)
38
 
39
  if len(results_dict) > 0:
40
+ st.write("## Answers:")
41
  for result in results_dict["results"][:n_answers]:
42
  annotate_answer(result)
43
+ f"[**Source**](<{result['link']}>)"
44
  else:
45
+ st.write("## 😞 No results found.")
backend/sa.py CHANGED
@@ -17,49 +17,9 @@ def write():
17
  """
18
  # Arabic Sentiment Analysis
19
 
20
- This is a simple sentiment analysis app that uses the prediction kernel from Wissam's (me) submission that won the [Arabic Senitment Analysis competition @ KAUST](https://www.kaggle.com/c/arabic-sentiment-analysis-2021-kaust)
21
  """
22
  )
23
- if st.checkbox("More info: "):
24
- st.markdown(
25
- """
26
- ### Submission Description:
27
-
28
- My submission is based on an ensemble of 5 models with varying preprocessing, and classifier design. All model variants are built over MARBERT [1], which is a BERT-based model pre-trained on 1B dialectal Arabic tweets.
29
-
30
- For preprocessing, all models shared the following steps:
31
- - Replacing user mentions with “USER” and links with “URL”
32
- - Replacing the “#” with “HASH”
33
- - Removed the underscore character since it is missing the MARBERT vocabulary.
34
- - Removed diacritics and elongations (tatweel)
35
- - Spacing out emojis
36
-
37
- For classifier design, all models use a dense layer on top of MARBERT unless otherwise specified. Model training is done by hyperparameter grid-search with 5-fold cross-validation with the following search space:
38
- - Learning rate: [2e-5,3e-5,4e-5]
39
- - Batch size: 128
40
- - Maximum sequence length: 64
41
- - Epochs: 3 (we select the best epoch for the final prediction)
42
- - Warmup ratio: [0,0.1]
43
- - Seed: [1,25,42,123,666]
44
-
45
- Model I is a vanilla variant with only the preprocessing steps mention above applied. Model II enhances the emoji representation by replacing OOV emojis with ones that have similar meaning, for example 💊  😷.
46
- We noticed the repetitive use of “السلام عليكم” and “ورحمة الله وبركاته” in neutral tweets, especially when users were directing questions to business accounts. This could confuse the classifier, if it encountered these words in a for example a negative tweet, hence in Model III we removed variation of the phrase mentioned before using fuzzy matching algorithms.
47
-
48
- In Model IV, we tried to help the model by appending a sarcasm label to the input. We first trained a separate MARBERT on the ArSarcasm [2] dataset, and then used it to label the training and test sets.
49
-
50
- Model V uses the vanilla preprocessing approach, but instead of a dense layer built on top of MARBERT, we follow the approach detailed by Safaya et.al. [3] which uses a CNN-based classifier instead.
51
-
52
- For the final prediction, we first average the predictions of the 5 models from cross-validation (this is done for each model separately), we then average the results from the 5 model variants. We observed that the distribution of the predicted sentiment classes, doesn’t quite match the true distribution, this is due to the model preferring the neutral class over the positive class. To counter that, we apply what we call Label-Weighted average where during after the final averaging we rescale the score with the following weights 1.57,0.98 and 0.93 for positive, neutral, and negative (note that the weights were determined empirically).
53
-
54
- 1- https://aclanthology.org/2021.acl-long.551/
55
-
56
- 2- https://github.com/iabufarha/ArSarcasm
57
-
58
- 3- https://github.com/alisafaya/OffensEval2020
59
-
60
-
61
- """
62
- )
63
  input_text = st.text_input(
64
  "Enter your text here:",
65
  )
 
17
  """
18
  # Arabic Sentiment Analysis
19
 
 
20
  """
21
  )
22
+
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
23
  input_text = st.text_input(
24
  "Enter your text here:",
25
  )