ibraheemmoosa commited on
Commit
c800e9f
1 Parent(s): 83798db

Add evaluation results

Browse files
Files changed (1) hide show
  1. README.md +16 -1
README.md CHANGED
@@ -127,7 +127,22 @@ The details of the sentence order prediction example generation procedure for ea
127
  The model was pretrained on TPUv3-8 for 1M steps. We have checkpoints available every 10k steps. We will upload these in the future.
128
 
129
  ## Evaluation results
130
- We evaluated this model on the [IndicGLUE](https://huggingface.co/datasets/indic_glue) benchmark dataset.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
131
 
132
  ## Intended uses & limitations
133
 
 
127
  The model was pretrained on TPUv3-8 for 1M steps. We have checkpoints available every 10k steps. We will upload these in the future.
128
 
129
  ## Evaluation results
130
+ We evaluated this model on the Indo-Aryan subset of languages (Panjabi, Oriya, Assamese, Bangla, Hindi, Marathi, Gujarati) from the [IndicGLUE](https://huggingface.co/datasets/indic_glue) benchmark dataset. We report the mean and standard deviation of nine fine-tuning runs for this model.
131
+
132
+ ### IndicGLUE
133
+ Task | mBERT | XLM-R | IndicBERT-Base | XLMIndic-Base-Uniscript (This Model)
134
+ -----| ----- | ----- | ------ | -------
135
+ Wikipedia Section Title Prediction| 71.90 | 65.45 | 69.40 | **81.78 ± 0.60**
136
+ Article Genre Classification | 88.64 | 96.61 | 97.72 | **98.70 ± 0.29**
137
+ Named Entity Recognition (F1-score) | 71.29 | 62.18 | 56.69 | **89.85 ± 1.14**
138
+ BBC Hindi News Article Classification | 60.55 | 75.52 | 74.60 | **77.28 ± 1.50**
139
+ Soham Bangla News Article Classification | 80.23 | 87.6 | 78.45 | **93.22 ± 0.49**
140
+ INLTK Gujarati Headlines | - | - | **92.91** | 90.41 ± 0.69
141
+ INLTK Marathi Headlines | - | - | **94.30** | 92.21 ± 0.23
142
+ IITP Hindi Product Reviews Sentiment Classification | 74.57 | **78.97** | 71.32 | 76.33 ± 0.84
143
+ IITP Hindi Movie Reviews Sentiment Classification | 56.77 | 61.61 | 59.03 | **65.91 ± 2.20**
144
+ MIDAS Hindi Discourse Classification | 71.20 | **79.94** | 78.44 | 78.39 ± 0.33
145
+ Cloze Style Question Answering | - | - | 37.16 | **41.54**
146
 
147
  ## Intended uses & limitations
148