---
license: apache-2.0
language: tr
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_8_0
- generated_from_trainer
- robust-speech-event
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xls-r-300m-tr
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice tr
      type: common_voice_8_0
      args: tr
    metrics:
       - name: Test WER
         type: wer
         value: 28.69
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-large-xls-r-300m-tr

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the MOZILLA-FOUNDATION/COMMON_VOICE_8_0 - TR dataset.
It achieves the following results on the evaluation set:
- Loss: 0.2224
- Wer: 0.2869

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 100.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 6.8222        | 0.64  | 500   | 3.5026          | 1.0    |
| 3.2136        | 1.28  | 1000  | 3.0593          | 1.0000 |
| 2.8882        | 1.91  | 1500  | 2.4670          | 0.9939 |
| 2.3743        | 2.55  | 2000  | 1.1844          | 0.8657 |
| 1.9456        | 3.19  | 2500  | 0.8228          | 0.7397 |
| 1.7781        | 3.83  | 3000  | 0.6826          | 0.6753 |
| 1.6848        | 4.46  | 3500  | 0.5885          | 0.6140 |
| 1.6228        | 5.1   | 4000  | 0.5274          | 0.5789 |
| 1.5768        | 5.74  | 4500  | 0.4900          | 0.5519 |
| 1.5431        | 6.38  | 5000  | 0.4508          | 0.5238 |
| 1.5019        | 7.02  | 5500  | 0.4248          | 0.5021 |
| 1.4684        | 7.65  | 6000  | 0.4009          | 0.4827 |
| 1.4635        | 8.29  | 6500  | 0.3830          | 0.4700 |
| 1.4291        | 8.93  | 7000  | 0.3707          | 0.4595 |
| 1.4271        | 9.57  | 7500  | 0.3570          | 0.4514 |
| 1.3938        | 10.2  | 8000  | 0.3479          | 0.4378 |
| 1.3914        | 10.84 | 8500  | 0.3396          | 0.4368 |
| 1.3767        | 11.48 | 9000  | 0.3253          | 0.4262 |
| 1.3641        | 12.12 | 9500  | 0.3251          | 0.4178 |
| 1.355         | 12.76 | 10000 | 0.3138          | 0.4136 |
| 1.336         | 13.39 | 10500 | 0.3121          | 0.4069 |
| 1.3292        | 14.03 | 11000 | 0.3041          | 0.4014 |
| 1.3249        | 14.67 | 11500 | 0.3014          | 0.3931 |
| 1.3156        | 15.31 | 12000 | 0.3014          | 0.3929 |
| 1.313         | 15.94 | 12500 | 0.2969          | 0.3968 |
| 1.3068        | 16.58 | 13000 | 0.2965          | 0.3966 |
| 1.2785        | 17.22 | 13500 | 0.2943          | 0.3850 |
| 1.2867        | 17.86 | 14000 | 0.2912          | 0.3782 |
| 1.2714        | 18.49 | 14500 | 0.2819          | 0.3747 |
| 1.2844        | 19.13 | 15000 | 0.2840          | 0.3740 |
| 1.2684        | 19.77 | 15500 | 0.2913          | 0.3828 |
| 1.26          | 20.41 | 16000 | 0.2739          | 0.3674 |
| 1.2543        | 21.05 | 16500 | 0.2740          | 0.3691 |
| 1.2532        | 21.68 | 17000 | 0.2709          | 0.3756 |
| 1.2409        | 22.32 | 17500 | 0.2669          | 0.3593 |
| 1.2404        | 22.96 | 18000 | 0.2673          | 0.3576 |
| 1.2347        | 23.6  | 18500 | 0.2678          | 0.3643 |
| 1.2351        | 24.23 | 19000 | 0.2715          | 0.3650 |
| 1.2409        | 24.87 | 19500 | 0.2637          | 0.3571 |
| 1.2152        | 25.51 | 20000 | 0.2785          | 0.3609 |
| 1.2046        | 26.15 | 20500 | 0.2610          | 0.3508 |
| 1.2082        | 26.79 | 21000 | 0.2619          | 0.3461 |
| 1.2109        | 27.42 | 21500 | 0.2597          | 0.3502 |
| 1.2014        | 28.06 | 22000 | 0.2608          | 0.3468 |
| 1.1948        | 28.7  | 22500 | 0.2573          | 0.3457 |
| 1.205         | 29.34 | 23000 | 0.2619          | 0.3464 |
| 1.2019        | 29.97 | 23500 | 0.2559          | 0.3474 |
| 1.1917        | 30.61 | 24000 | 0.2601          | 0.3462 |
| 1.1939        | 31.25 | 24500 | 0.2575          | 0.3387 |
| 1.1882        | 31.89 | 25000 | 0.2535          | 0.3368 |
| 1.191         | 32.53 | 25500 | 0.2489          | 0.3365 |
| 1.1767        | 33.16 | 26000 | 0.2501          | 0.3347 |
| 1.167         | 33.8  | 26500 | 0.2504          | 0.3347 |
| 1.1678        | 34.44 | 27000 | 0.2480          | 0.3378 |
| 1.1803        | 35.08 | 27500 | 0.2487          | 0.3345 |
| 1.167         | 35.71 | 28000 | 0.2442          | 0.3319 |
| 1.1661        | 36.35 | 28500 | 0.2495          | 0.3334 |
| 1.164         | 36.99 | 29000 | 0.2472          | 0.3292 |
| 1.1578        | 37.63 | 29500 | 0.2442          | 0.3242 |
| 1.1584        | 38.27 | 30000 | 0.2431          | 0.3314 |
| 1.1526        | 38.9  | 30500 | 0.2441          | 0.3347 |
| 1.1542        | 39.54 | 31000 | 0.2437          | 0.3330 |
| 1.1508        | 40.18 | 31500 | 0.2433          | 0.3294 |
| 1.1406        | 40.82 | 32000 | 0.2434          | 0.3271 |
| 1.1514        | 41.45 | 32500 | 0.2426          | 0.3255 |
| 1.1418        | 42.09 | 33000 | 0.2432          | 0.3233 |
| 1.1365        | 42.73 | 33500 | 0.2436          | 0.3240 |
| 1.1348        | 43.37 | 34000 | 0.2483          | 0.3257 |
| 1.1301        | 44.01 | 34500 | 0.2420          | 0.3271 |
| 1.1268        | 44.64 | 35000 | 0.2472          | 0.3225 |
| 1.1224        | 45.28 | 35500 | 0.2382          | 0.3205 |
| 1.1224        | 45.92 | 36000 | 0.2388          | 0.3184 |
| 1.1198        | 46.56 | 36500 | 0.2382          | 0.3202 |
| 1.1274        | 47.19 | 37000 | 0.2404          | 0.3172 |
| 1.1147        | 47.83 | 37500 | 0.2394          | 0.3164 |
| 1.121         | 48.47 | 38000 | 0.2406          | 0.3202 |
| 1.1109        | 49.11 | 38500 | 0.2384          | 0.3154 |
| 1.1164        | 49.74 | 39000 | 0.2375          | 0.3169 |
| 1.1105        | 50.38 | 39500 | 0.2387          | 0.3173 |
| 1.1054        | 51.02 | 40000 | 0.2362          | 0.3120 |
| 1.0893        | 51.66 | 40500 | 0.2399          | 0.3130 |
| 1.0913        | 52.3  | 41000 | 0.2357          | 0.3088 |
| 1.1017        | 52.93 | 41500 | 0.2345          | 0.3084 |
| 1.0937        | 53.57 | 42000 | 0.2330          | 0.3140 |
| 1.0945        | 54.21 | 42500 | 0.2399          | 0.3107 |
| 1.0933        | 54.85 | 43000 | 0.2383          | 0.3134 |
| 1.0912        | 55.48 | 43500 | 0.2372          | 0.3077 |
| 1.0898        | 56.12 | 44000 | 0.2339          | 0.3083 |
| 1.0903        | 56.76 | 44500 | 0.2367          | 0.3065 |
| 1.0947        | 57.4  | 45000 | 0.2352          | 0.3104 |
| 1.0751        | 58.04 | 45500 | 0.2334          | 0.3084 |
| 1.09          | 58.67 | 46000 | 0.2328          | 0.3100 |
| 1.0876        | 59.31 | 46500 | 0.2276          | 0.3050 |
| 1.076         | 59.95 | 47000 | 0.2309          | 0.3047 |
| 1.086         | 60.59 | 47500 | 0.2293          | 0.3047 |
| 1.082         | 61.22 | 48000 | 0.2328          | 0.3027 |
| 1.0714        | 61.86 | 48500 | 0.2290          | 0.3020 |
| 1.0746        | 62.5  | 49000 | 0.2313          | 0.3059 |
| 1.076         | 63.14 | 49500 | 0.2342          | 0.3050 |
| 1.0648        | 63.78 | 50000 | 0.2286          | 0.3025 |
| 1.0586        | 64.41 | 50500 | 0.2338          | 0.3044 |
| 1.0753        | 65.05 | 51000 | 0.2308          | 0.3045 |
| 1.0664        | 65.69 | 51500 | 0.2273          | 0.3009 |
| 1.0739        | 66.33 | 52000 | 0.2298          | 0.3027 |
| 1.0695        | 66.96 | 52500 | 0.2247          | 0.2996 |
| 1.06          | 67.6  | 53000 | 0.2276          | 0.3015 |
| 1.0742        | 68.24 | 53500 | 0.2280          | 0.2974 |
| 1.0618        | 68.88 | 54000 | 0.2291          | 0.2989 |
| 1.062         | 69.52 | 54500 | 0.2302          | 0.2971 |
| 1.0572        | 70.15 | 55000 | 0.2280          | 0.2990 |
| 1.055         | 70.79 | 55500 | 0.2278          | 0.2983 |
| 1.0553        | 71.43 | 56000 | 0.2282          | 0.2991 |
| 1.0509        | 72.07 | 56500 | 0.2261          | 0.2959 |
| 1.0469        | 72.7  | 57000 | 0.2216          | 0.2919 |
| 1.0476        | 73.34 | 57500 | 0.2267          | 0.2989 |
| 1.0494        | 73.98 | 58000 | 0.2260          | 0.2960 |
| 1.0517        | 74.62 | 58500 | 0.2297          | 0.2989 |
| 1.0458        | 75.26 | 59000 | 0.2246          | 0.2923 |
| 1.0382        | 75.89 | 59500 | 0.2255          | 0.2922 |
| 1.0462        | 76.53 | 60000 | 0.2258          | 0.2954 |
| 1.0375        | 77.17 | 60500 | 0.2251          | 0.2929 |
| 1.0332        | 77.81 | 61000 | 0.2277          | 0.2940 |
| 1.0423        | 78.44 | 61500 | 0.2243          | 0.2896 |
| 1.0379        | 79.08 | 62000 | 0.2274          | 0.2928 |
| 1.0398        | 79.72 | 62500 | 0.2237          | 0.2928 |
| 1.0395        | 80.36 | 63000 | 0.2265          | 0.2956 |
| 1.0397        | 80.99 | 63500 | 0.2240          | 0.2920 |
| 1.0262        | 81.63 | 64000 | 0.2244          | 0.2934 |
| 1.0335        | 82.27 | 64500 | 0.2265          | 0.2936 |
| 1.0385        | 82.91 | 65000 | 0.2238          | 0.2928 |
| 1.0289        | 83.55 | 65500 | 0.2219          | 0.2912 |
| 1.0372        | 84.18 | 66000 | 0.2236          | 0.2898 |
| 1.0279        | 84.82 | 66500 | 0.2219          | 0.2902 |
| 1.0325        | 85.46 | 67000 | 0.2240          | 0.2908 |
| 1.0202        | 86.1  | 67500 | 0.2206          | 0.2886 |
| 1.0166        | 86.73 | 68000 | 0.2219          | 0.2886 |
| 1.0259        | 87.37 | 68500 | 0.2235          | 0.2897 |
| 1.0337        | 88.01 | 69000 | 0.2210          | 0.2873 |
| 1.0264        | 88.65 | 69500 | 0.2216          | 0.2882 |
| 1.0231        | 89.29 | 70000 | 0.2223          | 0.2899 |
| 1.0281        | 89.92 | 70500 | 0.2214          | 0.2872 |
| 1.0135        | 90.56 | 71000 | 0.2218          | 0.2868 |
| 1.0291        | 91.2  | 71500 | 0.2209          | 0.2863 |
| 1.0321        | 91.84 | 72000 | 0.2199          | 0.2876 |
| 1.028         | 92.47 | 72500 | 0.2214          | 0.2858 |
| 1.0213        | 93.11 | 73000 | 0.2219          | 0.2875 |
| 1.0261        | 93.75 | 73500 | 0.2232          | 0.2869 |
| 1.0197        | 94.39 | 74000 | 0.2227          | 0.2866 |
| 1.0298        | 95.03 | 74500 | 0.2228          | 0.2868 |
| 1.0192        | 95.66 | 75000 | 0.2230          | 0.2865 |
| 1.0156        | 96.3  | 75500 | 0.2220          | 0.2869 |
| 1.0075        | 96.94 | 76000 | 0.2223          | 0.2866 |
| 1.0201        | 97.58 | 76500 | 0.2219          | 0.2866 |
| 1.0159        | 98.21 | 77000 | 0.2219          | 0.2876 |
| 1.0087        | 98.85 | 77500 | 0.2219          | 0.2873 |
| 1.0159        | 99.49 | 78000 | 0.2223          | 0.2867 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


---
license: apache-2.0
language:
- ru
tags:
- generated_from_trainer
- robust-speech-event
datasets:
- common_voice
model-index:
- name: wav2vec2-xls-r-300m-Russian-small
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice ru
      type: common_voice
      args: ru
    metrics:
       - name: Test WER
         type: wer
         value: 48.38
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-xls-r-300m-Russian-small

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:
- Loss: 0.3514
- Wer: 0.4838

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 10
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 5.512         | 1.32  | 400  | 3.2207          | 1.0    |
| 3.1562        | 2.65  | 800  | 3.0166          | 1.0    |
| 1.5211        | 3.97  | 1200 | 0.7134          | 0.8275 |
| 0.6724        | 5.3   | 1600 | 0.4713          | 0.6402 |
| 0.4693        | 6.62  | 2000 | 0.3904          | 0.5668 |
| 0.3693        | 7.95  | 2400 | 0.3609          | 0.5121 |
| 0.3004        | 9.27  | 2800 | 0.3514          | 0.4838 |

### Framework versions

- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.14.0
- Tokenizers 0.10.3


---
license: apache-2.0
language: ab
tags:
- generated_from_trainer
- robust-speech-event
datasets:
- common_voice
model-index:
- name: wav2vec2-xls-r-300m-ab-CV8
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice ab
      type: common_voice
      args: ab
    metrics:
       - name: Test WER
         type: wer
         value: 54.74
---

# wav2vec2-xls-r-300m-ab-CV8

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:
- Loss: 0.2105
- Wer: 0.5474

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 300
- num_epochs: 15
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 4.7729        | 0.63  | 500   | 3.0624          | 1.0021 |
| 2.7348        | 1.26  | 1000  | 1.0460          | 0.9815 |
| 1.2756        | 1.9   | 1500  | 0.4618          | 0.8309 |
| 1.0419        | 2.53  | 2000  | 0.3725          | 0.7449 |
| 0.9491        | 3.16  | 2500  | 0.3368          | 0.7345 |
| 0.9006        | 3.79  | 3000  | 0.3014          | 0.6936 |
| 0.8519        | 4.42  | 3500  | 0.2852          | 0.6767 |
| 0.8243        | 5.06  | 4000  | 0.2701          | 0.6504 |
| 0.7902        | 5.69  | 4500  | 0.2641          | 0.6221 |
| 0.7767        | 6.32  | 5000  | 0.2549          | 0.6192 |
| 0.7516        | 6.95  | 5500  | 0.2515          | 0.6179 |
| 0.737         | 7.59  | 6000  | 0.2408          | 0.5963 |
| 0.7217        | 8.22  | 6500  | 0.2429          | 0.6261 |
| 0.7101        | 8.85  | 7000  | 0.2366          | 0.5687 |
| 0.6922        | 9.48  | 7500  | 0.2277          | 0.5680 |
| 0.6866        | 10.11 | 8000  | 0.2242          | 0.5847 |
| 0.6703        | 10.75 | 8500  | 0.2222          | 0.5803 |
| 0.6649        | 11.38 | 9000  | 0.2247          | 0.5765 |
| 0.6513        | 12.01 | 9500  | 0.2182          | 0.5644 |
| 0.6369        | 12.64 | 10000 | 0.2128          | 0.5508 |
| 0.6425        | 13.27 | 10500 | 0.2132          | 0.5514 |
| 0.6399        | 13.91 | 11000 | 0.2116          | 0.5495 |
| 0.6208        | 14.54 | 11500 | 0.2105          | 0.5474 |

### Framework versions

- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.18.1
- Tokenizers 0.10.3


---
language: Guj
datasets:
- google
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: XLSR Wav2Vec2 Guj by Jaimin
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Google
      type: voice
      args: guj
    metrics:
       - name: Test WER
         type: wer
         value: 28.92
---

# wav2vec2-base-gujarati-demo

Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) in Guj
When using this model, make sure that your speech input is sampled at 16kHz.

## Usage

The model can be used directly (without a language model) as follows:

```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

common_voice_train,common_voice_test = load_dataset('csv', data_files={'train': 'train.csv','test': 'test.csv'},error_bad_lines=False,encoding='utf-8',split=['train', 'test']).

processor = Wav2Vec2Processor.from_pretrained("jaimin/wav2vec2-base-gujarati-demo")
model = Wav2Vec2ForCTC.from_pretrained("jaimin/wav2vec2-base-gujarati-demo")

resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch

test_dataset = common_voice_test.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

with torch.no_grad():
     logits = model(inputs.input_values).logits

predicted_ids = torch.argmax(logits, dim=-1)

print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][0].lower())
```

## Evaluation

The model can be evaluated as follows on the {language} test data of Common Voice.

```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re

common_voice_validation = load_dataset('csv', data_files={'test': 'validation.csv'},error_bad_lines=False,encoding='utf-8',split='test')
wer = load_metric("wer")

processor = Wav2Vec2Processor.from_pretrained("jaimin/wav2vec2-base-gujarati-demo")
model = Wav2Vec2ForCTC.from_pretrained("Amrrs/wav2vec2-base-gujarati-demo")
model.to("cuda")

chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch

test_dataset = common_voice_validation.map(speech_file_to_array_fn)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16000, return_tensors="pt", padding=True)

    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda")).logits

    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch

result = common_voice_validation.map(evaluate, batched=True, batch_size=8)

print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```

**Test Result**: 28.92 %

## Training

The Google datasets were used for training.

The script used for training can be found [here](https://colab.research.google.com/drive/1-Klkgr4f-C9SanHfVC5RhP0ELUH6TYlN?usp=sharing)


tokenizer = AutoTokenizer.from_pretrained("jaimin/Gujarati-Model")
model = AutoModel.from_pretrained("jaimin/Gujarati-Model")


---
license: mit
tags:
- generated_from_trainer
datasets:
- xtreme
metrics:
- f1
model-index:
- name: xlm-roberta-base-finetuned-panx-de
  results:
  - task:
      name: Token Classification
      type: token-classification
    dataset:
      name: xtreme
      type: xtreme
      args: PAN-X.de
    metrics:
    - name: F1
      type: f1
      value: 0.8657141230716685
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# xlm-roberta-base-finetuned-panx-de

This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the xtreme dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1604
- F1: 0.8657

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss | F1     |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 0.2648        | 1.0   | 1573 | 0.1923          | 0.8244 |
| 0.1398        | 2.0   | 3146 | 0.1573          | 0.8543 |
| 0.0851        | 3.0   | 4719 | 0.1604          | 0.8657 |

### Framework versions

- Transformers 4.15.0
- Pytorch 1.10.1
- Datasets 1.18.1
- Tokenizers 0.10.3


---
tags:
- ner
- token-classification
- japanese
- bert

language:
- ja

license: mit

---
## bert-japanese-ner
このモデルは日本語の固有表現抽出タスクを目的として、[京都大学 黒橋・褚・村脇研究室が公開しているBERT日本語Pretrainedモデル](https://nlp.ist.i.kyoto-u.ac.jp/?ku_bert_japanese)をベースに[ストックマーク株式会社が公開しているner-wikipedia-dataset](https://github.com/stockmarkteam/ner-wikipedia-dataset)でファインチューニングしたものです。

## How to use
このモデルはTokenizerに上述の京都大学BERT日本語PretrainedモデルのTokenizerを利用します。
当リポジトリにTokenizerは含まれていません。
利用する際は別途ダウンロードしてご用意ください。

また、Tokenizerとは別に[Juman++](https://nlp.ist.i.kyoto-u.ac.jp/?JUMAN%2B%2B)と[pyknp](https://nlp.ist.i.kyoto-u.ac.jp/?PyKNP)を利用します。
予めインストールしておいてください。
```py
from transformers import (
    BertForTokenClassification, BertTokenizer
)

from pyknp import Juman

jumanpp = Juman()
tokenizer = BertTokenizer.from_pretrained("ダウンロードした京都大学のTokenizerのファイルパス")

model = BertForTokenClassification.from_pretrained("ken11/bert-japanese-ner")

text = "なにか文章"
juman_result = jumanpp.analysis(text)
tokenized_text = [mrph.midasi for mrph in juman_result.mrph_list()]
inputs = tokenizer(tokenized_text, return_tensors="pt", padding='max_length', truncation=True, max_length=64, is_split_into_words=True)
pred = model(**inputs).logits[0]
pred = np.argmax(pred.detach().numpy(), axis=-1)
labels = []
for i, label in enumerate(pred):
    if i + 1 > len(tokenized_text):
        continue
    labels.append(model.config.id2label[label])
    print(f"{tokenized_text[i]}: {model.config.id2label[label]}")
print(tokenized_text)
print(labels)
```

## Training Data
学習には[ストックマーク株式会社が公開しているner-wikipedia-dataset](https://github.com/stockmarkteam/ner-wikipedia-dataset)を利用しました。
便利なデータセットを公開していただきありがとうございます。

## Note
固有表現抽出のラベルは学習データセットのものをBILUO形式に変換して使用しています。
ラベルの詳細については[ner-wikipedia-datasetの概要](https://github.com/stockmarkteam/ner-wikipedia-dataset#%E6%A6%82%E8%A6%81)をご確認ください。

## Licenese
[The MIT license](https://opensource.org/licenses/MIT)


---
tags:
- translation
- japanese

language:
- ja
- en

license: mit

widget:
- text: "今日もご安全に"

---
## mbart-ja-en
このモデルは[facebook/mbart-large-cc25](https://huggingface.co/facebook/mbart-large-cc25)をベースに[JESC dataset](https://nlp.stanford.edu/projects/jesc/index_ja.html)でファインチューニングしたものです。
This model is based on [facebook/mbart-large-cc25](https://huggingface.co/facebook/mbart-large-cc25) and fine-tuned with [JESC dataset](https://nlp.stanford.edu/projects/jesc/index_ja.html).

## How to use
```py
from transformers import (
    MBartForConditionalGeneration, MBartTokenizer
)

tokenizer = MBartTokenizer.from_pretrained("ken11/mbart-ja-en")
model = MBartForConditionalGeneration.from_pretrained("ken11/mbart-ja-en")

inputs = tokenizer("こんにちは", return_tensors="pt")
translated_tokens = model.generate(**inputs, decoder_start_token_id=tokenizer.lang_code_to_id["en_XX"], early_stopping=True, max_length=48)
pred = tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
print(pred)
```

## Training Data
I used the [JESC dataset](https://nlp.stanford.edu/projects/jesc/index_ja.html) for training.
Thank you for publishing such a large dataset.

## Tokenizer
The tokenizer uses the [sentencepiece](https://github.com/google/sentencepiece) trained on the JESC dataset.

## Note
The result of evaluating the sacrebleu score for [JEC Basic Sentence Data of Kyoto University](https://nlp.ist.i.kyoto-u.ac.jp/EN/?JEC+Basic+Sentence+Data#i0163896) was `18.18` .

## Licenese
[The MIT license](https://opensource.org/licenses/MIT)


---
tags:
- fill-mask
- japanese
- albert

language:
- ja

license: mit

widget:
- text: "2022年の[MASK]概要"

---
## albert-base-japanese-v1
日本語事前学習済みALBERTモデルです

## How to use
### ファインチューニング
このモデルはPreTrainedモデルです
基本的には各種タスク用にファインチューニングして使用されることを想定しています

### Fill-Mask
このモデルではTokenizerにSentencepieceを利用しています
そのままでは`[MASK]`トークンのあとに[余計なトークンが混入する問題](https://ken11.jp/blog/sentencepiece-tokenizer-bug)があるので、利用する際には以下のようにする必要があります
#### for PyTorch
```py
from transformers import (
    AlbertForMaskedLM, AlbertTokenizerFast
)
import torch

tokenizer = AlbertTokenizerFast.from_pretrained("ken11/albert-base-japanese-v1")
model = AlbertForMaskedLM.from_pretrained("ken11/albert-base-japanese-v1")

text = "大学で[MASK]の研究をしています"
tokenized_text = tokenizer.tokenize(text)
del tokenized_text[tokenized_text.index(tokenizer.mask_token) + 1]

input_ids = [tokenizer.cls_token_id]
input_ids.extend(tokenizer.convert_tokens_to_ids(tokenized_text))
input_ids.append(tokenizer.sep_token_id)

inputs = {"input_ids": [input_ids], "token_type_ids": [[0]*len(input_ids)], "attention_mask": [[1]*len(input_ids)]}
batch = {k: torch.tensor(v, dtype=torch.int64) for k, v in inputs.items()}
output = model(**batch)[0]
_, result = output[0, input_ids.index(tokenizer.mask_token_id)].topk(5)

print(tokenizer.convert_ids_to_tokens(result.tolist()))
# ['英語', '心理学', '数学', '医学', '日本語']
```

#### for TensorFlow
```py
from transformers import (
    TFAlbertForMaskedLM, AlbertTokenizerFast
)
import tensorflow as tf

tokenizer = AlbertTokenizerFast.from_pretrained("ken11/albert-base-japanese-v1")
model = TFAlbertForMaskedLM.from_pretrained("ken11/albert-base-japanese-v1")

text = "大学で[MASK]の研究をしています"
tokenized_text = tokenizer.tokenize(text)
del tokenized_text[tokenized_text.index(tokenizer.mask_token) + 1]

input_ids = [tokenizer.cls_token_id]
input_ids.extend(tokenizer.convert_tokens_to_ids(tokenized_text))
input_ids.append(tokenizer.sep_token_id)

inputs = {"input_ids": [input_ids], "token_type_ids": [[0]*len(input_ids)], "attention_mask": [[1]*len(input_ids)]}
batch = {k: tf.convert_to_tensor(v, dtype=tf.int32) for k, v in inputs.items()}
output = model(**batch)[0]
result = tf.math.top_k(output[0, input_ids.index(tokenizer.mask_token_id)], k=5)

print(tokenizer.convert_ids_to_tokens(result.indices.numpy()))
# ['英語', '心理学', '数学', '医学', '日本語']
```

## Training Data
学習には
- [日本語Wikipediaの全文](https://ja.wikipedia.org/wiki/Wikipedia:%E3%83%87%E3%83%BC%E3%82%BF%E3%83%99%E3%83%BC%E3%82%B9%E3%83%80%E3%82%A6%E3%83%B3%E3%83%AD%E3%83%BC%E3%83%89)
- [livedoorニュースコーパス](https://www.rondhuit.com/download.html#ldcc)

を利用しています

## Tokenizer
トークナイザーは[Sentencepiece](https://github.com/google/sentencepiece)を利用しています
こちらも学習データは同様です

## Licenese
[The MIT license](https://opensource.org/licenses/MIT)


---
tags:
- generated_from_trainer
datasets:
- ade_corpus_v2
model-index:
- name: sagemaker-BioclinicalBERT-ADR
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# sagemaker-BioclinicalBERT-ADR

This model is a fine-tuned version of [emilyalsentzer/Bio_ClinicalBERT](https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT) on the ade_corpus_v2 dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 32
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 1
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 171  | 0.9441          |

### Framework versions

- Transformers 4.12.3
- Pytorch 1.9.1
- Datasets 1.15.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- emotion
metrics:
- accuracy
model-index:
- name: sagemaker-distilbert-emotion
  results:
  - task:
      name: Text Classification
      type: text-classification
    dataset:
      name: emotion
      type: emotion
      args: default
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.9165
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# sagemaker-distilbert-emotion

This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the emotion dataset.
It achieves the following results on the evaluation set:
- Loss: 0.2434
- Accuracy: 0.9165

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 32
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 1
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 0.9423        | 1.0   | 500  | 0.2434          | 0.9165   |

### Framework versions

- Transformers 4.12.3
- Pytorch 1.9.1
- Datasets 1.15.1
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- glue
metrics:
- matthews_correlation
model-index:
- name: bert-base-uncased-8-10-2e-05
  results:
  - task:
      name: Text Classification
      type: text-classification
    dataset:
      name: glue
      type: glue
      args: cola
    metrics:
    - name: Matthews Correlation
      type: matthews_correlation
      value: 0.8411910241920598
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bert-base-uncased-8-10-2e-05

This model is a fine-tuned version of [bert-base-uncased](https://huggingface.co/bert-base-uncased) on the glue dataset.
It achieves the following results on the evaluation set:
- Loss: 0.3003
- Matthews Correlation: 0.8412

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10

### Training results

| Training Loss | Epoch | Step | Validation Loss | Matthews Correlation |
|:-------------:|:-----:|:----:|:---------------:|:--------------------:|
| No log        | 1.0   | 100  | 0.3470          | 0.6290               |
| No log        | 2.0   | 200  | 0.3379          | 0.7349               |
| No log        | 3.0   | 300  | 0.2487          | 0.8069               |
| No log        | 4.0   | 400  | 0.2720          | 0.8069               |
| 0.1403        | 5.0   | 500  | 0.3003          | 0.8412               |
| 0.1403        | 6.0   | 600  | 0.3365          | 0.8069               |
| 0.1403        | 7.0   | 700  | 0.3521          | 0.8069               |
| 0.1403        | 8.0   | 800  | 0.3473          | 0.8069               |
| 0.1403        | 9.0   | 900  | 0.3561          | 0.8069               |
| 0.0064        | 10.0  | 1000 | 0.3538          | 0.8069               |

### Framework versions

- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.3
- Tokenizers 0.11.0


# A Multi-task learning model with two prediction heads
* One prediction head classifies between keyword sentences vs statements/questions
* Other prediction head corresponds to classifier for statements vs questions

## Scores
##### Spaadia SQuaD Test acc: **0.9891**
##### Quora Keyword Pairs Test acc: **0.98048**

## Datasets:
Quora Keyword Pairs: https://www.kaggle.com/stefanondisponibile/quora-question-keyword-pairs
Spaadia SQuaD pairs: https://www.kaggle.com/shahrukhkhan/questions-vs-statementsclassificationdataset

## Article
[Medium article](https://medium.com/@shahrukhx01/multi-task-learning-with-transformers-part-1-multi-prediction-heads-b7001cf014bf)
## Demo Notebook
[Colab Notebook Multi-task Query classifiers](https://colab.research.google.com/drive/1R7WcLHxDsVvZXPhr5HBgIWa3BlSZKY6p?usp=sharing)
## Clone the model repo
```bash
git clone https://huggingface.co/shahrukhx01/bert-multitask-query-classifiers
```
```python
%cd bert-multitask-query-classifiers/
```
## Load model
```python
from multitask_model import BertForSequenceClassification
from transformers import AutoTokenizer
import torch
model = BertForSequenceClassification.from_pretrained(
        "shahrukhx01/bert-multitask-query-classifiers",
        task_labels_map={"quora_keyword_pairs": 2, "spaadia_squad_pairs": 2},
    )
tokenizer = AutoTokenizer.from_pretrained("shahrukhx01/bert-multitask-query-classifiers")
```
## Run inference on both Tasks
```python
from multitask_model import BertForSequenceClassification
from transformers import AutoTokenizer
import torch
model = BertForSequenceClassification.from_pretrained(
        "shahrukhx01/bert-multitask-query-classifiers",
        task_labels_map={"quora_keyword_pairs": 2, "spaadia_squad_pairs": 2},
    )
tokenizer = AutoTokenizer.from_pretrained("shahrukhx01/bert-multitask-query-classifiers")

## Keyword vs Statement/Question Classifier
input = ["keyword query", "is this a keyword query?"]
task_name="quora_keyword_pairs"
sequence = tokenizer(input, padding=True, return_tensors="pt")['input_ids']
logits = model(sequence, task_name=task_name)[0]
predictions = torch.argmax(torch.softmax(logits, dim=1).detach().cpu(), axis=1)
for input, prediction in zip(input, predictions):
  print(f"task: {task_name}, input: {input} \n prediction=> {prediction}")
  print()

## Statement vs Question Classifier
input = ["where is berlin?", "is this a keyword query?", "Berlin is in Germany."]
task_name="spaadia_squad_pairs"
sequence = tokenizer(input, padding=True, return_tensors="pt")['input_ids']
logits = model(sequence, task_name=task_name)[0]
predictions = torch.argmax(torch.softmax(logits, dim=1).detach().cpu(), axis=1)
for input, prediction in zip(input, predictions):
  print(f"task: {task_name}, input: {input} \n prediction=> {prediction}")
  print()
```


---
tags:
- fuzzy-matching
- fuzzy-search
- entity-resolution
- record-linking
- structured-data-search
---
A Siamese BERT architecture trained at character levels tokens for embedding based Fuzzy matching.

## Usage (Sentence-Transformers)
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
```
pip install -U sentence-transformers
```
Then you can use the model like this:
```python
from sentence_transformers import SentenceTransformer, util
word1 = "fuzzformer"
word1 = " ".join([char for char in word1]) ## divide the word to char level to fuzzy match
word2 = "fizzformer"
word2 = " ".join([char for char in word2]) ## divide the word to char level to fuzzy match
words = [word1, word2]

model = SentenceTransformer('shahrukhx01/paraphrase-mpnet-base-v2-fuzzy-matcher')
fuzzy_embeddings = model.encode(words)

print("Fuzzy Match score:")
print(util.cos_sim(fuzzy_embeddings[0], fuzzy_embeddings[1]))
```
## Usage (HuggingFace Transformers)
```python
import torch
from transformers import AutoTokenizer, AutoModel
from torch import Tensor, device

def cos_sim(a: Tensor, b: Tensor):
    """
    borrowed from sentence transformers repo
    Computes the cosine similarity cos_sim(a[i], b[j]) for all i and j.
    :return: Matrix with res[i][j]  = cos_sim(a[i], b[j])
    """
    if not isinstance(a, torch.Tensor):
        a = torch.tensor(a)

    if not isinstance(b, torch.Tensor):
        b = torch.tensor(b)

    if len(a.shape) == 1:
        a = a.unsqueeze(0)

    if len(b.shape) == 1:
        b = b.unsqueeze(0)

    a_norm = torch.nn.functional.normalize(a, p=2, dim=1)
    b_norm = torch.nn.functional.normalize(b, p=2, dim=1)
    return torch.mm(a_norm, b_norm.transpose(0, 1))

#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

# Words we want fuzzy embeddings for
word1 = "fuzzformer"
word1 = " ".join([char for char in word1]) ## divide the word to char level to fuzzy match
word2 = "fizzformer"
word2 = " ".join([char for char in word2]) ## divide the word to char level to fuzzy match
words = [word1, word2]
# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('shahrukhx01/paraphrase-mpnet-base-v2-fuzzy-matcher')
model = AutoModel.from_pretrained('shahrukhx01/paraphrase-mpnet-base-v2-fuzzy-matcher')

# Tokenize sentences
encoded_input = tokenizer(words, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, max pooling.
fuzzy_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Fuzzy Match score:")
print(cos_sim(fuzzy_embeddings[0], fuzzy_embeddings[1]))
```

## ACKNOWLEDGEMENT
A big thank you to [Sentence Transformers](https://github.com/UKPLab/sentence-transformers) as their implementation really expedited the implementation of Fuzzformer.
## Citation
To cite FuzzTransformer in your work, please use the following bibtex reference:
@misc{shahrukhkhan2021fuzzTransformer,

  author       = {Shahrukh Khan},

  title        = {FuzzTransformer: A character level embedding based Siamese transformer for fuzzy string matching.},

  year         = 2021,

  publisher    = {Coming soon},

  doi          = {Coming soon},

  url          = {Coming soon}

}


---
tags:
- wikisql
- text2sql
---
```python
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
model = BartForConditionalGeneration.from_pretrained('shahrukhx01/schema-aware-distilbart-cnn-12-6-text2sql')
tokenizer = BartTokenizer.from_pretrained('shahrukhx01/schema-aware-distilbart-cnn-12-6-text2sql')
## add NL query with table schema
question = "What is terrence ross' nationality? </s> <col0> Player : text <col1> No. : text <col2> Nationality : text <col3> Position : text <col4> Years in Toronto : text <col5>  School/Club Team : text"

inputs = tokenizer([question], max_length=1024, return_tensors='pt')

# Generate SQL
text_query_ids = model.generate(inputs['input_ids'], num_beams=4, min_length=0, max_length=125, early_stopping=True)
prediction = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in text_query_ids][0]
print(prediction)
```


---
language: "en"
tags:
- neural-search-query-classification
- neural-search
widget:
- text: "what did you eat in lunch?"
---
# KEYWORD STATEMENT VS QUESTION CLASSIFIER FOR NEURAL SEARCH

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("shahrukhx01/question-vs-statement-classifier")

model = AutoModelForSequenceClassification.from_pretrained("shahrukhx01/question-vs-statement-classifier")
```
Trained to add the feature for classifying queries between Question Query vs Statement Query using classification in [Haystack](https://github.com/deepset-ai/haystack/issues/611)


## Multiple Prediction Heads
* ExtractiveQA Head
* Three Class Classification Head, classes => (yes, no, extra_qa) to answer binary questions or direct to ExtractiveQA Head

## BoolQ Validation dataset Evaluation:

support => 3270

accuracy => 0.73

macro f1 => 0.71

## SQuAD Validation dataset Evaluation:

eval_HasAns_exact      = 78.0196

eval_HasAns_f1         = 84.0327

eval_HasAns_total      =    5928

eval_NoAns_exact       = 81.8167

eval_NoAns_f1          = 81.8167

eval_NoAns_total       =    5945

eval_best_exact        = 79.9208

eval_best_f1           = 82.9231

eval_exact             = 79.9208

eval_f1                = 82.9231

eval_samples           =   12165

eval_total             =   11873

## Uasge in transformers
Import the script from [here](https://huggingface.co/shahrukhx01/roberta-base-squad2-boolq-baseline/blob/main/multitask_model.py)
```python
from multitask_model import RobertaForMultitaskQA
from transformers import RobertaTokenizerFast
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = RobertaForMultitaskQA.from_pretrained(
        "shahrukhx01/roberta-base-squad2-boolq-baseline",
        task_labels_map={"squad_v2": 2, "boolq": 3},
    ).to(device)
tokenizer = RobertaTokenizerFast.from_pretrained("shahrukhx01/roberta-base-squad2-boolq-baseline")
```


```python
from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
)

model = AutoModelForSequenceClassification.from_pretrained("shahrukhx01/roberta-base-boolq")
model.to(device)
#model.push_to_hub("roberta-base-boolq")

tokenizer = AutoTokenizer.from_pretrained("shahrukhx01/roberta-base-boolq")

def predict(question, passage):
  sequence = tokenizer.encode_plus(question, passage, return_tensors="pt")['input_ids'].to(device)

  logits = model(sequence)[0]
  probabilities = torch.softmax(logits, dim=1).detach().cpu().tolist()[0]
  proba_yes = round(probabilities[1], 2)
  proba_no = round(probabilities[0], 2)

  print(f"Question: {question}, Yes: {proba_yes}, No: {proba_no}")

passage = """Berlin is the capital and largest city of Germany by both area and population. Its 3.8 million inhabitants make it the European Union's most populous city,
                        according to population within city limits."""

question = "Is Berlin the smallest city of Germany?"
predict(s_question, passage)
```


---
language: "en"
tags:
- neural-search-query-classification
- neural-search
widget:
- text: "keyword query."
---
# KEYWORD QUERY VS STATEMENT/QUESTION CLASSIFIER FOR NEURAL SEARCH
| Train Loss    | Validation Acc.| Test Acc.|
| ------------- |:-------------: | -----:   |
| 0.000806      | 0.99  | 0.997    |

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("shahrukhx01/bert-mini-finetune-question-detection")

model = AutoModelForSequenceClassification.from_pretrained("shahrukhx01/bert-mini-finetune-question-detection")
```
Trained to add feature for classifying queries between Keyword Query or Question + Statement Query using classification in [Haystack](https://github.com/deepset-ai/haystack/issues/611)

Problem Statement:
One common challenge that we saw in deployments: We need to distinguish between real questions and keyword queries that come in. We only want to route questions to the Reader branch in order to maximize the accuracy of results and minimize computation efforts/costs.

Baseline:
https://www.kaggle.com/shahrukhkhan/question-v-statement-detection

Dataset:
https://www.kaggle.com/stefanondisponibile/quora-question-keyword-pairs

Kaggle Notebook:
https://www.kaggle.com/shahrukhkhan/question-vs-statement-classification-mini-bert/


---
language: "en"
tags:
- schema-aware-text2sql
- text2sql
- wikisql
widget:
- text: "What is terrence ross' nationality? </s> <col0> Player : text <col1> No. : text <col2> Nationality : text <col3> Position : text <col4> Years in Toronto : text <col5>  School/Club Team : text"
---
```python
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
model = BartForConditionalGeneration.from_pretrained('shahrukhx01/schema-aware-denoising-bart-large-cnn-text2sql')
tokenizer = BartTokenizer.from_pretrained('shahrukhx01/schema-aware-denoising-bart-large-cnn-text2sql')
## add NL query with table schema
question = "What is terrence ross' nationality? </s> <col0> Player : text <col1> No. : text <col2> Nationality : text <col3> Position : text <col4> Years in Toronto : text <col5>  School/Club Team : text"

inputs = tokenizer([question], max_length=1024, return_tensors='pt')

# Generate SQL
text_query_ids = model.generate(inputs['input_ids'], num_beams=4, min_length=0, max_length=125, early_stopping=True)
prediction = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in text_query_ids][0]
print(prediction)
```


The distilbart-cnn-12-6-text2sql is fine-tuned on WIKISQL dataset.
```python
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig

model = BartForConditionalGeneration.from_pretrained('shahrukhx01/distilbart-cnn-12-6-text2sql')
tokenizer = BartTokenizer.from_pretrained('shahrukhx01/distilbart-cnn-12-6-text2sql')

TEXT_QUERY = "what is the temperature of berlin "
inputs = tokenizer([TEXT_QUERY], max_length=1024, return_tensors='pt')

# Generate SQL
text_query_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=5, early_stopping=True)
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in text_query_ids])
```


---
language: de
license: mit
thumbnail: "https://raw.githubusercontent.com/German-NLP-Group/german-transformer-training/master/model_cards/german-electra-logo.png"
tags:
- electra
- commoncrawl
- uncased
- umlaute
- umlauts
- german
- deutsch
---

# German Electra Uncased

[¹]

## Version 2 Release
We released an improved version of this model. Version 1 was trained for 766,000 steps. For this new version we continued the training for an additional 734,000 steps. It therefore follows that version 2 was trained on a total of 1,500,000 steps. See "Evaluation of Version 2: GermEval18 Coarse" below for details.

## Model Info
This Model is suitable for training on many downstream tasks in German (Q&A, Sentiment Analysis, etc.).

It can be used as a drop-in replacement for **BERT** in most down-stream tasks (**ELECTRA** is even implemented as an extended **BERT** Class).

At the time of release (August 2020) this model is the best performing publicly available German NLP model on various German evaluation metrics (CONLL03-DE, GermEval18 Coarse, GermEval18 Fine). For GermEval18 Coarse results see below. More will be published soon.

## Installation
This model has the special feature that it is **uncased** but does **not strip accents**.
This possibility was added by us with [PR #6280](https://github.com/huggingface/transformers/pull/6280).
To use it you have to use Transformers version 3.1.0 or newer.

```bash
pip install transformers -U
```

## Uncase and Umlauts ('Ö', 'Ä', 'Ü')
This model is uncased. This helps especially for domains where colloquial terms with uncorrect capitalization is often used.

The special characters 'ö', 'ü', 'ä' are included through the `strip_accent=False` option, as this leads to an improved precision.

## Creators
This model was trained and open sourced in conjunction with the [**German NLP Group**](https://github.com/German-NLP-Group) in equal parts by:
- [**Philip May**](https://May.la) - [T-Systems on site services GmbH](https://www.t-systems-onsite.de/)
- [**Philipp Reißel**](https://www.reissel.eu) - [ambeRoad](https://amberoad.de/)

## Evaluation of Version 2: GermEval18 Coarse
We evaluated all language models on GermEval18 with the F1 macro score. For each model we did an extensive automated hyperparameter search. With the best hyperparmeters we did fit the moodel multiple times on GermEval18. This is done to cancel random effects and get results of statistical relevance.

![GermEval18 Coarse Model Evaluation for Version 2](https://raw.githubusercontent.com/German-NLP-Group/german-transformer-training/master/model_cards/model-eval-v2.png)

## Checkpoint evaluation
Since it it not guaranteed that the last checkpoint is the best, we evaluated the checkpoints on GermEval18. We found that the last checkpoint is indeed the best. The training was stable and did not overfit the text corpus.

## Pre-training details

### Data
- Cleaned Common Crawl Corpus 2019-09 German: [CC_net](https://github.com/facebookresearch/cc_net) (Only head coprus and filtered for language_score > 0.98) - 62 GB
- German Wikipedia Article Pages Dump (20200701) - 5.5 GB
- German Wikipedia Talk Pages Dump (20200620) - 1.1 GB
- Subtitles - 823 MB
- News 2018 - 4.1 GB

The sentences were split with [SojaMo](https://github.com/tsproisl/SoMaJo). We took the German Wikipedia Article Pages Dump 3x to oversample. This approach was also used in a similar way in GPT-3 (Table 2.2).

More Details can be found here [Preperaing Datasets for German Electra Github](https://github.com/German-NLP-Group/german-transformer-training)

### Electra Branch no_strip_accents
Because we do not want to stip accents in our training data we made a change to Electra and used this repo [Electra no_strip_accents](https://github.com/PhilipMay/electra/tree/no_strip_accents) (branch `no_strip_accents`). Then created the tf dataset with:

```bash
python build_pretraining_dataset.py --corpus-dir <corpus_dir> --vocab-file <dir>/vocab.txt --output-dir ./tf_data --max-seq-length 512 --num-processes 8 --do-lower-case --no-strip-accents
```

### The training
The training itself can be performed with the Original Electra Repo (No special case for this needed).
We run it with the following Config:

The exact Training Config

debug False

disallow_correct False

disc_weight 50.0

do_eval False

do_lower_case True

do_train True

electra_objective True

embedding_size 768

eval_batch_size 128

gcp_project None

gen_weight 1.0

generator_hidden_size 0.33333

generator_layers 1.0

iterations_per_loop 200

keep_checkpoint_max 0

learning_rate 0.0002

lr_decay_power 1.0

mask_prob 0.15

max_predictions_per_seq 79

max_seq_length 512

model_dir gs://XXX

model_hparam_overrides {}

model_name 02_Electra_Checkpoints_32k_766k_Combined

model_size base

num_eval_steps 100

num_tpu_cores 8

num_train_steps 766000

num_warmup_steps 10000

pretrain_tfrecords gs://XXX

results_pkl gs://XXX

results_txt gs://XXX

save_checkpoints_steps 5000

temperature 1.0

tpu_job_name None

tpu_name electrav5

tpu_zone None

train_batch_size 256

uniform_generator False

untied_generator True

untied_generator_embeddings False

use_tpu True

vocab_file gs://XXX

vocab_size 32767

weight_decay_rate 0.01

![Training Loss](https://raw.githubusercontent.com/German-NLP-Group/german-transformer-training/master/model_cards/loss.png)

Please Note: *Due to the GAN like strucutre of Electra the loss is not that meaningful*

It took about 7 Days on a preemtible TPU V3-8. In total, the Model went through approximately 10 Epochs. For an automatically recreation of a cancelled TPUs we used [tpunicorn](https://github.com/shawwn/tpunicorn). The total cost of training summed up to about 450 $ for one run. The Data-pre processing and Vocab Creation needed approximately 500-1000 CPU hours. Servers were fully provided by [T-Systems on site services GmbH](https://www.t-systems-onsite.de/), [ambeRoad](https://amberoad.de/).
Special thanks to [Stefan Schweter](https://github.com/stefan-it) for your feedback and providing parts of the text corpus.

[¹]: Source for the picture [Pinterest](https://www.pinterest.cl/pin/371828512984142193/)

### Negative Results
We tried the following approaches which we found had no positive influence:

-  **Increased Vocab Size**: Leads to more parameters and thus reduced examples/sec while no visible Performance gains were measured
-  **Decreased Batch-Size**: The original Electra was trained with a Batch Size per TPU Core of 16 whereas this Model was trained with 32 BS / TPU Core. We found out that 32 BS leads to better results when you compare metrics over computation time

## License - The MIT License
Copyright 2020-2021 Philip May

Copyright 2020-2021 Philipp Reissel

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: distilbert-base-multilingual-cased-finetuned-squad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert-base-multilingual-cased-finetuned-squad

This model is a fine-tuned version of [distilbert-base-multilingual-cased](https://huggingface.co/distilbert-base-multilingual-cased) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.6587

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.923         | 1.0   | 579  | 0.8439          |
| 0.8479        | 2.0   | 1158 | 0.6784          |
| 0.6148        | 3.0   | 1737 | 0.6587          |

### Framework versions

- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: distilbert-base-multilingual-cased-finetuned-viquad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert-base-multilingual-cased-finetuned-viquad

This model is a fine-tuned version of [distilbert-base-multilingual-cased](https://huggingface.co/distilbert-base-multilingual-cased) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 3.4241

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 65   | 4.0975          |
| No log        | 2.0   | 130  | 3.9315          |
| No log        | 3.0   | 195  | 3.6742          |
| No log        | 4.0   | 260  | 3.4878          |
| No log        | 5.0   | 325  | 3.4241          |

### Framework versions

- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: bert-base-multilingual-cased-finetuned-viquad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bert-base-multilingual-cased-finetuned-viquad

This model is a fine-tuned version of [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 1.9815

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 65   | 2.5534          |
| No log        | 2.0   | 130  | 2.1165          |
| No log        | 3.0   | 195  | 1.9815          |

### Framework versions

- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


---
license: mit
tags:
- generated_from_trainer
model-index:
- name: xlm-roberta-base-finetuned-viquad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# xlm-roberta-base-finetuned-viquad

This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 2.3761

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 259  | 2.9945          |
| 3.3665        | 2.0   | 518  | 2.3761          |

### Framework versions

- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


---
license: mit
tags:
- generated_from_trainer
model-index:
- name: xlm-roberta-base-finetuned-squad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# xlm-roberta-base-finetuned-squad

This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5539

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.7665        | 1.0   | 2295 | 0.5231          |
| 0.5236        | 2.0   | 4590 | 0.5539          |

### Framework versions

- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: bert-base-multilingual-cased-finetuned-squad
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bert-base-multilingual-cased-finetuned-squad

This model is a fine-tuned version of [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4919

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.1782        | 1.0   | 579  | 0.5258          |
| 0.4938        | 2.0   | 1158 | 0.4639          |
| 0.32          | 3.0   | 1737 | 0.4919          |

### Framework versions

- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: distilbert-base-uncased-finetuned-squad
  results:
  - task:
      name: Question Answering
      type: question-answering
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert-base-uncased-finetuned-squad

This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on an unknown dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Framework versions

- Transformers 4.10.2
- Pytorch 1.9.0+cu102
- Tokenizers 0.10.3


---
language: ti
widget:
- text: "ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር"
datasets:
- TLMD
- NTC
metrics:
- f1
- precision
- recall
- accuracy
model-index:
- name: tiroberta-base-pos
  results:
  - task:
      name: Token Classification
      type: token-classification
    metrics:
    - name: F1
      type: f1
      value: 0.9562
    - name: Precision
      type: precision
      value: 0.9562
    - name: Recall
      type: recall
      value: 0.9562
    - name: Accuracy
      type: accuracy
      value: 0.9562
---

# Tigrinya POS tagging with TiRoBERTa

This model is a fine-tuned version of [TiRoBERTa](https://huggingface.co/fgaim/tiroberta) on the NTC-v1 dataset (Tedla et al. 2016).

## Training

### Hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10.0

### Results

The model achieves the following results on the test set:
- Loss: 0.3194
- Adj Precision: 0.9219
- Adj Recall: 0.9335
- Adj F1: 0.9277
- Adj Number: 1670
- Adv Precision: 0.8297
- Adv Recall: 0.8554
- Adv F1: 0.8423
- Adv Number: 484
- Con Precision: 0.9844
- Con Recall: 0.9763
- Con F1: 0.9804
- Con Number: 972
- Fw Precision: 0.7895
- Fw Recall: 0.5357
- Fw F1: 0.6383
- Fw Number: 28
- Int Precision: 0.6552
- Int Recall: 0.7308
- Int F1: 0.6909
- Int Number: 26
- N Precision: 0.9650
- N Recall: 0.9662
- N F1: 0.9656
- N Number: 3992
- Num Precision: 0.9747
- Num Recall: 0.9665
- Num F1: 0.9706
- Num Number: 239
- N Prp Precision: 0.9308
- N Prp Recall: 0.9447
- N Prp F1: 0.9377
- N Prp Number: 470
- N V Precision: 0.9854
- N V Recall: 0.9736
- N V F1: 0.9794
- N V Number: 416
- Pre Precision: 0.9722
- Pre Recall: 0.9625
- Pre F1: 0.9673
- Pre Number: 907
- Pro Precision: 0.9448
- Pro Recall: 0.9236
- Pro F1: 0.9341
- Pro Number: 445
- Pun Precision: 1.0
- Pun Recall: 0.9994
- Pun F1: 0.9997
- Pun Number: 1607
- Unc Precision: 1.0
- Unc Recall: 0.875
- Unc F1: 0.9333
- Unc Number: 16
- V Precision: 0.8780
- V Recall: 0.9231
- V F1: 0.9
- V Number: 78
- V Aux Precision: 0.9685
- V Aux Recall: 0.9878
- V Aux F1: 0.9780
- V Aux Number: 654
- V Ger Precision: 0.9388
- V Ger Recall: 0.9571
- V Ger F1: 0.9479
- V Ger Number: 513
- V Imf Precision: 0.9634
- V Imf Recall: 0.9497
- V Imf F1: 0.9565
- V Imf Number: 914
- V Imv Precision: 0.8793
- V Imv Recall: 0.7286
- V Imv F1: 0.7969
- V Imv Number: 70
- V Prf Precision: 0.8960
- V Prf Recall: 0.9082
- V Prf F1: 0.9020
- V Prf Number: 294
- V Rel Precision: 0.9678
- V Rel Recall: 0.9538
- V Rel F1: 0.9607
- V Rel Number: 757
- Overall Precision: 0.9562
- Overall Recall: 0.9562
- Overall F1: 0.9562
- Overall Accuracy: 0.9562

### Framework versions

- Transformers 4.12.0.dev0
- Pytorch 1.9.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3

## Citation

If you use this model in your product or research, please cite as follows:

```
@article{Fitsum2021TiPLMs,
  author={Fitsum Gaim and Wonsuk Yang and Jong C. Park},
  title={Monolingual Pre-trained Language Models for Tigrinya},
  year=2021,
  publisher={WiNLP 2021/EMNLP 2021}
}
```

## References

```
Tedla, Y., Yamamoto, K. & Marasinghe, A. 2016.
Tigrinya Part-of-Speech Tagging with Morphological Patterns and the New Nagaoka Tigrinya Corpus.
International Journal Of Computer Applications 146 pp. 33-41 (2016).
```


---
language:
- en
datasets:
- c4
- squad
tags:
- text2text-generation
widget:
- text: "question: What is the atomic number for oxygen? context: Oxygen is a chemical element with symbol O and atomic number 8."
- text: "question: What is the chemical symbol of Oxygen? context: Oxygen is a chemical element with symbol O and atomic number 8."

license: apache-2.0
---

T5-small for QA
---

[Google's T5-small](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) pre-trained on the [C4](https://huggingface.co/datasets/c4) dataset, fine-tuned for Question-Answering on [SQuAD v2](https://huggingface.co/datasets/squad_v2) with the following hyperparameters:

```
  optimizer=adamw_hf
  learning_rate=3e-5
  adam_beta1=0.9
  adam_beta2=0.999
  adam_epsilon=1e-08
  num_train_epochs=2
  per_device_train_batch_size=12
```

Usage
---

The input [context and question] has to be prepared in a specific way as follows:

```python
from transformers import pipeline

def prep_input(_context, _question):
    return " ".join(["question:", _question.strip(), "context:", _context.strip()])

t5qa = pipeline("text2text-generation", "fgaim/t5-small-squad-v2")

context = """
Oxygen is a chemical element with symbol O and atomic number 8. It is a member of the chalcogen group on the periodic table and is a highly reactive nonmetal and oxidizing agent that readily forms compounds (notably oxides) with most elements. By mass, oxygen is the third-most abundant element in the universe, after hydrogen and helium. At standard temperature and pressure, two atoms of the element bind to form dioxygen, a colorless and odorless diatomic gas with the formula O.
"""

t5qa(prep_input(context, "How many atoms combine to form dioxygen?"))
# [{'generated_text': 'two'}]

t5qa(prep_input(context, "What element makes up almost half of the earth's crust by mass?"))
# [{'generated_text': 'oxygen'}]

t5qa(prep_input(context, "What are the most abundent elements of the universe by mass?"))
# [{'generated_text': 'hydrogen and helium'}]
```


---
language: ti
widget:
- text: "ዓቕሚ መንእሰይ ኤርትራ [MASK] ተራእዩ"
---

# Pre-trained ELECTRA small for Tigrinya Language

We pre-train ELECTRA small on the [TLMD](https://zenodo.org/record/5139094) dataset, with over 40 million tokens.

Contained are trained Flax and PyTorch models.

## Hyperparameters

The hyperparameters corresponding to model sizes mentioned above are as follows:

| Model Size | L  | AH | HS  | FFN  | P    | Seq  |
|------------|----|----|-----|------|------|------|
| SMALL      | 12 | 4  | 256 | 1024 | 14M  | 512  |

(L = number of layers; AH = number of attention heads; HS = hidden size; FFN = feedforward network dimension; P = number of parameters; Seq = maximum sequence length.)


---
language: ti
widget:
- text: "ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር"
metrics:
- f1
- precision
- recall
- accuracy
model-index:
- name: tielectra-small-sentiment
  results:
  - task:
      name: Text Classification
      type: text-classification
    metrics:
    - name: F1
      type: f1
      value: 0.8228962818003914
    - name: Precision
      type: precision
      value: 0.8055555555555556
    - name: Recall
      type: recall
      value: 0.841
    - name: Accuracy
      type: accuracy
      value: 0.819
---

# Sentiment Analysis for Tigrinya with TiELECTRA small

This model is a fine-tuned version of [TiELECTRA small](https://huggingface.co/fgaim/tielectra-small) on a YouTube comments Sentiment Analysis dataset for Tigrinya (Tela et al. 2020).

## Basic usage

```python
from transformers import pipeline

ti_sent = pipeline("sentiment-analysis", model="fgaim/tielectra-small-sentiment")
ti_sent("ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር")
```

## Training

### Hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Results

The model achieves the following results on the evaluation set:
- F1: 0.8229
- Precision: 0.8056
- Recall: 0.841
- Accuracy: 0.819
- Loss: 0.4299

### Framework versions

- Transformers 4.10.3
- Pytorch 1.9.0+cu111
- Datasets 1.10.2
- Tokenizers 0.10.1

## Citation

If you use this model in your product or research, please cite as follows:

```
@article{Fitsum2021TiPLMs,
  author={Fitsum Gaim and Wonsuk Yang and Jong C. Park},
  title={Monolingual Pre-trained Language Models for Tigrinya},
  year=2021,
  publisher= {WiNLP 2021/EMNLP 2021}
}
```

## References

```
Tela, A., Woubie, A. and Hautamäki, V. 2020.
Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinya.
ArXiv, abs/2006.07698.
```


---
language: ti
widget:
- text: "ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር"
datasets:
- TLMD
metrics:
- accuracy
- f1
- precision
- recall
model-index:
- name: tiroberta-sentiment
  results:
  - task:
      name: Text Classification
      type: text-classification
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.828
    - name: F1
      type: f1
      value: 0.8476527900797165
    - name: Precision
      type: precision
      value: 0.760731319554849
    - name: Recall
      type: recall
      value: 0.957
---

# Sentiment Analysis for Tigrinya with TiRoBERTa

This model is a fine-tuned version of [TiRoBERTa](https://huggingface.co/fgaim/roberta-base-tigrinya) on a YouTube comments Sentiment Analysis dataset for Tigrinya (Tela et al. 2020).

## Basic usage

```python
from transformers import pipeline

ti_sent = pipeline("sentiment-analysis", model="fgaim/tiroberta-sentiment")
ti_sent("ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር")
```

## Training

### Hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Results

It achieves the following results on the evaluation set:
- F1: 0.8477
- Precision: 0.7607
- Recall: 0.957
- Accuracy: 0.828
- Loss: 0.6796

### Framework versions

- Transformers 4.10.3
- Pytorch 1.9.0+cu111
- Datasets 1.10.2
- Tokenizers 0.10.1

## Citation

If you use this model in your product or research, please cite as follows:

```
@article{Fitsum2021TiPLMs,
  author={Fitsum Gaim and Wonsuk Yang and Jong C. Park},
  title={Monolingual Pre-trained Language Models for Tigrinya},
  year=2021,
  publisher={WiNLP 2021/EMNLP 2021}
}
```

## References

```
Tela, A., Woubie, A. and Hautamäki, V. 2020.
Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinya.
ArXiv, abs/2006.07698.
```


---
language: ti
widget:
- text: "ዓቕሚ መንእሰይ ኤርትራ <mask> ተራእዩ"
---

# RoBERTa Pretrained for Tigrinya Language

We pretrain a RoBERTa base model for Tigrinya on a dataset of 40 million tokens trained for 40 epochs.

Contained in this repo is the original pretrained Flax model that was trained on a TPU v3.8 and it's corresponding PyTorch version.

## Hyperparameters

The hyperparameters corresponding to model sizes mentioned above are as follows:

| Model Size | L  | AH | HS  | FFN  | P    | Seq  |
|------------|----|----|-----|------|------|------|
| BASE       | 12 | 12 | 768 | 3072 | 125M | 512  |

(L = number of layers; AH = number of attention heads; HS = hidden size; FFN = feedforward network dimension; P = number of parameters; Seq = maximum sequence length.)


---
language: ti
widget:
- text: "ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር"
datasets:
- TLMD
- NTC
metrics:
- f1
- precision
- recall
- accuracy
model-index:
- name: tielectra-small-pos
  results:
  - task:
      name: Token Classification
      type: token-classification
    metrics:
    - name: F1
      type: f1
      value: 0.9456
    - name: Precision
      type: precision
      value: 0.9456
    - name: Recall
      type: recall
      value: 0.9456
    - name: Accuracy
      type: accuracy
      value: 0.9456
---

# Tigrinya POS tagging with TiELECTRA

This model is a fine-tuned version of [TiELECTRA](https://huggingface.co/fgaim/tielectra-small) on the NTC-v1 dataset (Tedla et al. 2016).

## Basic usage

```python
from transformers import pipeline

ti_pos = pipeline("token-classification", model="fgaim/tielectra-small-pos")
ti_pos("ድምጻዊ ኣብርሃም ኣፈወርቂ ንዘልኣለም ህያው ኮይኑ ኣብ ልብና ይነብር")
```

## Training

### Hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10.0

### Results

The model achieves the following results on the test set:
- Loss: 0.2236
- Adj Precision: 0.9148
- Adj Recall: 0.9192
- Adj F1: 0.9170
- Adj Number: 1670
- Adv Precision: 0.8228
- Adv Recall: 0.8058
- Adv F1: 0.8142
- Adv Number: 484
- Con Precision: 0.9793
- Con Recall: 0.9743
- Con F1: 0.9768
- Con Number: 972
- Fw Precision: 0.5
- Fw Recall: 0.3214
- Fw F1: 0.3913
- Fw Number: 28
- Int Precision: 0.64
- Int Recall: 0.6154
- Int F1: 0.6275
- Int Number: 26
- N Precision: 0.9525
- N Recall: 0.9587
- N F1: 0.9556
- N Number: 3992
- Num Precision: 0.9825
- Num Recall: 0.9372
- Num F1: 0.9593
- Num Number: 239
- N Prp Precision: 0.9132
- N Prp Recall: 0.9404
- N Prp F1: 0.9266
- N Prp Number: 470
- N V Precision: 0.9667
- N V Recall: 0.9760
- N V F1: 0.9713
- N V Number: 416
- Pre Precision: 0.9645
- Pre Recall: 0.9592
- Pre F1: 0.9619
- Pre Number: 907
- Pro Precision: 0.9395
- Pro Recall: 0.9079
- Pro F1: 0.9234
- Pro Number: 445
- Pun Precision: 1.0
- Pun Recall: 0.9994
- Pun F1: 0.9997
- Pun Number: 1607
- Unc Precision: 0.9286
- Unc Recall: 0.8125
- Unc F1: 0.8667
- Unc Number: 16
- V Precision: 0.7609
- V Recall: 0.8974
- V F1: 0.8235
- V Number: 78
- V Aux Precision: 0.9581
- V Aux Recall: 0.9786
- V Aux F1: 0.9682
- V Aux Number: 654
- V Ger Precision: 0.9183
- V Ger Recall: 0.9415
- V Ger F1: 0.9297
- V Ger Number: 513
- V Imf Precision: 0.9473
- V Imf Recall: 0.9442
- V Imf F1: 0.9458
- V Imf Number: 914
- V Imv Precision: 0.8163
- V Imv Recall: 0.5714
- V Imv F1: 0.6723
- V Imv Number: 70
- V Prf Precision: 0.8927
- V Prf Recall: 0.8776
- V Prf F1: 0.8851
- V Prf Number: 294
- V Rel Precision: 0.9535
- V Rel Recall: 0.9485
- V Rel F1: 0.9510
- V Rel Number: 757
- Overall Precision: 0.9456
- Overall Recall: 0.9456
- Overall F1: 0.9456
- Overall Accuracy: 0.9456

### Framework versions

- Transformers 4.10.3
- Pytorch 1.9.0+cu111
- Datasets 1.10.2
- Tokenizers 0.10.1

## Citation

If you use this model in your product or research, please cite as follows:

```
@article{Fitsum2021TiPLMs,
  author= {Fitsum Gaim and Wonsuk Yang and Jong C. Park},
  title= {Monolingual Pre-trained Language Models for Tigrinya},
  year= 2021,
  publisher= {WiNLP 2021/EMNLP 2021}
}
```

## References

```
Tedla, Y., Yamamoto, K. & Marasinghe, A. 2016.
Tigrinya Part-of-Speech Tagging with Morphological Patterns and the New Nagaoka Tigrinya Corpus.
International Journal Of Computer Applications 146 pp. 33-41 (2016).
```


---
license: mit
tags:
- generated_from_trainer
datasets:
- amazon_reviews_multi
model-index:
- name: xlm-roberta-base-finetuned-marc-en
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# xlm-roberta-base-finetuned-marc-en

This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the amazon_reviews_multi dataset.
It achieves the following results on the evaluation set:
- Loss: 0.9211
- Mae: 0.5122

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2

### Training results

| Training Loss | Epoch | Step | Validation Loss | Mae    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 1.1436        | 1.0   | 235  | 1.0181          | 0.5366 |
| 0.9756        | 2.0   | 470  | 0.9211          | 0.5122 |

### Framework versions

- Transformers 4.12.3
- Pytorch 1.9.0+cu111
- Datasets 1.15.1
- Tokenizers 0.10.3