README.md

---
language:
- de
- en
- es
- fr
- it
- nl
- pl
- pt
- ru
- zh
license: cc-by-nc-sa-4.0
tags:
- generated_from_trainer
- ner
- named-entity-recognition
- span-marker
datasets:
- Babelscape/multinerd
metrics:
- precision
- recall
- f1
pipeline_tag: token-classification
widget:
- text: Amelia Earthart flog mit ihrer einmotorigen Lockheed Vega 5B über den Atlantik
    nach Paris.
  example_title: German
- text: Amelia Earhart flew her single engine Lockheed Vega 5B across the Atlantic
    to Paris.
  example_title: English
- text: Amelia Earthart voló su Lockheed Vega 5B monomotor a través del Océano Atlántico
    hasta París.
  example_title: Spanish
- text: Amelia Earthart a fait voler son monomoteur Lockheed Vega 5B à travers l'ocean
    Atlantique jusqu'à Paris.
  example_title: French
- text: Amelia Earhart ha volato con il suo monomotore Lockheed Vega 5B attraverso
    l'Atlantico fino a Parigi.
  example_title: Italian
- text: Amelia Earthart vloog met haar één-motorige Lockheed Vega 5B over de Atlantische
    Oceaan naar Parijs.
  example_title: Dutch
- text: Amelia Earthart przeleciała swoim jednosilnikowym samolotem Lockheed Vega
    5B przez Ocean Atlantycki do Paryża.
  example_title: Polish
- text: Amelia Earhart voou em seu monomotor Lockheed Vega 5B através do Atlântico
    para Paris.
  example_title: Portuguese
- text: Амелия Эртхарт перелетела на своем одномоторном самолете Lockheed Vega 5B
    через Атлантический океан в Париж.
  example_title: Russian
- text: Amelia Earthart flaug eins hreyfils Lockheed Vega 5B yfir Atlantshafið til
    Parísar.
  example_title: Icelandic
- text: Η Amelia Earthart πέταξε το μονοκινητήριο Lockheed Vega 5B της πέρα ​​από
    τον Ατλαντικό Ωκεανό στο Παρίσι.
  example_title: Greek
- text: Amelia Earhartová přeletěla se svým jednomotorovým Lockheed Vega 5B přes Atlantik
    do Paříže.
  example_title: Czech
- text: Amelia Earhart lensi yksimoottorisella Lockheed Vega 5B:llä Atlantin yli Pariisiin.
  example_title: Finnish
- text: Amelia Earhart fløj med sin enmotoriske Lockheed Vega 5B over Atlanten til
    Paris.
  example_title: Danish
- text: Amelia Earhart flög sin enmotoriga Lockheed Vega 5B över Atlanten till Paris.
  example_title: Swedish
- text: Amelia Earhart fløy sin enmotoriske Lockheed Vega 5B over Atlanterhavet til
    Paris.
  example_title: Norwegian
- text: Amelia Earhart și-a zburat cu un singur motor Lockheed Vega 5B peste Atlantic
    până la Paris.
  example_title: Romanian
- text: Amelia Earhart menerbangkan mesin tunggal Lockheed Vega 5B melintasi Atlantik
    ke Paris.
  example_title: Indonesian
- text: Амелія Эрхарт пераляцела на сваім аднаматорным Lockheed Vega 5B праз Атлантыку
    ў Парыж.
  example_title: Belarusian
- text: Амелія Ергарт перелетіла на своєму одномоторному літаку Lockheed Vega 5B через
    Атлантику до Парижа.
  example_title: Ukrainian
- text: Amelia Earhart preletjela je svojim jednomotornim zrakoplovom Lockheed Vega
    5B preko Atlantika do Pariza.
  example_title: Croatian
- text: Amelia Earhart lendas oma ühemootoriga Lockheed Vega 5B üle Atlandi ookeani
    Pariisi .
  example_title: Estonian
base_model: bert-base-multilingual-cased
model-index:
- name: span-marker-bert-base-multilingual-cased-multinerd
  results:
  - task:
      type: token-classification
      name: Named Entity Recognition
    dataset:
      name: MultiNERD
      type: Babelscape/multinerd
      split: test
      revision: 2814b78e7af4b5a1f1886fe7ad49632de4d9dd25
    metrics:
    - type: f1
      value: 0.927
      name: F1
    - type: precision
      value: 0.9281
      name: Precision
    - type: recall
      value: 0.9259
      name: Recall
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# span-marker-bert-base-multilingual-cased-multinerd

This model is a fine-tuned version of [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased) on an [Babelscape/multinerd](https://huggingface.co/datasets/Babelscape/multinerd) dataset.

Is your data not (always) capitalized correctly? Then consider using the uncased variant of this model instead for better performance: 
[lxyuan/span-marker-bert-base-multilingual-uncased-multinerd](https://huggingface.co/lxyuan/span-marker-bert-base-multilingual-uncased-multinerd).

This model achieves the following results on the evaluation set:
- Loss: 0.0049
- Overall Precision: 0.9242
- Overall Recall: 0.9281
- Overall F1: 0.9261
- Overall Accuracy: 0.9852

Test set results:
- test_loss: 0.005226554349064827,
- test_overall_accuracy: 0.9851129807294873,
- test_overall_f1: 0.9270450073152169,
- test_overall_precision: 0.9281906912835416,
- test_overall_recall: 0.9259021481405626,
- test_runtime: 2690.9722,
- test_samples_per_second: 150.748,
- test_steps_per_second: 4.711


This is a replication of Tom's work. Everything remains unchanged, 
except that we extended the number of training epochs to 3 for a 
slightly longer training duration and set the gradient_accumulation_steps to 2. 
Please refer to the official [model page](https://huggingface.co/tomaarsen/span-marker-mbert-base-multinerd) to review their results and training script

## Results:


| **Language** | **Precision** | **Recall** | **F1**     |
|--------------|---------------|------------|------------|
| **all**      | 92.42         | 92.81      | **92.61**  |
| **de**       | 95.03         | 95.07      | **95.05**  |
| **en**       | 95.00         | 95.40      | **95.20**  |
| **es**       | 92.05         | 91.37      | **91.71**  |
| **fr**       | 92.37         | 91.41      | **91.89**  | 
| **it**       | 91.45         | 93.15      | **92.29**  | 
| **nl**       | 93.85         | 92.98      | **93.41**  |
| **pl**       | 93.13         | 92.66      | **92.89**  |
| **pt**       | 93.60         | 92.50      | **93.05**  |
| **ru**       | 93.25         | 93.32      | **93.29**  | 
| **zh**       | 89.47         | 88.40      | **88.93**  |

- Special thanks to Tom for creating the evaluation script and generating the [results](https://huggingface.co/lxyuan/span-marker-bert-base-multilingual-cased-multinerd/discussions/1).


## Label set

| Class | Description | Examples |
|-------|-------------|----------|
| **PER (person)** | People | Ray Charles, Jessica Alba, Leonardo DiCaprio, Roger Federer, Anna Massey. |
| **ORG (organization)** | Associations, companies, agencies, institutions, nationalities and religious or political groups | University of Edinburgh, San Francisco Giants, Google, Democratic Party. |
| **LOC (location)** | Physical locations (e.g. mountains, bodies of water), geopolitical entities (e.g. cities, states), and facilities (e.g. bridges, buildings, airports). | Rome, Lake Paiku, Chrysler Building, Mount Rushmore, Mississippi River. |
| **ANIM (animal)** | Breeds of dogs, cats and other animals, including their scientific names. | Maine Coon, African Wild Dog, Great White Shark, New Zealand Bellbird. |
| **BIO (biological)** | Genus of fungus, bacteria and protoctists, families of viruses, and other biological entities. | Herpes Simplex Virus, Escherichia Coli, Salmonella, Bacillus Anthracis. |
| **CEL (celestial)** | Planets, stars, asteroids, comets, nebulae, galaxies and other astronomical objects. | Sun, Neptune, Asteroid 187 Lamberta, Proxima Centauri, V838 Monocerotis. |
| **DIS (disease)** | Physical, mental, infectious, non-infectious, deficiency, inherited, degenerative, social and self-inflicted diseases. | Alzheimer’s Disease, Cystic Fibrosis, Dilated Cardiomyopathy, Arthritis. |
| **EVE (event)** | Sport events, battles, wars and other events. | American Civil War, 2003 Wimbledon Championships, Cannes Film Festival. |
| **FOOD (food)** | Foods and drinks. | Carbonara, Sangiovese, Cheddar Beer Fondue, Pizza Margherita. |
| **INST (instrument)** | Technological instruments, mechanical instruments, musical instruments, and other tools. | Spitzer Space Telescope, Commodore 64, Skype, Apple Watch, Fender Stratocaster. |
| **MEDIA (media)** | Titles of films, books, magazines, songs and albums, fictional characters and languages. | Forbes, American Psycho, Kiss Me Once, Twin Peaks, Disney Adventures. |
| **PLANT (plant)** | Types of trees, flowers, and other plants, including their scientific names. | Salix, Quercus Petraea, Douglas Fir, Forsythia, Artemisia Maritima. |
| **MYTH (mythological)** | Mythological and religious entities. | Apollo, Persephone, Aphrodite, Saint Peter, Pope Gregory I, Hercules. |
| **TIME (time)** | Specific and well-defined time intervals, such as eras, historical periods, centuries, years and important days. No months and days of the week. | Renaissance, Middle Ages, Christmas, Great Depression, 17th Century, 2012. |
| **VEHI (vehicle)** | Cars, motorcycles and other vehicles. | Ferrari Testarossa, Suzuki Jimny, Honda CR-X, Boeing 747, Fairey Fulmar. |



## Inference Example

```python
# install span_marker
(env)$ pip install span_marker


from span_marker import SpanMarkerModel

model = SpanMarkerModel.from_pretrained("lxyuan/span-marker-bert-base-multilingual-cased-multinerd")

description = "Singapore is renowned for its hawker centers offering dishes \
like Hainanese chicken rice and laksa, while Malaysia boasts dishes such as \
nasi lemak and rendang, reflecting its rich culinary heritage."

entities = model.predict(description)

entities
>>>
[
  {'span': 'Singapore', 'label': 'LOC', 'score': 0.999988317489624, 'char_start_index': 0, 'char_end_index': 9},
  {'span': 'Hainanese chicken rice', 'label': 'FOOD', 'score': 0.9894770383834839, 'char_start_index': 66, 'char_end_index': 88},
  {'span': 'laksa', 'label': 'FOOD', 'score': 0.9224908947944641, 'char_start_index': 93, 'char_end_index': 98},
  {'span': 'Malaysia', 'label': 'LOC', 'score': 0.9999839067459106, 'char_start_index': 106, 'char_end_index': 114}]

# missed: nasi lemak as FOOD
# missed: rendang as FOOD
# :(
```

#### Quick test on Chinese
```python
from span_marker import SpanMarkerModel

model = SpanMarkerModel.from_pretrained("lxyuan/span-marker-bert-base-multilingual-cased-multinerd")

# translate to chinese
description = "Singapore is renowned for its hawker centers offering dishes \
like Hainanese chicken rice and laksa, while Malaysia boasts dishes such as \
nasi lemak and rendang, reflecting its rich culinary heritage."

zh_description = "新加坡因其小贩中心提供海南鸡饭和叻沙等菜肴而闻名, 而马来西亚则拥有椰浆饭和仁当等菜肴，反映了其丰富的烹饪传统."

entities = model.predict(zh_description)

entities
>>>
[
  {'span': '新加坡', 'label': 'LOC', 'score': 0.9282007813453674, 'char_start_index': 0, 'char_end_index': 3},
  {'span': '马来西亚', 'label': 'LOC', 'score': 0.7439665794372559, 'char_start_index': 27, 'char_end_index': 31}]

# It only managed to capture two countries: Singapore and Malaysia.
# All other entities were missed out.
```


## Training procedure

One can reproduce the result running this [script](https://huggingface.co/tomaarsen/span-marker-mbert-base-multinerd/blob/main/train.py)

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step   | Validation Loss | Overall Precision | Overall Recall | Overall F1 | Overall Accuracy |
|:-------------:|:-----:|:------:|:---------------:|:-----------------:|:--------------:|:----------:|:----------------:|
| 0.0129        | 1.0   | 50436  | 0.0042          | 0.9226            | 0.9169         | 0.9197     | 0.9837           |
| 0.0027        | 2.0   | 100873 | 0.0043          | 0.9255            | 0.9206         | 0.9230     | 0.9846           |
| 0.0015        | 3.0   | 151308 | 0.0049          | 0.9242            | 0.9281         | 0.9261     | 0.9852           |


### Framework versions

- Transformers 4.30.2
- Pytorch 2.0.1+cu117
- Datasets 2.14.3
- Tokenizers 0.13.3