---
language:
- en
license: cc-by-sa-4.0
library_name: span-marker
tags:
- span-marker
- token-classification
- ner
- named-entity-recognition
- generated_from_span_marker_trainer
datasets:
- tomaarsen/ner-orgs
metrics:
- precision
- recall
- f1
widget:
- text: De Napoli played for FC Luzern in the second half of the 2005–06 Swiss Super
    League campaign, scoring five times in fifteen games and helping Luzern to promotion
    from the Swiss Challenge League.
- text: The issue continued to simmer while full-communion agreements with the Presbyterian
    Church USA, Reformed Church in America, United Church of Christ, and Episcopal
    Church (United States) were debated and adopted in 1997 and 1999.
- text: Rune Gerhardsen (born 13 June 1946) is a Norwegian politician, representing
    the Norwegian Labour Party and a former sports leader at Norwegian Skating Association
    representing from Aktiv SK.
- text: Konstantin Vladimirovich Pushkaryov (; born February 12, 1985) is a Kazakhstani
    professional ice hockey winger who is currently playing with HK Kurbads of the
    Latvian Hockey League (LAT).
- text: SCL claims that its methodology has been approved or endorsed by agencies
    of the Government of the United Kingdom and the Federal government of the United
    States, among others.
pipeline_tag: token-classification
base_model: microsoft/xtremedistil-l12-h384-uncased
model-index:
- name: SpanMarker with microsoft/xtremedistil-l12-h384-uncased on FewNERD, CoNLL2003,
    and OntoNotes v5
  results:
  - task:
      type: token-classification
      name: Named Entity Recognition
    dataset:
      name: FewNERD, CoNLL2003, and OntoNotes v5
      type: tomaarsen/ner-orgs
      split: test
    metrics:
    - type: f1
      value: 0.7558602090122487
      name: F1
    - type: precision
      value: 0.7620428694430598
      name: Precision
    - type: recall
      value: 0.749777064383806
      name: Recall
---

# SpanMarker with microsoft/xtremedistil-l12-h384-uncased on FewNERD, CoNLL2003, and OntoNotes v5

This is a [SpanMarker](https://github.com/tomaarsen/SpanMarkerNER) model trained on the [FewNERD, CoNLL2003, and OntoNotes v5](https://huggingface.co/datasets/tomaarsen/ner-orgs) dataset that can be used for Named Entity Recognition. This SpanMarker model uses [microsoft/xtremedistil-l12-h384-uncased](https://huggingface.co/microsoft/xtremedistil-l12-h384-uncased) as the underlying encoder.

## Model Details

### Model Description
- **Model Type:** SpanMarker
- **Encoder:** [microsoft/xtremedistil-l12-h384-uncased](https://huggingface.co/microsoft/xtremedistil-l12-h384-uncased)
- **Maximum Sequence Length:** 256 tokens
- **Maximum Entity Length:** 8 words
- **Training Dataset:** [FewNERD, CoNLL2003, and OntoNotes v5](https://huggingface.co/datasets/tomaarsen/ner-orgs)
- **Language:** en
- **License:** cc-by-sa-4.0

### Model Sources

- **Repository:** [SpanMarker on GitHub](https://github.com/tomaarsen/SpanMarkerNER)
- **Thesis:** [SpanMarker For Named Entity Recognition](https://raw.githubusercontent.com/tomaarsen/SpanMarkerNER/main/thesis.pdf)

### Model Labels
| Label | Examples                                     |
|:------|:---------------------------------------------|
| ORG   | "Texas Chicken", "IAEA", "Church 's Chicken" |

## Evaluation

### Metrics
| Label   | Precision | Recall | F1     |
|:--------|:----------|:-------|:-------|
| **all** | 0.7620    | 0.7498 | 0.7559 |
| ORG     | 0.7620    | 0.7498 | 0.7559 |

## Uses

### Direct Use for Inference

```python
from span_marker import SpanMarkerModel

# Download from the 🤗 Hub
model = SpanMarkerModel.from_pretrained("nbroad/span-marker-xdistil-l12-h384-orgs-v3")
# Run inference
entities = model.predict("SCL claims that its methodology has been approved or endorsed by agencies of the Government of the United Kingdom and the Federal government of the United States, among others.")
```

### Downstream Use
You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

```python
from span_marker import SpanMarkerModel, Trainer

# Download from the 🤗 Hub
model = SpanMarkerModel.from_pretrained("nbroad/span-marker-xdistil-l12-h384-orgs-v3")

# Specify a Dataset with "tokens" and "ner_tag" columns
dataset = load_dataset("conll2003") # For example CoNLL2003

# Initialize a Trainer using the pretrained model & dataset
trainer = Trainer(
    model=model,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
)
trainer.train()
trainer.save_model("nbroad/span-marker-xdistil-l12-h384-orgs-v3-finetuned")
```
</details>

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set          | Min | Median  | Max |
|:----------------------|:----|:--------|:----|
| Sentence length       | 1   | 23.5706 | 263 |
| Entities per sentence | 0   | 0.7865  | 39  |

### Training Hyperparameters
- learning_rate: 0.0003
- train_batch_size: 128
- eval_batch_size: 128
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 3
- mixed_precision_training: Native AMP

### Training Results
| Epoch  | Step | Validation Loss | Validation Precision | Validation Recall | Validation F1 | Validation Accuracy |
|:------:|:----:|:---------------:|:--------------------:|:-----------------:|:-------------:|:-------------------:|
| 0.5720 | 600  | 0.0086          | 0.7150               | 0.7095            | 0.7122        | 0.9660              |
| 1.1439 | 1200 | 0.0074          | 0.7556               | 0.7253            | 0.7401        | 0.9682              |
| 1.7159 | 1800 | 0.0073          | 0.7482               | 0.7619            | 0.7550        | 0.9702              |
| 2.2879 | 2400 | 0.0072          | 0.7761               | 0.7573            | 0.7666        | 0.9713              |
| 2.8599 | 3000 | 0.0070          | 0.7691               | 0.7688            | 0.7689        | 0.9720              |

### Framework Versions
- Python: 3.10.12
- SpanMarker: 1.5.0
- Transformers: 4.35.2
- PyTorch: 2.1.0a0+32f93b1
- Datasets: 2.15.0
- Tokenizers: 0.15.0

## Citation

### BibTeX
```
@software{Aarsen_SpanMarker,
    author = {Aarsen, Tom},
    license = {Apache-2.0},
    title = {{SpanMarker for Named Entity Recognition}},
    url = {https://github.com/tomaarsen/SpanMarkerNER}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->