tomaarsen
/

span-marker-bert-base-uncased-cross-ner

@@ -8,12 +8,88 @@ tags:
 - ner
 - named-entity-recognition
 pipeline_tag: token-classification
 ---
-# SpanMarker for Named Entity Recognition
-This is a [SpanMarker](https://github.com/tomaarsen/SpanMarkerNER) model that can be used for Named Entity Recognition. In particular, this SpanMarker model uses [bert-base-uncased](https://huggingface.co/bert-base-uncased) as the underlying encoder.
 ## Usage
@@ -29,9 +105,48 @@ You can then run inference with this model like so:
 from span_marker import SpanMarkerModel
 # Download from the 🤗 Hub
-model = SpanMarkerModel.from_pretrained("span_marker_model_name")
 # Run inference
-entities = model.predict("Amelia Earhart flew her single engine Lockheed Vega 5B across the Atlantic to Paris.")
 ```
 See the [SpanMarker](https://github.com/tomaarsen/SpanMarkerNER) repository for documentation and additional information on this library.

 - ner
 - named-entity-recognition
 pipeline_tag: token-classification
+model-index:
+  - name: SpanMarker w. bert-base-uncased on CrossNER by Tom Aarsen
+    results:
+      - task:
+          type: token-classification
+          name: Named Entity Recognition
+        dataset:
+          type: P3ps/Cross_ner
+          name: CrossNER
+          split: test
+          revision: 7cecbbb3d2eb8c75c8571c53e5a5270cfd0c5a9e
+        metrics:
+          - type: f1
+            value: 0.8708
+            name: F1
+          - type: precision
+            value: 0.8763
+            name: Precision
+          - type: recall
+            value: 0.8654
+            name: Recall
+datasets:
+  - P3ps/Cross_ner
+language:
+  - en
+metrics:
+  - f1
+  - recall
+  - precision
 ---
+# SpanMarker for uncased Named Entity Recognition
+This is a [SpanMarker](https://github.com/tomaarsen/SpanMarkerNER) model that can be used for Named Entity Recognition. In particular, this SpanMarker model uses [bert-base-uncased](https://huggingface.co/bert-base-uncased) as the underlying encoder. See [train.py](train.py) for the training script.
+Is your data always capitalized correctly? Then consider using the cased variant of this model instead for better performance:
+[tomaarsen/span-marker-bert-base-cross-ner](https://huggingface.co/tomaarsen/span-marker-bert-base-cross-ner).
+## Labels & Metrics
+| **Label**          | **Examples** |  **Precision** |   **Recall** |    **F1** |
+|:-------------------|-|------------:|---------:|------:|
+| **all**            | - |       87.63 |    86.54 | 87.08 |
+| academicjournal    | "new journal of physics", "epl", "european physical journal b" |       82.22 |    90.24 | 86.05 |
+| album              | "tellin' stories", "generation terrorists", "country airs" |       84.46 |    84.46 | 84.46 |
+| algorithm          | "lda", "pca", "gradient descent" |       82.86 |    76.99 | 79.82 |
+| astronomicalobject | "earth", "sun", "halley's comet" |       88.61 |    94.59 | 91.50 |
+| award              | "nobel prize for literature", "acamedy award for best actress", "mandelbrot's awards" |       87.76 |    91.63 | 89.66 |
+| band               | "clash", "parliament funkadelic", "sly and the family stone" |       82.72 |    85.35 | 84.01 |
+| book               | "nietzsche contra wagner" , "dionysian-dithyrambs", "the rebel" |       68.51 |    79.49 | 73.59 |
+| chemicalcompound   | "hydrogen sulfide", "starch", "lactic acid" |       73.33 |    66.67 | 69.84 |
+| chemicalelement    | "potassium", "fluorine", "chlorine" |       95.65 |    73.33 | 83.02 |
+| conference         | "siggraph", "ijcai", "ieee transactions on speech and audio processing" |       72.41 |    60.00 | 65.62 |
+| country            | "united arab emirates", "u.s.", "canada" |       81.03 |    86.08 | 83.48 |
+| discipline         | "physics", "meteorology", "geography" |       35.48 |    40.74 | 37.93 |
+| election           | "2004 canadian federal election", "2006 canadian federal election", "1999 scottish parliament election" |       96.22 |    98.28 | 97.24 |
+| enzyme             | "rna polymerase", "phosphoinositide 3-kinase", "protein kinase c" |       72.09 |    83.78 | 77.50 |
+| event              | "cannes film festival", "2019 special olympics world summer games", "2017 western iraq campaign" |       68.12 |    60.22 | 63.93 |
+| field              | "computational imaging", "electronics", "information theory" |       92.13 |    77.36 | 84.10 |
+| literarygenre      | "novel", "satire", "short story" |       65.26 |    72.09 | 68.51 |
+| location           | "china", "bombay", "serbia" |       94.78 |    93.68 | 94.23 |
+| magazine           | "the atlantic", "the american spectator", "astounding science fiction" |       60.71 |    60.71 | 60.71 |
+| metrics            | "bleu", "precision", "dcg" |       77.01 |    82.72 | 79.76 |
+| misc               | "serbian", "belgian", "the birth of a nation" |       80.11 |    72.12 | 75.91 |
+| musicalartist      | "chuck burgi", "john miceli", "john o'reilly" |       78.84 |    84.44 | 81.55 |
+| musicalinstrument  | "koto", "bubens", "def" |       75.00 |    33.33 | 46.15 |
+| musicgenre         | "christian rock", "punk rock", "romantic melodicism" |       88.21 |    88.21 | 88.21 |
+| organisation       | "irish times", "comintern", "wimbledon" |       89.17 |    89.98 | 89.57 |
+| person             | "gong zhichao", "liu lufung", "margret crowley" |       95.87 |    92.65 | 94.23 |
+| poem               | "historia destructionis troiae", "i am joaquin", "the snow man" |       94.29 |    64.71 | 76.74 |
+| politicalparty     | "new democratic party", "bloc québécois", "liberal party of canada" |       87.16 |    84.50 | 85.81 |
+| politician         | "susan kadis", "simon strelchik", "lloyd helferty" |       85.23 |    90.71 | 87.89 |
+| product            | "alphago", "wordnet", "facial recognition system" |       63.95 |    65.48 | 64.71 |
+| programlang        | "r", "c++", "java" |       75.00 |    84.38 | 79.41 |
+| protein            | "dna methyltransferase", "tau protein", "amyloid beta" |       57.50 |    66.67 | 61.74 |
+| researcher         | "sirovich", "kirby", "matthew turk" |       93.06 |    75.28 | 83.23 |
+| scientist          | "matjaž perc", "cotton", "singer" |       80.27 |    93.72 | 86.47 |
+| song               | "right where i'm supposed to be", "easy", "three times a lady" |       89.87 |    82.56 | 86.06 |
+| task               | "robot control", "elevator scheduling", "telecommunications" |       73.86 |    75.58 | 74.71 |
+| theory             | "big bang", "general theory of relativity", "ptolemaic planetary theories" |        0.00 |     0.00 |  0.00 |
+| university         | "university of göttingen", "duke", "imperial academy of sciences" |       79.78 |    79.78 | 79.78 |
+| writer             | "thomas mann", "george bernard shaw", "thomas hardy" |       77.78 |    86.19 | 81.77 |
 ## Usage
 from span_marker import SpanMarkerModel
 # Download from the 🤗 Hub
+model = SpanMarkerModel.from_pretrained("tomaarsen/span-marker-bert-base-uncased-cross-ner")
 # Run inference
+entities = model.predict("amelia earhart flew her single engine lockheed vega 5b across the atlantic to paris.")
 ```
 See the [SpanMarker](https://github.com/tomaarsen/SpanMarkerNER) repository for documentation and additional information on this library.
+## Training procedure
+### Training hyperparameters
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- train_batch_size: 32
+- eval_batch_size: 32
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 3
+### Training results
+| Training Loss | Epoch | Step | Validation Loss | Overall Precision | Overall Recall | Overall F1 | Overall Accuracy |
+|:-------------:|:-----:|:----:|:---------------:|:-----------------:|:--------------:|:----------:|:----------------:|
+| 0.0641        | 0.25  | 200  | 0.0445          | 0.7141            | 0.5496         | 0.6212     | 0.8700           |
+| 0.0268        | 0.5   | 400  | 0.0224          | 0.8171            | 0.7510         | 0.7827     | 0.9314           |
+| 0.0213        | 0.76  | 600  | 0.0187          | 0.8387            | 0.8013         | 0.8196     | 0.9444           |
+| 0.017         | 1.01  | 800  | 0.0162          | 0.8623            | 0.8231         | 0.8422     | 0.9497           |
+| 0.0141        | 1.26  | 1000 | 0.0163          | 0.8571            | 0.8384         | 0.8477     | 0.9535           |
+| 0.0132        | 1.51  | 1200 | 0.0149          | 0.8711            | 0.8470         | 0.8589     | 0.9563           |
+| 0.0113        | 1.76  | 1400 | 0.0150          | 0.8603            | 0.8523         | 0.8563     | 0.9556           |
+| 0.0097        | 2.02  | 1600 | 0.0150          | 0.8710            | 0.8553         | 0.8631     | 0.9573           |
+| 0.0083        | 2.27  | 1800 | 0.0148          | 0.8809            | 0.8568         | 0.8687     | 0.9586           |
+| 0.0075        | 2.52  | 2000 | 0.0150          | 0.8733            | 0.8573         | 0.8652     | 0.9583           |
+| 0.0068        | 2.77  | 2200 | 0.0148          | 0.8745            | 0.8642         | 0.8693     | 0.9600           |
+### Framework versions
+- SpanMarker 1.2.4
+- Transformers 4.31.0
+- Pytorch 2.0.1+cu118
+- Datasets 2.14.3
+- Tokenizers 0.13.2