tomaarsen/span-marker-xlm-roberta-base-fewnerd-fine-super

SpanMarker with xlm-roberta-base on FewNERD

This is a SpanMarker model trained on the FewNERD dataset that can be used for Named Entity Recognition. This SpanMarker model uses xlm-roberta-base as the underlying encoder.

Model Details

Model Description

• Model Type: SpanMarker
• Encoder: xlm-roberta-base
• Maximum Sequence Length: 256 tokens
• Maximum Entity Length: 8 words
• Training Dataset: FewNERD
• Languages: en, multilingual
• License: cc-by-sa-4.0

Model Sources

• Repository: SpanMarker on GitHub
• Thesis: SpanMarker For Named Entity Recognition

Model Labels

Label	Examples
art-broadcastprogram	"The Gale Storm Show : Oh , Susanna", "Corazones", "Street Cents"
art-film	"L'Atlantide", "Shawshank Redemption", "Bosch"
art-music	"Hollywood Studio Symphony", "Atkinson , Danko and Ford ( with Brockie and Hilton )", "Champion Lover"
art-other	"Venus de Milo", "Aphrodite of Milos", "The Today Show"
art-painting	"Cofiwch Dryweryn", "Production/Reproduction", "Touit"
art-writtenart	"The Seven Year Itch", "Time", "Imelda de ' Lambertazzi"
building-airport	"Newark Liberty International Airport", "Luton Airport", "Sheremetyevo International Airport"
building-hospital	"Hokkaido University Hospital", "Yeungnam University Hospital", "Memorial Sloan-Kettering Cancer Center"
building-hotel	"Radisson Blu Sea Plaza Hotel", "The Standard Hotel", "Flamingo Hotel"
building-library	"British Library", "Berlin State Library", "Bayerische Staatsbibliothek"
building-other	"Communiplex", "Henry Ford Museum", "Alpha Recording Studios"
building-restaurant	"Fatburger", "Carnegie Deli", "Trumbull"
building-sportsfacility	"Boston Garden", "Glenn Warner Soccer Facility", "Sports Center"
building-theater	"Pittsburgh Civic Light Opera", "National Paris Opera", "Sanders Theatre"
event-attack/battle/war/militaryconflict	"Jurist", "Easter Offensive", "Vietnam War"
event-disaster	"1693 Sicily earthquake", "1990s North Korean famine", "the 1912 North Mount Lyell Disaster"
event-election	"March 1898 elections", "Elections to the European Parliament", "1982 Mitcham and Morden by-election"
event-other	"Eastwood Scoring Stage", "Union for a Popular Movement", "Masaryk Democratic Movement"
event-protest	"Russian Revolution", "French Revolution", "Iranian Constitutional Revolution"
event-sportsevent	"World Cup", "Stanley Cup", "National Champions"
location-GPE	"Mediterranean Basin", "Croatian", "the Republic of Croatia"
location-bodiesofwater	"Norfolk coast", "Atatürk Dam Lake", "Arthur Kill"
location-island	"Laccadives", "Staten Island", "new Samsat district"
location-mountain	"Ruweisat Ridge", "Miteirya Ridge", "Salamander Glacier"
location-other	"Victoria line", "Northern City Line", "Cartuther"
location-park	"Painted Desert Community Complex Historic District", "Shenandoah National Park", "Gramercy Park"
location-road/railway/highway/transit	"Newark-Elizabeth Rail Link", "NJT", "Friern Barnet Road"
organization-company	"Church 's Chicken", "Texas Chicken", "Dixy Chicken"
organization-education	"MIT", "Belfast Royal Academy and the Ulster College of Physical Education", "Barnard College"
organization-government/governmentagency	"Congregazione dei Nobili", "Diet", "Supreme Court"
organization-media/newspaper	"TimeOut Melbourne", "Al Jazeera", "Clash"
organization-other	"IAEA", "4th Army", "Defence Sector C"
organization-politicalparty	"Al Wafa ' Islamic", "Shimpotō", "Kenseitō"
organization-religion	"UPCUSA", "Jewish", "Christian"
organization-showorganization	"Bochumer Symphoniker", "Mr. Mister", "Lizzy"
organization-sportsleague	"First Division", "NHL", "China League One"
organization-sportsteam	"Tottenham", "Arsenal", "Luc Alphand Aventures"
other-astronomything	"Algol", "Zodiac", "`` Caput Larvae ''"
other-award	"Grand Commander of the Order of the Niger", "Order of the Republic of Guinea and Nigeria", "GCON"
other-biologything	"Amphiphysin", "BAR", "N-terminal lipid"
other-chemicalthing	"carbon dioxide", "sulfur", "uranium"
other-currency	"$", "lac crore", "Travancore Rupee"
other-disease	"hypothyroidism", "bladder cancer", "French Dysentery Epidemic of 1779"
other-educationaldegree	"Master", "Bachelor", "BSc ( Hons ) in physics"
other-god	"El", "Fujin", "Raijin"
other-language	"Breton-speaking", "Latin", "English"
other-law	"United States Freedom Support Act", "Thirty Years ' Peace", "Leahy–Smith America Invents Act ( AIA"
other-livingthing	"insects", "patchouli", "monkeys"
other-medical	"amitriptyline", "pediatrician", "Pediatrics"
person-actor	"Tchéky Karyo", "Edmund Payne", "Ellaline Terriss"
person-artist/author	"George Axelrod", "Hicks", "Gaetano Donizett"
person-athlete	"Jaguar", "Neville", "Tozawa"
person-director	"Richard Quine", "Frank Darabont", "Bob Swaim"
person-other	"Campbell", "Richard Benson", "Holden"
person-politician	"Rivière", "Emeric", "William"
person-scholar	"Stedman", "Wurdack", "Stalmine"
person-soldier	"Joachim Ziegler", "Krukenberg", "Helmuth Weidling"
product-airplane	"EC135T2 CPDS", "Spey-equipped FGR.2s", "Luton"
product-car	"Phantom", "Corvettes - GT1 C6R", "100EX"
product-food	"V. labrusca", "red grape", "yakiniku"
product-game	"Hardcore RPG", "Airforce Delta", "Splinter Cell"
product-other	"PDP-1", "Fairbottom Bobs", "X11"
product-ship	"Essex", "Congress", "HMS `` Chinkara ''"
product-software	"Wikipedia", "Apdf", "AmiPDF"
product-train	"55022", "Royal Scots Grey", "High Speed Trains"
product-weapon	"AR-15 's", "ZU-23-2MR Wróbel II", "ZU-23-2M Wróbel"

Evaluation

Metrics

Label	Precision	Recall	F1
all	0.6890	0.6879	0.6885
art-broadcastprogram	0.6	0.5771	0.5883
art-film	0.7384	0.7453	0.7419
art-music	0.7930	0.7221	0.7558
art-other	0.4245	0.2900	0.3446
art-painting	0.5476	0.4035	0.4646
art-writtenart	0.6400	0.6539	0.6469
building-airport	0.8219	0.8242	0.8230
building-hospital	0.7024	0.8104	0.7526
building-hotel	0.7175	0.7283	0.7228
building-library	0.74	0.7296	0.7348
building-other	0.5828	0.5910	0.5869
building-restaurant	0.5525	0.5216	0.5366
building-sportsfacility	0.6187	0.7881	0.6932
building-theater	0.7067	0.7626	0.7336
event-attack/battle/war/militaryconflict	0.7544	0.7468	0.7506
event-disaster	0.5882	0.5314	0.5584
event-election	0.4167	0.2198	0.2878
event-other	0.4902	0.4042	0.4430
event-protest	0.3643	0.2831	0.3186
event-sportsevent	0.6125	0.6239	0.6182
location-GPE	0.8102	0.8553	0.8321
location-bodiesofwater	0.6888	0.7725	0.7282
location-island	0.7285	0.6440	0.6836
location-mountain	0.7129	0.7327	0.7227
location-other	0.4376	0.2560	0.3231
location-park	0.6991	0.6900	0.6945
location-road/railway/highway/transit	0.6936	0.7259	0.7094
organization-company	0.6921	0.6912	0.6917
organization-education	0.7838	0.7963	0.7900
organization-government/governmentagency	0.5363	0.4394	0.4831
organization-media/newspaper	0.6215	0.6705	0.6451
organization-other	0.5766	0.5157	0.5444
organization-politicalparty	0.6449	0.7324	0.6859
organization-religion	0.5139	0.6057	0.5560
organization-showorganization	0.5620	0.5657	0.5638
organization-sportsleague	0.6348	0.6542	0.6443
organization-sportsteam	0.7138	0.7566	0.7346
other-astronomything	0.7418	0.7625	0.752
other-award	0.7291	0.6736	0.7002
other-biologything	0.6735	0.6275	0.6497
other-chemicalthing	0.6025	0.5651	0.5832
other-currency	0.6843	0.8411	0.7546
other-disease	0.6284	0.7089	0.6662
other-educationaldegree	0.5856	0.6033	0.5943
other-god	0.6089	0.6913	0.6475
other-language	0.6608	0.7968	0.7225
other-law	0.6693	0.7246	0.6958
other-livingthing	0.6070	0.6014	0.6042
other-medical	0.5062	0.5113	0.5088
person-actor	0.8274	0.7673	0.7962
person-artist/author	0.6761	0.7294	0.7018
person-athlete	0.8132	0.8347	0.8238
person-director	0.675	0.6823	0.6786
person-other	0.6472	0.6388	0.6429
person-politician	0.6621	0.6593	0.6607
person-scholar	0.5181	0.5007	0.5092
person-soldier	0.4750	0.5131	0.4933
product-airplane	0.6230	0.6717	0.6464
product-car	0.7293	0.7176	0.7234
product-food	0.5758	0.5185	0.5457
product-game	0.7049	0.6734	0.6888
product-other	0.5477	0.4067	0.4668
product-ship	0.6247	0.6395	0.6320
product-software	0.6497	0.6760	0.6626
product-train	0.5505	0.5732	0.5616
product-weapon	0.6004	0.4744	0.5300

Uses

Direct Use for Inference

from span_marker import SpanMarkerModel

# Download from the 🤗 Hub
model = SpanMarkerModel.from_pretrained("tomaarsen/span-marker-xlm-roberta-base-fewnerd-fine-super")
# Run inference
entities = model.predict("Most of the Steven Seagal movie \"Under Siege \"(co-starring Tommy Lee Jones) was filmed on the, which is docked on Mobile Bay at Battleship Memorial Park and open to the public.")

Downstream Use

You can finetune this model on your own dataset.

Click to expand

from span_marker import SpanMarkerModel, Trainer

# Download from the 🤗 Hub
model = SpanMarkerModel.from_pretrained("tomaarsen/span-marker-xlm-roberta-base-fewnerd-fine-super")

# Specify a Dataset with "tokens" and "ner_tag" columns
dataset = load_dataset("conll2003") # For example CoNLL2003

# Initialize a Trainer using the pretrained model & dataset
trainer = Trainer(
    model=model,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
)
trainer.train()
trainer.save_model("tomaarsen/span-marker-xlm-roberta-base-fewnerd-fine-super-finetuned")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Sentence length	1	24.4945	267
Entities per sentence	0	2.5832	88

Training Hyperparameters

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3

Training Results

Epoch	Step	Validation Loss	Validation Precision	Validation Recall	Validation F1	Validation Accuracy
0.2947	3000	0.0318	0.6058	0.5990	0.6024	0.9020
0.5893	6000	0.0266	0.6556	0.6679	0.6617	0.9173
0.8840	9000	0.0250	0.6691	0.6804	0.6747	0.9206
1.1787	12000	0.0239	0.6865	0.6761	0.6813	0.9212
1.4733	15000	0.0234	0.6872	0.6812	0.6842	0.9226
1.7680	18000	0.0231	0.6919	0.6821	0.6870	0.9227
2.0627	21000	0.0231	0.6909	0.6871	0.6890	0.9233
2.3573	24000	0.0231	0.6903	0.6875	0.6889	0.9238
2.6520	27000	0.0229	0.6918	0.6926	0.6922	0.9242
2.9467	30000	0.0228	0.6927	0.6930	0.6928	0.9243

Environmental Impact

Carbon emissions were measured using CodeCarbon.

• Carbon Emitted: 0.453 kg of CO2
• Hours Used: 3.118 hours

Training Hardware

• On Cloud: No
• GPU Model: 1 x NVIDIA GeForce RTX 3090
• CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
• RAM Size: 31.78 GB

Framework Versions

• Python: 3.9.16
• SpanMarker: 1.4.1.dev
• Transformers: 4.30.0
• PyTorch: 2.0.1+cu118
• Datasets: 2.14.0
• Tokenizers: 0.13.2

Citation

BibTeX

@software{Aarsen_SpanMarker,
    author = {Aarsen, Tom},
    license = {Apache-2.0},
    title = {{SpanMarker for Named Entity Recognition}},
    url = {https://github.com/tomaarsen/SpanMarkerNER}
}