--- license: apache-2.0 datasets: - Babelscape/multinerd language: - en metrics: - f1 - precision - recall pipeline_tag: token-classification tags: - ner - named-entity-recognition - token-classification model-index: - name: robert-base on MultiNERD by Jayant Yadav results: - task: type: named-entity-recognition-ner name: Named Entity Recognition dataset: type: Babelscape/multinerd name: MultiNERD (English) split: test revision: 2814b78e7af4b5a1f1886fe7ad49632de4d9dd25 config: Babelscape/multinerd args: split: train[:50%] metrics: - type: f1 value: 0.943 name: F1 - type: precision value: 0.939 name: Precision - type: recall value: 0.947 name: Recall config: seqeval paper: https://aclanthology.org/2022.findings-naacl.60.pdf base_model: roberta-base library_name: transformers --- # Model Card for Model ID [roBERTa-base](https://huggingface.co/roberta-base) model was fine-tuned on 50% training English only split of MultiNERD dataset and later evaluated on full test split of the same. The finetuning script can be fetched from [fintuning.py](https://github.com/jayant-yadav/RISE-NER/blob/main/finetuning.ipynb). Various other model were tested on the same selection of dataset and the best checkpoint was uploaded. The detailed configuration summary can be found in Appendix section of [report](https://github.com/jayant-yadav/RISE-NER/blob/main/MultiNERD_NER___RISE.pdf). ## Model Details ### Model Description Head over to [github repo](https://github.com/jayant-yadav/RISE-NER) for all the scripts used to finetune and evalute token-classification model. The model is ready to use on [Kaggle](https://www.kaggle.com/datasets/jayantyadav/multinerd-ner-models/) too! - **Developed by:** Jayant Yadav ## Uses Token-classification of the following entities are possible: | Class | Description | Examples | |-------|-------------|----------| PER (person) | People | Ray Charles, Jessica Alba, Leonardo DiCaprio, Roger Federer, Anna Massey. | ORG (organization) | Associations, companies, agencies, institutions, nationalities and religious or political groups | University of Edinburgh, San Francisco Giants, Google, Democratic Party. | LOC (location) | Physical locations (e.g. mountains, bodies of water), geopolitical entities (e.g. cities, states), and facilities (e.g. bridges, buildings, airports). | Rome, Lake Paiku, Chrysler Building, Mount Rushmore, Mississippi River. | ANIM (animal) | Breeds of dogs, cats and other animals, including their scientific names. | Maine Coon, African Wild Dog, Great White Shark, New Zealand Bellbird. | BIO (biological) | Genus of fungus, bacteria and protoctists, families of viruses, and other biological entities. | Herpes Simplex Virus, Escherichia Coli, Salmonella, Bacillus Anthracis. | CEL (celestial) | Planets, stars, asteroids, comets, nebulae, galaxies and other astronomical objects. | Sun, Neptune, Asteroid 187 Lamberta, Proxima Centauri, V838 Monocerotis. | DIS (disease) | Physical, mental, infectious, non-infectious, deficiency, inherited, degenerative, social and self-inflicted diseases. | Alzheimer’s Disease, Cystic Fibrosis, Dilated Cardiomyopathy, Arthritis. | EVE (event) | Sport events, battles, wars and other events. | American Civil War, 2003 Wimbledon Championships, Cannes Film Festival. | FOOD (food) | Foods and drinks. | Carbonara, Sangiovese, Cheddar Beer Fondue, Pizza Margherita. | INST (instrument) | Technological instruments, mechanical instruments, musical instruments, and other tools. | Spitzer Space Telescope, Commodore 64, Skype, Apple Watch, Fender Stratocaster. | MEDIA (media) | Titles of films, books, magazines, songs and albums, fictional characters and languages. | Forbes, American Psycho, Kiss Me Once, Twin Peaks, Disney Adventures. | PLANT (plant) | Types of trees, flowers, and other plants, including their scientific names. | Salix, Quercus Petraea, Douglas Fir, Forsythia, Artemisia Maritima. | MYTH (mythological) | Mythological and religious entities. | Apollo, Persephone, Aphrodite, Saint Peter, Pope Gregory I, Hercules. | TIME (time) | Specific and well-defined time intervals, such as eras, historical periods, centuries, years and important days. No months and days of the week. | Renaissance, Middle Ages, Christmas, Great Depression, 17th Century, 2012. | VEHI (vehicle) | Cars, motorcycles and other vehicles. | Ferrari Testarossa, Suzuki Jimny, Honda CR-X, Boeing 747, Fairey Fulmar. ## Bias, Risks, and Limitations Only trained on English split of MultiNERD dataset. Therefore will not perform well on other languages. ## How to Get Started with the Model Use the code below to get started with the model: ```py from transformers import AutoTokenizer, AutoModelForTokenClassification from transformers import pipeline tokenizer = AutoTokenizer.from_pretrained("jayant-yadav/roberta-base-multinerd") model = AutoModelForTokenClassification.from_pretrained("jayant-yadav/roberta-base-multinerd") nlp = pipeline("ner", model=model, tokenizer=tokenizer) example = "My name is Wolfgang and I live in Berlin" ner_results = nlp(example) print(ner_results) ``` ## Training Details ### Training Data 50% of train split of MultiNERD dataset was used to finetune the model. ### Training Procedure #### Preprocessing English dataset was filterd out : ```train_dataset = train_dataset.filter(lambda x: x['lang'] == 'en')``` #### Training Hyperparameters The following hyperparameters were used during training: learning_rate: 5e-05 train_batch_size: 32 eval_batch_size: 32 seed: 42 optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 lr_scheduler_type: linear lr_scheduler_warmup_ratio: 0.1 num_epochs: 1 ## Evaluation Evaluation was perfored on 50% of evaluation split of MultiNERD dataset. ### Testing Data & Metrics #### Testing Data Tested on Full test split of MultiNERD dataset. #### Metrics Model versions and checkpoint were evaluated using F1, Precision and Recall. For this `seqeval` metric was used: ```metric = load_metric("seqeval")```. ### Results |Entity | Precision | Recall | F1 score | Support | |---|---|---|---|---| |ANIM | 0.71 | 0.77 | 0.739 | 1604 | |BIO | 0.5 | 0.125 | 0.2 | 8 | |CEL | 0.738 | 0.756 | 0.746 | 41 | |DIS | 0.737 | 0.772 | 0.754 | 759 | |EVE | 0.952 | 0.968 | 0.960 | 352 | |FOOD | 0.679 | 0.545 | 0.605 | 566 | |INST | 0.75 | 0.75 | 0.75 | 12 | |LOC | 0.994 | 0.991 | 0.993 | 12024 | |MEDIA | 0.940 | 0.969 | 0.954 | 458 | |ORG | 0.977 | 0.981 | 0.979 | 3309 | |PER | 0.992 | 0.995 | 0.993 | 5265 | |PLANT | 0.617 | 0.730 | 0.669 | 894 | |MYTH | 0.647 | 0.687 | 0.666 | 32 | |TIME | 0.825 | 0.820 | 0.822 | 289 | |VEHI | 0.812 | 0.812 | 0.812 | 32 | |**Overall** | **0.939** | **0.947** | **0.943** | ## Technical Specifications ### Model Architecture and Objective Follows the same as RoBERTa-BASE ### Compute Infrastructure #### Hardware Kaggle - GPU T4x2 Google Colab - GPU T4x1 #### Software pandas==1.5.3 numpy==1.23.5 seqeval==1.2.2 datasets==2.15.0 huggingface_hub==0.19.4 transformers[torch]==4.35.2 evaluate==0.4.1 matplotlib==3.7.1 collections torch==2.0.0 ## Model Card Contact [jayant-yadav](https://huggingface.co/jayant-yadav)