File size: 5,135 Bytes
34c66a7 8710755 34c66a7 1b8365c 8710755 34c66a7 f04404f 6266163 ee98ef9 8710755 739b726 8710755 1b8365c 34c66a7 8710755 34c66a7 f04404f 34c66a7 11a0baf 9075b99 34c66a7 9075b99 34c66a7 f9247ce 8ead242 34c66a7 8710755 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 |
---
license: cc-by-nc-sa-4.0
tags:
- generated_from_trainer
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: req_mod_ner_modelv2
results: []
widget:
- text: >-
De Oplossing ondersteunt het zoeken op de metadata van zaken, documenten en
objecten en op gegevens uit de basisregistraties die gekoppeld zijn aan een
zaak.
- text: >-
De Oplossing ondersteunt parafering en het plaatsen van een gecertificeerde
elektronische handtekening.
- text: >-
De Aangeboden oplossing stelt de medewerker in staat een zaak te
registreren.
- text: >-
Het Financieel systeem heeft functionaliteit om een debiteurenadministratie
te voeren.
- text: >-
Als gebruiker wil ik dat de oplossing mij naar zaken laat zoeken op basis
van zaaknummer, zaaktitel, omschrijving en datum.
language:
- nl
---
# req_mod_ner_modelv2
This model is a fine-tuned version of [pdelobelle/robbert-v2-dutch-ner](https://huggingface.co/pdelobelle/robbert-v2-dutch-ner) on a
private dataset with 300 sentences/phrases with 1,954 token labels (IOB2 format) aimed at extracting software requirements
related named entities in Dutch. The following labels are used:
- Actor (used for all types of software users and groups of users)
- COTS (abbreviation for Commercial Off-The-Shelf Software)
- Function (used for functions, functionality, features)
- Result (used for system result, goals and system output)
- Entity (used for all entities stored/processed by the software)
- Attribute (used for attributes of entities)
Please contact me via [LinkedIn](https://www.linkedin.com/in/denizayhan/) if you have any questions about this model or the dataset used.
The dataset and this model were created as part of the final project assignment of the Natural Language Understanding course (XCS224U) from the Professional AI Program of the Stanford School of Engineering.
The model achieves the following results on the evaluation set:
- Loss: 0.6791
- Precision: 0.7515
- Recall: 0.7299
- F1: 0.7405
- Accuracy: 0.9253
# Metrics per named-entity
| NER-tag | Precision | Recall | F1 | Support |
|:---------:|:---------:|:------:|:----:|:-------:|
| Actor | 0.86 | 1.00 | 0.92 | 12 |
| COTS | 0.79 | 0.79 | 0.79 | 24 |
| Function | 0.73 | 0.66 | 0.69 | 62 |
| Result | 0.29 | 0.40 | 0.33 | 10 |
| Entity | 0.78 | 0.83 | 0.81 | 35 |
| Attribute | 0.92 | 0.71 | 0.80 | 31 |
## Intended uses & limitations
The model performs automated extraction of functionality concepts from source documents for which software requirements are needed. Its intended use is as a preceding processing step for Question-Answering.
## Training and evaluation data
The model was trained on the ReqModNer dataset. This dataset is private and contains 300 sentences/phrases and 1,954 IOB2 labels. The dataset is split 240/30/30 into train, validation and test. The reported metrics are from the evaluation on the test set. The validation set was used for cross-validation during training.
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 16
### Training results
| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1 | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
| No log | 1.0 | 270 | 0.5418 | 0.6065 | 0.5402 | 0.5714 | 0.8802 |
| 0.5551 | 2.0 | 540 | 0.4299 | 0.5481 | 0.6552 | 0.5969 | 0.8896 |
| 0.5551 | 3.0 | 810 | 0.4987 | 0.6358 | 0.5517 | 0.5908 | 0.9020 |
| 0.1935 | 4.0 | 1080 | 0.5620 | 0.6159 | 0.4885 | 0.5449 | 0.8935 |
| 0.1935 | 5.0 | 1350 | 0.4922 | 0.6786 | 0.6552 | 0.6667 | 0.9121 |
| 0.0913 | 6.0 | 1620 | 0.5406 | 0.6087 | 0.5632 | 0.5851 | 0.8950 |
| 0.0913 | 7.0 | 1890 | 0.6307 | 0.7425 | 0.7126 | 0.7273 | 0.9222 |
| 0.0702 | 8.0 | 2160 | 0.4425 | 0.6684 | 0.7414 | 0.7030 | 0.9277 |
| 0.0702 | 9.0 | 2430 | 0.6028 | 0.7158 | 0.7529 | 0.7339 | 0.9285 |
| 0.0472 | 10.0 | 2700 | 0.6491 | 0.7303 | 0.7471 | 0.7386 | 0.9246 |
| 0.0472 | 11.0 | 2970 | 0.6442 | 0.7198 | 0.7529 | 0.7360 | 0.9292 |
| 0.0305 | 12.0 | 3240 | 0.5980 | 0.7412 | 0.7241 | 0.7326 | 0.9230 |
| 0.0209 | 13.0 | 3510 | 0.6186 | 0.7232 | 0.7356 | 0.7293 | 0.9238 |
| 0.0209 | 14.0 | 3780 | 0.6791 | 0.7515 | 0.7299 | 0.7405 | 0.9253 |
| 0.0148 | 15.0 | 4050 | 0.6832 | 0.7283 | 0.7241 | 0.7262 | 0.9238 |
| 0.0148 | 16.0 | 4320 | 0.6908 | 0.7412 | 0.7241 | 0.7326 | 0.9238 |
### Framework versions
- Transformers 4.24.0
- Pytorch 2.0.0
- Datasets 2.9.0
- Tokenizers 0.11.0 |