Spaces:
Build error
Build error
File size: 5,303 Bytes
b525a4b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 |
---
tags:
- spacy
- token-classification
language:
- en
model-index:
- name: en_bib_references_trf
results:
- task:
name: NER
type: token-classification
metrics:
- name: NER Precision
type: precision
value: 0.9926182519
- name: NER Recall
type: recall
value: 0.9902421615
- name: NER F Score
type: f_score
value: 0.9914287831
- task:
name: SENTS
type: token-classification
metrics:
- name: Sentences F-Score
type: f_score
value: 0.9619008264
---
| Feature | Description |
| --- | --- |
| **Name** | `en_bib_references_trf` |
| **Version** | `1.0.1` |
| **spaCy** | `>=3.4.0,<3.5.0` |
| **Default Pipeline** | `transformer`, `senter`, `ner`, `spancat` |
| **Components** | `transformer`, `senter`, `ner`, `spancat` |
| **Vectors** | 0 keys, 0 unique vectors (0 dimensions) |
| **Sources** | n/a |
| **License** | n/a |
| **Author** | [Vitaly Davidenko]() |
### Problem to solve
This pipeline parses list of bibliographic references. It is not required each reference to be on a separate line.
1. [SentenceRecognizer](https://spacy.io/api/sentencerecognizer) and [SpanCategorizer](https://spacy.io/api/spancategorizer) components are included into the pipeline to split up the bibliography section of a scientific paper into separate references.
2. [NER](https://spacy.io/api/entityrecognizer) in the pipeline annotates the reference structure
### Dataset
The [distillroberta-base](https://huggingface.co/distilroberta-base) checkpoint has been fine-tuned on artificial data: bibliography sections were generated using [Citations Style Language](https://github.com/citation-style-language/styles) from 6000 [citeproc-json](https://citation.crosscite.org/docs.html) files [downloaded](https://github.com/vitaly-d/GIANT-The-1-Billion-Annotated-Synthetic-Bibliographic-Reference-String-Dataset/blob/master/dataset-creation/crossref/crossrefDownload.py) from [CrossRef](https://www.crossref.org). 95 selected styles were used to generate different representations of bibliography sections.
This work is based on the ["GIANT: The 1-Billion Annotated Synthetic Bibliographic-Reference-String Dataset for Deep Citation Parsing"](https://www.semanticscholar.org/paper/GIANT%3A-The-1-Billion-Annotated-Synthetic-Dataset-Grennan-Schibel/8438d1497a01827aa278632f517d3af31fb6bc5a) paper with code in this [GitHub repo](https://github.com/BeelGroup/GIANT-The-1-Billion-Annotated-Synthetic-Bibliographic-Reference-String-Dataset).
[Modifications](https://github.com/vitaly-d/GIANT-The-1-Billion-Annotated-Synthetic-Bibliographic-Reference-String-Dataset/tree/master/dataset-creation) required to extend this approach to support the bibliography section as well as [code](https://github.com/vitaly-d/GIANT-The-1-Billion-Annotated-Synthetic-Bibliographic-Reference-String-Dataset/tree/master/training-model/references) for training the SpaCy pipeline are in this [GitHub fork](https://github.com/vitaly-d/GIANT-The-1-Billion-Annotated-Synthetic-Bibliographic-Reference-String-Dataset).
```
=============================== Training stats ===============================
Language: en
Training pipeline: tok2vec, senter, ner, spancat
91178 training docs
1908 evaluation docs
```
### Preprocessing
Although the end-of-line character '\n' seems to give a useful signal to a model that splits up the bibliography section, it was also challenging to create a balanced artificial dataset with multiline references. Instead, it was decided to train the model on data that does not contain the line separator characters at all:
```python
lines = io.StringIO(references)
# normalization applied: strip lines and remove any extra space between lines
norm_doc = nlp(" ".join([line.strip() for line in lines if line.strip()]))
```
### Postprocessing
If your data never contains more than one reference per line, you can use SpanCat scores to estimate whether the next line is the next reference or it is the next part of the current multiline reference. See [code](https://huggingface.co/spaces/vitaly/bibliography-parser/blob/main/app.py) for additional details
### Spaces App
[Bibliography Parser](https://huggingface.co/spaces/vitaly/bibliography-parser)
### Label Scheme
Essentially the pipeline is the tokens classification task.
- NER Labels come from not overlapped CSL tags.
- SentenceRecognizer:`Token.is_sent_start=1` is set for the first token of each reference.
- SpanCategorizer: the 'bib' span is created for the first token of each reference. It is an alternative for SentenceRecognizer that returns scores.
<details>
<summary>View label scheme (13 labels for 2 components)</summary>
| Component | Labels |
| --- | --- |
| **`ner`** | `citation-label`, `citation-number`, `container-title`, `doi`, `family`, `given`, `issued`, `page`, `publisher`, `title`, `url`, `volume` |
| **`spancat`** | `bib` |
</details>
### Accuracy
| Type | Score |
| --- | --- |
| `SENTS_F` | 96.19 |
| `SENTS_P` | 97.36 |
| `SENTS_R` | 95.04 |
| `ENTS_F` | 99.14 |
| `ENTS_P` | 99.26 |
| `ENTS_R` | 99.02 |
| `SPANS_SC_F` | 98.47 |
| `SPANS_SC_P` | 99.87 |
| `SPANS_SC_R` | 97.10 |
| `TRANSFORMER_LOSS` | 1042090.07 |
| `SENTER_LOSS` | 1079996.00 |
| `NER_LOSS` | 931993.00 |
| `SPANCAT_LOSS` | 119923.94 |
|