Spaces:
Build error
Build error
| tags: | |
| - spacy | |
| - token-classification | |
| language: | |
| - en | |
| model-index: | |
| - name: en_bib_references_trf | |
| results: | |
| - task: | |
| name: NER | |
| type: token-classification | |
| metrics: | |
| - name: NER Precision | |
| type: precision | |
| value: 0.9926182519 | |
| - name: NER Recall | |
| type: recall | |
| value: 0.9902421615 | |
| - name: NER F Score | |
| type: f_score | |
| value: 0.9914287831 | |
| - task: | |
| name: SENTS | |
| type: token-classification | |
| metrics: | |
| - name: Sentences F-Score | |
| type: f_score | |
| value: 0.9619008264 | |
| | Feature | Description | | |
| | --- | --- | | |
| | **Name** | `en_bib_references_trf` | | |
| | **Version** | `1.0.1` | | |
| | **spaCy** | `>=3.4.0,<3.5.0` | | |
| | **Default Pipeline** | `transformer`, `senter`, `ner`, `spancat` | | |
| | **Components** | `transformer`, `senter`, `ner`, `spancat` | | |
| | **Vectors** | 0 keys, 0 unique vectors (0 dimensions) | | |
| | **Sources** | n/a | | |
| | **License** | n/a | | |
| | **Author** | [Vitaly Davidenko]() | | |
| ### Problem to solve | |
| This pipeline parses list of bibliographic references. It is not required each reference to be on a separate line. | |
| 1. [SentenceRecognizer](https://spacy.io/api/sentencerecognizer) and [SpanCategorizer](https://spacy.io/api/spancategorizer) components are included into the pipeline to split up the bibliography section of a scientific paper into separate references. | |
| 2. [NER](https://spacy.io/api/entityrecognizer) in the pipeline annotates the reference structure | |
| ### Dataset | |
| The [distillroberta-base](https://huggingface.co/distilroberta-base) checkpoint has been fine-tuned on artificial data: bibliography sections were generated using [Citations Style Language](https://github.com/citation-style-language/styles) from 6000 [citeproc-json](https://citation.crosscite.org/docs.html) files [downloaded](https://github.com/vitaly-d/GIANT-The-1-Billion-Annotated-Synthetic-Bibliographic-Reference-String-Dataset/blob/master/dataset-creation/crossref/crossrefDownload.py) from [CrossRef](https://www.crossref.org). 95 selected styles were used to generate different representations of bibliography sections. | |
| This work is based on the ["GIANT: The 1-Billion Annotated Synthetic Bibliographic-Reference-String Dataset for Deep Citation Parsing"](https://www.semanticscholar.org/paper/GIANT%3A-The-1-Billion-Annotated-Synthetic-Dataset-Grennan-Schibel/8438d1497a01827aa278632f517d3af31fb6bc5a) paper with code in this [GitHub repo](https://github.com/BeelGroup/GIANT-The-1-Billion-Annotated-Synthetic-Bibliographic-Reference-String-Dataset). | |
| [Modifications](https://github.com/vitaly-d/GIANT-The-1-Billion-Annotated-Synthetic-Bibliographic-Reference-String-Dataset/tree/master/dataset-creation) required to extend this approach to support the bibliography section as well as [code](https://github.com/vitaly-d/GIANT-The-1-Billion-Annotated-Synthetic-Bibliographic-Reference-String-Dataset/tree/master/training-model/references) for training the SpaCy pipeline are in this [GitHub fork](https://github.com/vitaly-d/GIANT-The-1-Billion-Annotated-Synthetic-Bibliographic-Reference-String-Dataset). | |
| ``` | |
| =============================== Training stats =============================== | |
| Language: en | |
| Training pipeline: tok2vec, senter, ner, spancat | |
| 91178 training docs | |
| 1908 evaluation docs | |
| ``` | |
| ### Preprocessing | |
| Although the end-of-line character '\n' seems to give a useful signal to a model that splits up the bibliography section, it was also challenging to create a balanced artificial dataset with multiline references. Instead, it was decided to train the model on data that does not contain the line separator characters at all: | |
| ```python | |
| lines = io.StringIO(references) | |
| # normalization applied: strip lines and remove any extra space between lines | |
| norm_doc = nlp(" ".join([line.strip() for line in lines if line.strip()])) | |
| ``` | |
| ### Postprocessing | |
| If your data never contains more than one reference per line, you can use SpanCat scores to estimate whether the next line is the next reference or it is the next part of the current multiline reference. See [code](https://huggingface.co/spaces/vitaly/bibliography-parser/blob/main/app.py) for additional details | |
| ### Spaces App | |
| [Bibliography Parser](https://huggingface.co/spaces/vitaly/bibliography-parser) | |
| ### Label Scheme | |
| Essentially the pipeline is the tokens classification task. | |
| - NER Labels come from not overlapped CSL tags. | |
| - SentenceRecognizer:`Token.is_sent_start=1` is set for the first token of each reference. | |
| - SpanCategorizer: the 'bib' span is created for the first token of each reference. It is an alternative for SentenceRecognizer that returns scores. | |
| <details> | |
| <summary>View label scheme (13 labels for 2 components)</summary> | |
| | Component | Labels | | |
| | --- | --- | | |
| | **`ner`** | `citation-label`, `citation-number`, `container-title`, `doi`, `family`, `given`, `issued`, `page`, `publisher`, `title`, `url`, `volume` | | |
| | **`spancat`** | `bib` | | |
| </details> | |
| ### Accuracy | |
| | Type | Score | | |
| | --- | --- | | |
| | `SENTS_F` | 96.19 | | |
| | `SENTS_P` | 97.36 | | |
| | `SENTS_R` | 95.04 | | |
| | `ENTS_F` | 99.14 | | |
| | `ENTS_P` | 99.26 | | |
| | `ENTS_R` | 99.02 | | |
| | `SPANS_SC_F` | 98.47 | | |
| | `SPANS_SC_P` | 99.87 | | |
| | `SPANS_SC_R` | 97.10 | | |
| | `TRANSFORMER_LOSS` | 1042090.07 | | |
| | `SENTER_LOSS` | 1079996.00 | | |
| | `NER_LOSS` | 931993.00 | | |
| | `SPANCAT_LOSS` | 119923.94 | | |