README.md

metadata

language: python
datasets:
  - fever
  - glue
  - tals/vitaminc
model-index:
  - name: tals/albert-base-vitaminc
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: tals/vitaminc
          type: tals/vitaminc
          config: tals--vitaminc
          split: test
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.4377049477326666
            verified: true
          - name: Precision Macro
            type: precision
            value: 0.3355703134284889
            verified: true
          - name: Precision Micro
            type: precision
            value: 0.4377049477326666
            verified: true
          - name: Precision Weighted
            type: precision
            value: 0.39076019579837573
            verified: true
          - name: Recall Macro
            type: recall
            value: 0.31910363117265955
            verified: true
          - name: Recall Micro
            type: recall
            value: 0.4377049477326666
            verified: true
          - name: Recall Weighted
            type: recall
            value: 0.4377049477326666
            verified: true
          - name: F1 Macro
            type: f1
            value: 0.2979001062785211
            verified: true
          - name: F1 Micro
            type: f1
            value: 0.43770494773266655
            verified: true
          - name: F1 Weighted
            type: f1
            value: 0.39742293746508284
            verified: true
          - name: loss
            type: loss
            value: 1.2437485456466675
            verified: true

Details

Model used in Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (Schuster et al., NAACL 21`).

For more details see: https://github.com/TalSchuster/VitaminC

When using this model, please cite the paper.

BibTeX entry and citation info

@inproceedings{schuster-etal-2021-get,
    title = "Get Your Vitamin {C}! Robust Fact Verification with Contrastive Evidence",
    author = "Schuster, Tal  and
      Fisch, Adam  and
      Barzilay, Regina",
    booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jun,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.naacl-main.52",
    doi = "10.18653/v1/2021.naacl-main.52",
    pages = "624--643",
    abstract = "Typical fact verification models use retrieved written evidence to verify claims. Evidence sources, however, often change over time as more information is gathered and revised. In order to adapt, models must be sensitive to subtle differences in supporting evidence. We present VitaminC, a benchmark infused with challenging cases that require fact verification models to discern and adjust to slight factual changes. We collect over 100,000 Wikipedia revisions that modify an underlying fact, and leverage these revisions, together with additional synthetically constructed ones, to create a total of over 400,000 claim-evidence pairs. Unlike previous resources, the examples in VitaminC are contrastive, i.e., they contain evidence pairs that are nearly identical in language and content, with the exception that one supports a given claim while the other does not. We show that training using this design increases robustness{---}improving accuracy by 10{\%} on adversarial fact verification and 6{\%} on adversarial natural language inference (NLI). Moreover, the structure of VitaminC leads us to define additional tasks for fact-checking resources: tagging relevant words in the evidence for verifying the claim, identifying factual revisions, and providing automatic edits via factually consistent text generation.",
}