LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

LUKE (Language Understanding with Knowledge-based Embeddings) is a new pre-trained contextualized representation of words and entities based on transformer. LUKE treats words and entities in a given text as independent tokens, and outputs contextualized representations of them. LUKE adopts an entity-aware self-attention mechanism that is an extension of the self-attention mechanism of the transformer, and considers the types of tokens (words or entities) when computing attention scores.

LUKE achieves state-of-the-art results on five popular NLP benchmarks including SQuAD v1.1 (extractive question answering), CoNLL-2003 (named entity recognition), ReCoRD (cloze-style question answering), TACRED (relation classification), and Open Entity (entity typing).

Please check the official repository for more details and updates.

This is the LUKE large model with 24 hidden layers, 1024 hidden size. The total number of parameters in this model is 483M. It is trained using December 2018 version of Wikipedia.

Experimental results

The experimental results are provided as follows:

Task Dataset Metric LUKE-large luke-base Previous SOTA
Extractive Question Answering SQuAD v1.1 EM/F1 90.2/95.4 86.1/92.3 89.9/95.1 (Yang et al., 2019)
Named Entity Recognition CoNLL-2003 F1 94.3 93.3 93.5 (Baevski et al., 2019)
Cloze-style Question Answering ReCoRD EM/F1 90.6/91.2 - 83.1/83.7 (Li et al., 2019)
Relation Classification TACRED F1 72.7 - 72.0 (Wang et al. , 2020)
Fine-grained Entity Typing Open Entity F1 78.2 - 77.6 (Wang et al. , 2020)

Citation

If you find LUKE useful for your work, please cite the following paper:

@inproceedings{yamada2020luke,
  title={LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention},
  author={Ikuya Yamada and Akari Asai and Hiroyuki Shindo and Hideaki Takeda and Yuji Matsumoto},
  booktitle={EMNLP},
  year={2020}
}
Downloads last month
1,344
Hosted inference API

Unable to determine this model’s pipeline type. Check the docs .