BERT (מודל שפה)
הערות שוליים

↑  "Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing". Google AI Blog (באנגלית). 2 בנובמבר 2018. נבדק ב-2019-11-27. {{cite web}}:  (עזרה)
1 2 3  Rogers, Anna; Kovaleva, Olga; Rumshisky, Anna (2020). "A Primer in BERTology: What We Know About How BERT Works". Transactions of the Association for Computational Linguistics. 8: 842–866. arXiv:2002.12327. doi:10.1162/tacl_a_00349.
1 2  Ethayarajh, Kawin (2019-09-01), How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings, arXiv:1909.00512
↑  Anderson, Dawn (2019-11-05). "A deep dive into BERT: How BERT launched a rocket into natural language understanding". Search Engine Land (באנגלית). נבדק ב-2024-08-06.
↑  Zhu, Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books
1 2 3  "BERT". GitHub. נבדק ב-28 במרץ 2023. {{cite web}}:  (עזרה)
↑  Zhang, Tianyi; Wu, Felix; Katiyar, Arzoo; Weinberger, Kilian Q.; Artzi, Yoav (2021-03-11), Revisiting Few-sample BERT Fine-tuning, arXiv:2006.05987
↑  Turc, Iulia; Chang, Ming-Wei; Lee, Kenton; Toutanova, Kristina (2019-09-25), Well-Read Students Learn Better: On the Importance of Pre-training Compact Models, arXiv:1908.08962
↑  "Summary of the models — transformers 3.4.0 documentation". huggingface.co. נבדק ב-2023-02-16.
↑  Tay, Yi; Dehghani, Mostafa; Tran, Vinh Q.; Garcia, Xavier; Wei, Jason; Wang, Xuezhi; Chung, Hyung Won; Shakeri, Siamak; Bahri, Dara (2023-02-28), UL2: Unifying Language Learning Paradigms, arXiv:2205.05131
↑  Zhang, Aston; Lipton, Zachary; Li, Mu; Smola, Alexander J. (2024). "11.9. Large-Scale Pretraining with Transformers". Dive into deep learning. Cambridge New York Port Melbourne New Delhi Singapore: Cambridge University Press. ISBN 978-1-009-38943-3.
↑  Devlin, Jacob; Chang, Ming-Wei; Lee, Kenton; Toutanova, Kristina (11 באוקטובר 2018). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". arXiv:1810.04805v2 [cs.CL]. {{cite arxiv}}:  (עזרה)
1 2 3  Clark, Kevin; Khandelwal, Urvashi; Levy, Omer; Manning, Christopher D. (2019). "What Does BERT Look at? An Analysis of BERT's Attention". Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Stroudsburg, PA, USA: Association for Computational Linguistics: 276–286. arXiv:1906.04341. doi:10.18653/v1/w19-4828.
1 2  Kovaleva, Olga; Romanov, Alexey; Rogers, Anna; Rumshisky, Anna (בנובמבר 2019). "Revealing the Dark Secrets of BERT". Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (באנגלית אמריקאית). pp. 4364–4373. doi:10.18653/v1/D19-1445. {{cite book}}:  (עזרה)
↑  Khandelwal, Urvashi; He, He; Qi, Peng; Jurafsky, Dan (2018). "Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context". Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, PA, USA: Association for Computational Linguistics: 284–294. arXiv:1805.04623. doi:10.18653/v1/p18-1027.
↑  Gulordava, Kristina; Bojanowski, Piotr; Grave, Edouard; Linzen, Tal; Baroni, Marco (2018). "Colorless Green Recurrent Networks Dream Hierarchically". Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Stroudsburg, PA, USA: Association for Computational Linguistics. pp. 1195–1205. doi:10.18653/v1/n18-1108.
↑  Giulianelli, Mario; Harding, Jack; Mohnert, Florian; Hupkes, Dieuwke; Zuidema, Willem (2018). "Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information". Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Stroudsburg, PA, USA: Association for Computational Linguistics: 240–248. arXiv:1808.08079. doi:10.18653/v1/w18-5426.
↑  Zhang, Kelly; Bowman, Samuel (2018). "Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Syntactic Task Analysis". Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Stroudsburg, PA, USA: Association for Computational Linguistics: 359–361. doi:10.18653/v1/w18-5448.
↑  Patel. "Bidirectional Language Models Are Also Few-shot Learners". arXiv:2209.14500.
↑  Nayak, Pandu (25 באוקטובר 2019). "Understanding searches better than ever before". Google Blog. נבדק ב-10 בדצמבר 2019. {{cite web}}:  (עזרה)
↑  "Understanding searches better than ever before". Google (באנגלית אמריקאית). 2019-10-25. נבדק ב-2024-08-06.
↑  Montti, Roger (10 בדצמבר 2019). "Google's BERT Rolls Out Worldwide". Search Engine Journal. נבדק ב-10 בדצמבר 2019. {{cite web}}:  (עזרה)
↑  "Google: BERT now used on almost every English query". Search Engine Land. 2020-10-15. נבדק ב-2020-11-24.
↑  Liu. "RoBERTa: A Robustly Optimized BERT Pretraining Approach". arXiv:1907.11692.
↑  Sanh, Victor; Debut, Lysandre; Chaumond, Julien; Wolf, Thomas (2020-02-29), DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, arXiv:1910.01108
↑  "DistilBERT". huggingface.co. נבדק ב-2024-08-05.
↑  Jiao, Xiaoqi; Yin, Yichun; Shang, Lifeng; Jiang, Xin; Chen, Xiao; Li, Linlin; Wang, Fang; Liu, Qun (2020-10-15), TinyBERT: Distilling BERT for Natural Language Understanding, arXiv:1909.10351
↑  Lan, Zhenzhong; Chen, Mingda; Goodman, Sebastian; Gimpel, Kevin; Sharma, Piyush; Soricut, Radu (2020-02-08), ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, arXiv:1909.11942
↑  Clark, Kevin; Luong, Minh-Thang; Le, Quoc V.; Manning, Christopher D. (2020-03-23), ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators, arXiv:2003.10555
↑  He, Pengcheng; Liu, Xiaodong; Gao, Jianfeng; Chen, Weizhu (2021-10-06), DeBERTa: Decoding-enhanced BERT with Disentangled Attention, arXiv:2006.03654
דיאגרמה סכמטית ברמה גבוהה של BERT. הוא קולט טקסט, מסמל אותו לרצף של טוקנים, מוסיף טוקנים מיוחדים אופציונליים ומחיל מקודד טרנספורמר. ניתן להשתמש במצבים המוסתרים של השכבה האחרונה כהטמעות מילים בהקשר.
תשומת הלב של המקודד בלבד היא הכל לכל.
שלושת סוגי ההטמעה המשמשים את BERT: סוג טוקן, מיקום וסוג מקטע.
משימת דוגמנות השפה במסכה.
משימת חיזוי המשפט הבא.