T5 (مدل زبان)
آموزش دادن > مراجع

HS & PG
Yan
1 2   {{cite book}}: Empty citation (help)
↑  Raffel, Colin; Shazeer, Noam; Roberts, Adam; Lee, Katherine; Narang, Sharan; Matena, Michael; Zhou, Yanqi; Li, Wei; Liu, Peter J. (2020). "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer". Journal of Machine Learning Research. 21 (140): 1–67. arXiv:1910.10683. ISSN 1533-7928.
↑  Shaw, Peter; Uszkoreit, Jakob; Vaswani, Ashish (2018-04-12), Self-Attention with Relative Position Representations, arXiv:1803.02155
↑  "t5x/docs/models.md at main · google-research/t5x". GitHub (به انگلیسی). Retrieved 2024-08-05.
↑
↑
↑
↑
↑
↑
↑  Lester, Brian; Al-Rfou, Rami; Constant, Noah (2021-09-02), The Power of Scale for Parameter-Efficient Prompt Tuning, arXiv:2104.08691
↑  Fedus, William; Zoph, Barret; Shazeer, Noam (2022-06-16), Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, arXiv:2101.03961
↑  "SwitchTransformers". huggingface.co. Retrieved 2024-08-05.
↑  Sanh, Victor; Webson, Albert; Raffel, Colin; Bach, Stephen H.; Sutawika, Lintang; Alyafeai, Zaid; Chaffin, Antoine; Stiegler, Arnaud; Scao, Teven Le (2022-03-17), Multitask Prompted Training Enables Zero-Shot Task Generalization, arXiv:2110.08207
↑  "bigscience/T0 · Hugging Face". huggingface.co. 2024-03-04. Retrieved 2024-08-21.
↑  Xue, Linting; Barua, Aditya; Constant, Noah; Al-Rfou, Rami; Narang, Sharan; Kale, Mihir; Roberts, Adam; Raffel, Colin (2022-03-25). "ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models". Transactions of the Association for Computational Linguistics (به انگلیسی). 10: 291–306. arXiv:2105.13626. doi:10.1162/tacl_a_00461. ISSN 2307-387X.
↑  Chung, Hyung Won; Hou, Le; Longpre, Shayne; Zoph, Barret; Tay, Yi; Fedus, William; Li, Yunxuan; Wang, Xuezhi; Dehghani, Mostafa (2024). "Scaling Instruction-Finetuned Language Models". Journal of Machine Learning Research. 25 (70): 1–53. arXiv:2210.11416. ISSN 1533-7928.
↑  Longpre, Shayne; Hou, Le; Vu, Tu; Webson, Albert; Chung, Hyung Won; Tay, Yi; Zhou, Denny; Le, Quoc V.; Zoph, Barret (2023-07-03). "The Flan Collection: Designing Data and Methods for Effective Instruction Tuning". Proceedings of the 40th International Conference on Machine Learning (به انگلیسی). PMLR: 22631–22648. arXiv:2301.13688.
1 2  google-research/FLAN, Google Research, 2024-08-03, retrieved 2024-08-05
↑  "google/flan-t5-xl · Hugging Face". huggingface.co. 2024-01-04. Retrieved 2024-08-05.
↑  Roberts, Adam; Chung, Hyung Won; Mishra, Gaurav; Levskaya, Anselm; Bradbury, James; Andor, Daniel; Narang, Sharan; Lester, Brian; Gaffney, Colin (2023). "Scaling Up Models and Data with t5x and seqio". Journal of Machine Learning Research. 24 (377): 1–8. ISSN 1533-7928.
↑  google-research/text-to-text-transfer-transformer, Google Research, 2024-08-21, retrieved 2024-08-21
1 2  Tay, Yi; Dehghani, Mostafa; Tran, Vinh Q.; Garcia, Xavier; Wei, Jason; Wang, Xuezhi; Chung, Hyung Won; Shakeri, Siamak; Bahri, Dara (2023-02-28), UL2: Unifying Language Learning Paradigms, arXiv:2205.05131
↑  "Training great LLMs entirely from ground up in the wilderness as a startup". Yi Tay (به انگلیسی). Retrieved 2024-10-18.
↑  Sutawika, Lintang; Komatsuzaki, Aran; Raffel, Colin (2024-04-15). "Pile-T5". EleutherAI Blog (به انگلیسی). Retrieved 2024-05-05.
↑  "Imagen: Text-to-Image Diffusion Models". imagen.research.google. Retrieved 2024-08-23.
↑  "AuraFlow". huggingface.co. Retrieved 2024-08-23.
چگونه یک T5 را می توان برای یک کار خلاصه سازی به خوبی تنظیم کرد.
ساختار رمزگذار-رمزگشا T5، ساختار توجه را نشان می دهد. در خود توجهی رمزگذار (مربع پایین)، همه نشانه های ورودی به یکدیگر توجه می کنند. در توجه متقابل رمزگذار-رمزگشا (مستطیل بالایی)، هر نشانه هدف به تمام نشانه های ورودی توجه می کند. در خودتوجهی رمزگشا (مثلث بالایی)، هر نشانه هدف فقط به نشانه های هدف فعلی و گذشته توجه می کند (علت).