大型语言模型列表

大型语言模型 (LLM) 是一种机器学习模型，专为语言生成等自然语言处理任务而设计。LLM 是具有许多参数的语言模型，并通过对大量文本进行自监督学习进行训练。
本页列出了值得注意的大型语言模型。
对于训练成本一列，1 petaFLOP-day = 1 petaFLOP/sec × 1 天 = 8.64×1019 FLOP。此外，仅列出最大模型的成本。
| 名称 | 发布日期 | 开发者 | 参数量 (十亿)  | 语料库大小 | 训练成本 (petaFLOP-day) | 许可证 | 注解 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-1 | 000000002018-06-01-00002018年6月 | OpenAI | 0.117 !0.117 |  | 1 | MIT | 首个GPT模型，为仅解码器transformer。 在8个P600GPU上训练了30天。 |
| BERT | 000000002018-10-01-00002018年10月 | Google | 0.340 !0.340 | 3300000000 !33亿单词 | 9 !9 | Apache 2.0 | 这是一个早期且有影响力的语言模型。 仅用于编码器，因此并非为提示或生成而构建。 在 64 个 TPUv2 芯片上训练耗时 4 天。 |
| T5 | 000000002019-10-01-00002019年10月 | Google | 11 !11 | 340亿 tokens |  | Apache 2.0 | 许多Google项目的基础模型，例如Imagen。 |
| XLNet | 000000002019-06-01-00002019年6月 | Google | 0.340 !0.340 | 3300000000 !330亿单词 | 330 | Apache 2.0 | 作为BERT的替代，设计为仅编码器 。在512个TPU v3芯片上训练了5.5天。 |
| GPT-2 | 000000002019-02-01-00002019年2月 | OpenAI | 1.5 !1.5 | 40 GB (~10000000000 !100亿 tokens) | 28 | MIT | 在32个TPU v3芯片上训练了一周。 |
| GPT-3 | 000000002020-05-01-00002020年5月 | OpenAI | 175 !175 | 300000000000 !3000亿 tokens | 3640 | Proprietary | A fine-tuned variant of GPT-3, termed GPT-3.5, was made available to the public through a web interface called ChatGPT in 2022. |
| GPT-Neo | 000000002021-03-01-00002021年3月 | EleutherAI | 2.7 !2.7 | 825 GiB |  | MIT | The first of a series of free GPT-3 alternatives released by EleutherAI. GPT-Neo outperformed an equivalent-size GPT-3 model on some benchmarks, but was significantly worse than the largest GPT-3. |
| GPT-J | 000000002021-06-01-00002021年6月 | EleutherAI | 6 !6 | 825 GiB | 200 | Apache 2.0 | GPT-3-style language model |
| Megatron-Turing NLG | 000000002021-10-01-00002021年10月  | Microsoft and Nvidia | 530 !530 | 338600000000 !338.6 billion tokens | 38000 | Restricted web access | Trained for 3 months on over 2000 A100 GPUs on the NVIDIA Selene Supercomputer, for over 3 million GPU-hours. |
| Ernie 3.0 Titan | 000000002021-12-01-00002021年12月 | Baidu | 260 !260 | 4 Tb |  | Proprietary | Chinese-language LLM. Ernie Bot is based on this model. |
| Claude | 000000002021-12-01-00002021年12月 | Anthropic | 52 !52 | 400000000000 !400 billion tokens |  | beta | Fine-tuned for desirable behavior in conversations. |
| GLaM (Generalist Language Model) | 000000002021-12-01-00002021年12月 | Google | 1200 !1200 | 1600000000000 !1.6 trillion tokens | 5600 | Proprietary | Sparse mixture of experts model, making it more expensive to train but cheaper to run inference compared to GPT-3. |
| Gopher | 000000002021-12-01-00002021年12月 | DeepMind | 280 !280 | 300000000000 !300 billion tokens | 5833 | Proprietary | Later developed into the Chinchilla model. |
| LaMDA (Language Models for Dialog Applications) | 000000002022-01-01-00002022年1月 | Google | 137 !137 | 1.56T words, 168000000000 !168 billion tokens | 4110 | Proprietary | Specialized for response generation in conversations. |
| GPT-NeoX | 000000002022-02-01-00002022年2月 | EleutherAI | 20 !20 | 825 GiB | 740 | Apache 2.0 | based on the Megatron architecture |
| Chinchilla | 000000002022-03-01-00002022年3月 | DeepMind | 70 !70 | 1400000000000 !1.4 trillion tokens | 6805 | Proprietary | Reduced-parameter model trained on more data. Used in the Sparrow bot. Often cited for its neural scaling law. |
| PaLM (Pathways Language Model) | 000000002022-04-01-00002022年4月 | Google | 540 !540 | 768000000000 !768 billion tokens | 29250 !29,250 | Proprietary | Trained for ~60 days on ~6000 TPU v4 chips. 截至2024年10月 (2024-10), it is the largest dense Transformer published. |
| OPT (Open Pretrained Transformer) | 000000002022-05-01-00002022年5月 | Meta | 175 !175 | 180000000000 !180 billion tokens | 310 | Non-commercial research | GPT-3 architecture with some adaptations from Megatron. Uniquely, the training logbook written by the team was published. |
| YaLM 100B | 000000002022-06-01-00002022年6月 | Yandex | 100 !100 | 1.7TB |  | Apache 2.0 | English-Russian model based on Microsoft's Megatron-LM. |
| Minerva | 000000002022-06-01-00002022年6月 | Google | 540 !540 | 38.5B tokens from webpages filtered for mathematical content and from papers submitted to the arXiv preprint server |  | Proprietary | For solving "mathematical and scientific questions using step-by-step reasoning". Initialized from PaLM models, then finetuned on mathematical and scientific data. |
| BLOOM | 000000002022-07-01-00002022年7月 | Large collaboration led by Hugging Face | 175 !175 | 350000000000 !350 billion tokens (1.6TB) |  | Responsible AI | Essentially GPT-3 but trained on a multi-lingual corpus (30% English excluding programming languages) |
| Galactica | 000000002022-11-01-00002022年11月 | Meta | 120 !120 | 350000000000 !106 billion tokens | 未知 | CC-BY-NC-4.0 | Trained on scientific text and modalities. |
| AlexaTM (Teacher Models) | 000000002022-11-01-00002022年11月 | Amazon | 20 !20 | 1300000000000 !1.3 trillion |  | proprietary | bidirectional sequence-to-sequence architecture |
| LLaMA (Large Language Model Meta AI) | 000000002023-02-01-00002023年2月 | Meta AI | 65 !65 | 1400000000000 !1.4 trillion | 6300 | Non-commercial research | Corpus has 20 languages. "Overtrained" (compared to Chinchilla scaling law) for better performance with fewer parameters. |
| GPT-4 | 000000002023-03-01-00002023年3月 | OpenAI | 未知 (According to rumors: 1760) | 未知 | 未知 | proprietary | Available for ChatGPT Plus users and used in several products. |
| Chameleon | 000000002024-06-01-00002024年6月 | Meta AI | 34 !34 | 4400000000000 !4.4 trillion |  |  |  |
| Cerebras-GPT | 000000002023-03-01-00002023年3月 | Cerebras | 13 !13 |  | 270 | Apache 2.0 | Trained with Chinchilla formula. |
| Falcon | 000000002023-03-01-00002023年3月 | Technology Innovation Institute | 40 !40 | 1 trillion tokens, from RefinedWeb (filtered web text corpus) plus some "curated corpora". | 2800 | Apache 2.0 |  |
| BloombergGPT | 000000002023-03-01-00002023年3月 | Bloomberg L.P. | 50 !50 | 363 billion token dataset based on Bloomberg's data sources, plus 345 billion tokens from general purpose datasets |  | Proprietary | Trained on financial data from proprietary sources, for financial tasks. |
| PanGu-Σ | 000000002023-03-01-00002023年3月 | Huawei | 1085 !1085 | 329 billion tokens |  | Proprietary |  |
| OpenAssistant | 000000002023-03-01-00002023年3月 | LAION | 17 !17 | 1.5 trillion tokens |  | Apache 2.0 | Trained on crowdsourced open data |
| Jurassic-2 | 000000002023-03-01-00002023年3月 | AI21 Labs | 未知 | 未知 |  | Proprietary | Multilingual |
| PaLM 2 (Pathways Language Model 2) | 000000002023-05-01-00002023年5月 | Google | 340 !340 | 3600000000000 !3.6 trillion tokens | 85000 !85,000 | Proprietary | Was used in Bard chatbot. |
| Llama 2 | 000000002023-07-01-00002023年7月 | Meta AI | 70 !70 | 2000000000000 !2 trillion tokens | 21000 !21,000 | Llama 2 license | 1.7 million A100-hours. |
| Claude 2 | 000000002023-07-01-00002023年7月 | Anthropic | 未知 | 未知 | 未知 | Proprietary | Used in Claude chatbot. |
| Granite 13b | 000000002023-07-01-00002023年7月 | IBM | 未知 | 未知 | 未知 | Proprietary | Used in IBM Watsonx. |
| Mistral 7B | 000000002023-09-01-00002023年9月 | Mistral AI | 7.3 !7.3 | 未知 |  | Apache 2.0 |  |
| Claude 2.1 | 000000002023-11-01-00002023年11月 | Anthropic | 未知 | 未知 | 未知 | Proprietary | Used in Claude chatbot. Has a context window of 200,000 tokens, or ~500 pages. |
| Grok-1 | 000000002023-11-01-00002023年11月 | xAI | 314 | 未知 | 未知 | Apache 2.0 | Used in Grok chatbot. Grok-1 has a context length of 8,192 tokens and has access to X (Twitter). |
| Gemini 1.0 | 000000002023-12-01-00002023年12月 | Google DeepMind | 未知 | 未知 | 未知 | Proprietary | Multimodal model, comes in three sizes. Used in the chatbot of the same name. |
| Mixtral 8x7B | 000000002023-12-01-00002023年12月 | Mistral AI | 46.7 | 未知 | 未知 | Apache 2.0 | Outperforms GPT-3.5 and Llama 2 70B on many benchmarks. Mixture of experts model, with 12.9 billion parameters activated per token. |
| Mixtral 8x22B | 000000002024-04-01-00002024年4月 | Mistral AI | 141 | 未知 | 未知 | Apache 2.0 |  |
| DeepSeek LLM | 000000002023-11-29-00002023年11月29日 | DeepSeek | 67 | 2T tokens | 12,000}} | DeepSeek License | Trained on English and Chinese text. 1e24 FLOPs for 67B. 1e23 FLOPs for 7B |
| Phi-2 | 000000002023-12-01-00002023年12月 | Microsoft | 2.7 | 1.4T tokens | 419 | MIT | Trained on real and synthetic "textbook-quality" data, for 14 days on 96 A100 GPUs. |
| Gemini 1.5 | 000000002024-02-01-00002024年2月 | Google DeepMind | 未知 | 未知 | 未知 | Proprietary | Multimodal model, based on a Mixture-of-Experts (MoE) architecture. Context window above 1 million tokens. |
| Gemini Ultra | 000000002024-02-01-00002024年2月 | Google DeepMind | 未知 | 未知 | 未知 |  |  |
| Gemma | 000000002024-02-01-00002024年2月 | Google DeepMind | 7 | 6T tokens | 未知 | Gemma Terms of Use |  |
| Claude 3 | 000000002024-03-01-00002024年3月 | Anthropic | 未知 | 未知 | 未知 | Proprietary | Includes three models, Haiku, Sonnet, and Opus. |
| Nova | 000000002024-10-01-00002024年10月 | Rubik's AI | 未知 | 未知 | 未知 | Proprietary | Includes three models, Nova-Instant, Nova-Air, and Nova-Pro. |
| DBRX | 000000002024-03-01-00002024年3月 | Databricks and Mosaic ML | 136 !136 | 12T Tokens |  | Databricks Open Model License | Training cost 10 million USD. |
| Fugaku-LLM | 000000002024-05-01-00002024年5月 | Fujitsu, Tokyo Institute of Technology, etc. | 13 !13 | 380B Tokens |  |  | The largest model ever trained on CPU-only, on the Fugaku. |
| Phi-3 | 000000002024-04-01-00002024年4月 | Microsoft | 14 | 4.8T Tokens |  | MIT | Microsoft markets them as "small language model". |
| Granite Code Models | 000000002024-05-01-00002024年5月 | IBM | 未知 | 未知 | 未知 | Apache 2.0 |  |
| Qwen2 | 000000002024-06-01-00002024年6月 | Alibaba Cloud | 72 | 3T Tokens | 未知 | Qwen License | Multiple sizes, the smallest being 0.5B. |
| DeepSeek V2 | 000000002024-06-01-00002024年6月 | DeepSeek | 236 | 8.1T tokens | 28000 !28,000 | DeepSeek License | 1.4M hours on H800. |
| Nemotron-4 | 000000002024-06-01-00002024年6月 | Nvidia | 340 !340 | 9T Tokens | 200000 !200,000 | NVIDIA Open Model License | Trained for 1 epoch. Trained on 6144 H100 GPUs between December 2023 and May 2024. |
| Llama 3.1 | 000000002024-07-01-00002024年7月 | Meta AI | 405 | 15.6T tokens | 440000 !440,000 | Llama 3 license | 405B version took 31 million hours on H100-80GB, at 3.8E25 FLOPs. |
| DeepSeek V3 | 000000002024-12-01-00002024年12月 | DeepSeek | 671 | 14.8T tokens | 56000 !56,000 | DeepSeek License | 2.788M hours on H800 GPUs. |
| Amazon Nova | 000000002024-12-01-00002024年12月 | Amazon | 未知 | 未知 | 未知 | Proprietary | Includes three models, Nova Micro, Nova Lite, and Nova Pro |
| DeepSeek R1 | 000000002025-01-01-00002025年1月 | DeepSeek | 671 | 未知 | 未知 | MIT | No pretraining. Reinforcement-learned upon V3-Base. |
| Qwen2.5 | 000000002025-01-01-00002025年1月 | Alibaba | 72 | 18T tokens | 未知 | Qwen License |  |
| MiniMax-Text-01 | January 2025 | Minimax | 456 | 4.7T tokens | 未知 | Minimax Model license |  |
| Gemini 2.0 | 000000002025-02-01-00002025年2月 | Google DeepMind | 未知 | 未知 | 未知 | Proprietary | Three models released: Flash, Flash-Lite and Pro |