T ele C hat: An Open-source Billingual Large Language Model (2024)

Abstract: In this paper, we present TeleChat , a collection of large language models (LLMs) with parameters of 7 billion and 12 billion. TeleChat is initially pretrained on an extensive corpus containing a diverse collection of texts from both English and Chinese languages, encompassing trillions of tokens. Subsequently, the model undergoes fine-tuning to align with human preferences, following a detailed methodology that we describe. We evaluate the performance of TeleChat on various tasks, including general dialogue generation, language understanding, mathematics, reasoning, code generation, and knowledge-based question answering. Our findings indicate that TeleChat achieves state-of-the-art performance to other open-source models of similar size across a wide range of public benchmarks. To support future research and applications utilizing LLMs, we release the fine-tuned model checkpoints of TeleChat-7B and TeleChat-12B, along with code and a portion of our filtered high-quality pretraining data, to the public community.

## introduction the research community has witnessed substantial proliferation of open large language models (llms). following the introduction of chat-gpt(openai, 2022), there have been thrilling advancements and applications of llms, but the majority of prominent llms, such as @xcite and @xcite , are restrictive in their technological sharing. in contrast, a steady stream of openly accessible text-based llms has emerged, including opt @xcite , bloom @xcite , llama @xcite , llama 2 @xcite , etc. furthermore, there exist various other llms that have been designed with a focus on chinese-english bilingual language generation, including models such as baichuan-2 @xcite , @xcite , internlm @xcite and skywork @xcite . while these models offer comprehensive details about their pretraining strategies, they often lack transparency in their instruction finetuning processes for chat models, including limited disclosure of the finetuning data composition, methods for concatenating multiturn dialog data, and techniques employed to enhance conversational performance. to encourage reproducibility of fine-tuned llms and foster responsible development of llms, we release telechat, a collection of pretrained language models and chat models that have been fine-tuned using human alignment techniques including supervised fine-tuning and reinforcement learning. in particular, we provide a comprehensive explanation of our model architecture and the approach we used to extend telechat's context window to 96k in section 2. furthermore, in section 3, we delve into the specifics of our pretraining dataset and cleaning techniques. we then discuss alignment with human preferences in section 4 and 5. additionally, in section 6, we conduct a thorough analysis of the model's performance on standard benchmark tasks and general dialogue generation. throughout the development of telechat, we gain insights regarding mitigating hallucination with a knowledge graph, which is discussed in section 7. furthermore, we describe our parallel computing method in section 8. our contribution are listed as follows: ## pretraining stage during pretraining stage, we train the model from scratch using a substantial amount of data. in this section, we introduce our data collection and cleaning method (section 3.1 and 3.2), training details (section 3.3), and tokenizer (section 3.4). ## supervised fine-tuning stage we employ supervised fine-tuning (sft) stage after the pretraining stage to effectively accomplish various real-world tasks. in this section, we provide detailed information about our data collection and annotation method in section 4.1, followed by an in-depth discussion of our methodology and experimental details in section 4.2 and section 4.3. ## reinforcement learning in this section, we introduce reinforcement learning to align chat models with human preference, aiming to make model outputs consistent with safety and norms. ## experiment 6.1 evaluation on standard benchmarks in this chapter, we evaluate the zero-shot and fewshot capabilities of telechat from various perspectives using standard benchmarks. we select a list of open source models as baselines, including llama 2-chat (7b, 13b), internlm-chat (7b), belle-llama-2 (13b), baichuan 2 (7b, 13b), chatglm 2-6b, chatglm 3-6b, qwen-chat (7b, 14b). type of data good & bad medium & bad good & medium distribution 18.2% 21.1% 65.7% margin 1 2/3 1/3 test accuracy 70.1% 66.0% 86.4% ## alleviating hallucination with knowledge graph hallucination problems are frequently observed in llms, where there is a tendency to generate text that appears coherent and meaningful but lacks real-world existence. in this paper, we address the first type of hallucinations by utilizing structured information representation provided by knowledge graphs (kg). when a query comes, candidate entities are firstly retrieved based on n-gram similarity with query. subsequently, a random walk of n steps is conducted within the graph, starting from these candidate entities. finally, all paths obtained through the random walk are sorted based on their relevance to the user's query. the top-k paths are then returned as the final result of the knowledge graph retrieval process. by combining this retrieved knowledge with a prompt, the large language model can process the augmented query, taking into consideration the background knowledge provided by the knowledge graph. we evaluated the telechat's ability to answer factual questions in the china conference on knowledge graph and semantic computing (ccks) 2020 knowledge graph based q&a task foot_1 . without the introduction of the knowl-edge graph, the accuracy of telechat on this task is recorded at 0.19. however, after incorporating the relevant knowledge by adding the top 10 relevant paths from the knowledge graph, the accuracy significantly improves to 0.69. this demonstrates the effectiveness of integrating the knowledge graph in enhancing the telechat's ability to provide accurate answers to factual questions. ## engineering 8.1 hardware telechat is trained on a total of 80 nodes, each having 8 nvidia a100 sxm 40gb gpus. each node is equipped with 2x intel 6348 (28 cores, 2.60 ghz) cpus, 8x nvlink a100 gpus, 512gb of ram, and a 2gb cache raid card. all nodes are interconnected using infiniband (ib) for networking. to enhance data transmission speed and mitigate bandwidth constraints, we employ nvidia's gpudirect rdma (grdma) and utilize the scalable hierarchical aggregation and reduction protocol (sharp). ## conclusions in this paper, we introduced telechat, a collection of large language models (llms) with 7 billion and 12 billion parameters. we detailed the pretraining process, supervised fine-tuning, reinforcement learning, and the integration of a knowledge graph to enhance the model's performance. we evaluated telechat on various benchmarks and compared its performance with other open-source models, telechat demonstrates superior performance in general dialogue tasks, knowledge-based question answering, and various other benchmarks, showcasing its potential for diverse real-world applications. we release model checkpoints, code, and a portion of our filtered high-quality pretraining data totaling 160 billion tokens to the public community.