T el B ench: A Benchmark for Evaluating Telco-Specific Large Language Models (2024)

Abstract: The telecommunications industry, characterized by its vast customer base and complex service offerings, necessitates a high level of domain expertise and proficiency in customer service center operations. Consequently, there is a growing demand for Large Language Models (LLMs) to augment the capabilities of customer service representatives. This paper introduces a methodology for developing a specialized Telecommunications LLM (Telco LLM) designed to enhance the efficiency of customer service agents and promote consistency in service quality across representatives. We present the construction process of TelBench, a novel dataset created for performance evaluation of customer service expertise in the telecommunications domain. We also evaluate various LLMs and demonstrate the ability to benchmark both proprietary and open-source LLMs on predefined telecommunications-related tasks, thereby establishing metrics that define telcommunications performance.

## introduction recent advancements in large language models (llms) have significantly enhanced natural language understanding and generation capabilities, leading to increased development of domainspecific llms across various sectors, including law, finance, and science. these specialized models aim to leverage llms' general linguistic proficiency while incorporating domain-specific knowledge @xcite @xcite . the telecommunications (telco) industry is characterized by its large subscriber base, complex network infrastructure, diverse service offerings, and 24-hour global connectivity. this complexity results in a wide variety of customer inquiries, requiring extensive training for customer service representatives. to enhance customer satisfaction in call center interactions, we leveraged llms to augment the expertise of customer service representatives and reduce response times. our approach improved service efficiency by enabling llms to perform postinteraction tasks that previously required manual searching, reasoning, and documentation. this paper's primary contributions are: this paper is structured as follows: section 2 reviews related research, section 3 details the dataset composition and development methodology, section 4 presents llm evaluation results on the dataset, and section 5 summarizes findings and proposes future research directions. ## related work evaluating large language models (llms) for domain-specific knowledge and task performance is crucial when considering their deployment as a service. to address this, domain-specific datasets have emerged across various fields, including medicine @xcite @xcite , science @xcite , law(guha et al., 2023), finance @xcite , education @xcite , and coding @xcite . recently, this trend also has extended to telecommunications network infrastructure @xcite . all of these tailored datasets highlight the growing importance of adapting llms to specialized domains. with the rapid advancement of llms' instruction-following capabilities, novel evaluation datasets have been developed to assess agentic behavior @xcite and responses to toxic language @xcite . evaluation approaches now integrate conventional automatic evaluation frameworks @xcite with llms-based, reference-free approaches @xcite , enhancing the effectiveness and diversity of evaluating llms. ## dataset construction and development methodology customer interactions with contact center agents involve issue comprehension, information retrieval, problem resolution, and post-call activities (figure 1 ). these interactions often span multiple encounters with different agents, emphasizing the importance of the post-work phase. large language models (llms) can enhance this phase by improving accuracy, reducing time spent, and standard-izing practices, ultimately reducing human agent workload and improving service levels. to support telco customer service with llms, we defined essential tasks and categorized them into two groups: teltask and telinstruct. teltask is a comprehensive and involves contextual language understanding capability in conversations for post-interaction tasks, while telinstruct is a benchmark set that assesses telco domain knowledge and instruction following capability. sample data is available in appendix a. two common pre-processing stages are applied to each dataset, followed by additional datasetspecific processing: detailed explanations of each task and associated development methodologies are provided in subsequent sections. ## evaluation of llms this section presents the evaluation methodology and results for various large language models (llms) to validate the utility of the telbench benchmark set. ## conclusion this paper introduces telbench, the first (to our knowledge) benchmark dataset focused on telco customer service centers, and evaluates the performance of large language models (llms) using the dataset we designed and built. leveraging proprietary assets and domain expertise, we have created a benchmark dataset to measure the telcospecific performance of various llms, both proprietary and open source. the methodology employed in developing tel-bench can be extended to create specialized training datasets for llms in the telco sector, and such datasets can help facilitate the development of llms optimized for telco-specific tasks. future research will focus on the development and performance evaluation of these specialized telco llms. additionally, we plan to expand the scope of telbench to include other areas of the telco industry, such as infrastructure operations, task planning, and contract reviews. furthermore, we are preparing further llm-based evaluation methods to reduce the burden of human assessment and also developing an evaluation-as-a-service platform.