Towards Comprehensive Evaluation of Open-Source Language Models: A Multi-Dimensional, User-Driven Approach (2025)

Abstract: With rapid advancements in large language models (LLMs) across artificial intelligence, machine learning, and data sci-ence, there is a growing need for evaluation frameworks that go beyond traditional performance metrics. Conventional methods focus mainly on accuracy and computational metrics, often neglecting user experience and community interaction—key elements in open-source environments. This paper intro-duces a multi-dimensional, user-centered evaluation frame-work, integrating metrics like User Engagement Index (UEI), Community Response Rate (CRR), and a Time Weight Factor (TWF) to assess LLMs’ real-world impact. Additionally, we propose an adaptive weighting mechanism using Bayesian op-timization to dynamically adjust metric weights for more ac-curate model evaluation. Experimental results confirm that our framework effectively identifies models with strong user engagement and community support, offering a balanced, data-driven approach to open-source LLM evaluation. This frame-work serves as a valuable tool for developers and researchers in selecting and improving open-source models.

## introduction in recent years, large language models (llms) in the field of natural language processing (nlp) have achieved remarkable advancements, driving performance improvements across various applications such as machine translation, text generation, and automated question answering @xcite . since the introduction of gpt-3, open-source llms have continued to expand in scale and performance, drawing substantial interest from developers and researchers alike @xcite @xcite . as the number of models increases rapidly, selecting the most suitable llm among numerous options has become a critical challenge in practical applications. existing methods for evaluating llms primarily focus on performance testing, usually measuring accuracy or other technical metrics on standardized datasets @xcite . however, performance-based evaluations alone often fall short of comprehensively capturing a model's real-world application value. this is particularly true in open-source environments, where user experience and community engagement are increasingly recognized as key factors in evaluating a model's actual impact. in open-source communities, the practical value of llms depends not only on their technical performance but also on user feedback and community support and interaction. for example, user interaction data on platforms like hugging face foot_0 and github foot_1 -such as download counts, likes, issue reports, and pull requests-provides essential insights for evaluating models, reflecting the realworld demand for and user experience with these models. therefore, traditional evaluation methods that focus solely on performance metrics have significant limitations, as they fail to capture the full impact of open-source llms. based on this observation, this paper proposes a multi-dimensional, user-driven evaluation framework. by integrating metrics such as user engagement index (uei), community response rate (crr), and a time weight factor (twf), we aim to establish a more practically valuable framework for comprehensive llm evaluation. to enhance the flexibility and adaptability of the evaluation framework, this paper further introduces an adaptive weight optimization mechanism. since the impact of user interaction and community response may vary across different models, a fixed weight allocation is often inadequate for all models. therefore, we employ a bayesian optimization approach to automatically adjust the weights of each metric, ensuring that different models receive a fair and accurate evaluation across all evaluation dimensions. this adaptive weight optimization mechanism effectively improves the scientific rigor and representativeness of evaluation results, providing a more objective reference for model selection. additionally, this paper introduces a twf to address the balance in scoring between newer and older models. models released more recently may have limited accumulated user and community data, and traditional scoring methods often treat these models unfairly. the introduction of the twf reduces time-related bias in scoring to a certain extent, ensuring that evaluation results maintain a high level of fairness across models with different release dates. the main contributions of this paper include the following: the proposed evaluation framework not only offers a new perspective for evaluating open-source llms but also provides developers and researchers with a scientific reference for optimizing model design and enhancing user experience. we hope this study will offer valuable support for selecting, improving, and advancing open-source llms in the future. ## related works existing methods for evaluating llms primarily focus on standardized datasets, using metrics such as accuracy and f1 scores to gauge model performance on specific tasks @xcite @xcite . while these methods provide a direct reference for evaluating a model's technical performance, in real-world applications, user feedback and community interaction are equally important components of a model's overall impact. moreover, many models may be fine-tuned on particular datasets, potentially resulting in overfitting, which limits their ability to accurately reflect performance across diverse scenarios @xcite . in recent years, increasing research attention has been directed toward user experience and community support for models @xcite . in open-source projects, user interaction and community engagement are regarded as critical factors in measuring a project's value. metrics such as download counts and likes on the hugging face platform, as well as stars and issue reports on github, are increasingly used as indicators of a model's popularity and community activity level. however, most current evaluation frameworks are limited to singledimensional metrics of user or community engagement, lacking a comprehensive, multi-dimensional analysis. this paper constructs a multi-dimensional evaluation system based on user engagement, community response rate, and a time-weighting factor, complemented by an adaptive weight optimization method, to provide a more holistic, user-centered perspective for evaluating llms. ## conclusion this paper proposes a multi-dimensional evaluation framework for open-source llms, which uses a comprehensive assessment of metrics such as user engagement, community response rate, and timeweighted factors to reveal differences in model performance in real-world applications. based on data from the hugging face and github platforms, we validated the effectiveness of this evaluation system. experimental results show that user-oriented engagement and active community support have a significant impact on the final model scores. in this paper, we observed that models with high user engagement and active community support tend to receive higher final scores, which underscores the importance of user experience and community response in the open-source model ecosystem. however, some models performed poorly in user engagement and community interaction, indicating room for improvement in user-oriented optimization strategies. this evaluation framework not only provides a powerful tool for comprehensive model evaluation but also offers insights for developers and researchers to optimize their model design and user support strategies. future work will focus on expanding the evaluation metrics to cover different application scenarios of the models. additionally, to address the dynamic nature of platform data, future research can explore real-time updates and adaptive optimization methods for evaluation, thereby enhancing the timeliness and adaptability of the evaluation results.