F in NLP - FNP - LLMF in L egal @ COLING 2025 Shared Task: Agent-Based Single Cryptocurrency Trading Challenge (2025)

Abstract: Despite the promise of large language models based agent framework in stock trading task, their capabilities for comprehensive analysis and multiple different financial assets remain largely unexplored, such as cryptocurrency trading. To evaluate the capabilities of LLM-based agent framework in cryptocurrency trading, we introduce an LLMs-based financial shared task featured at COLING 2025 FinNLP-FNP-LLMFinLegal workshop, named Agent-based Single Cryptocurrency Trading Challenge. This challenge includes two cryptocurrencies: BitCoin and Ethereum. In this paper, we provide an overview of these tasks and datasets, summarize participants’ methods, and present their experimental evaluations, highlighting the effectiveness of LLMs in addressing cryptocurrency trading challenges. To the best of our knowledge, the Agent-based Single Cryptocurrency Trading Challenge is one of the first challenges for assessing LLMs in the financial area. In consequence, we provide detailed observations and take away conclusions for future development in this area.

## introduction large language models (llms) have showcased remarkable capabilities in text generation @xcite and reasoning @xcite @xcite across various domains, including healthcare @xcite and education @xcite . these advancements have sparked growing interest within the financial sector. recent research @xcite has highlighted the substantial potential of cutting-edge llms in financial q&a @xcite , financial text analysis @xcite , financial risk prediction @xcite , and financial forecasting @xcite . furthermore, significant research has begun to explore the utilization of llms as backbone models for developing agent frameworks to address complex financial decision-making tasks, such as asset trading @xcite and market simulation @xcite . for instance, finmem @xcite ) introduces a single-agent framework leveraging an llm to enhance trading performance by establishing a memory database to store historical trading experiences. similarly, stockagent @xcite simulates market dynamics by facilitating interactions among multiple agents. fincon @xcite incorporates a reflection mechanism through verbal reinforcement, improving risk management and extending its applicability to multi-asset trading tasks. despite the notable achievements of llm-based agent frameworks in stock trading, several critical aspects remain underexplored: 1) the predictive performance across diverse financial assets, such as cryptocurrencies, warrants further investigation; 2) the reliance on closed-source models in existing frameworks necessitates additional validation of open-source models to assess their effectiveness in these contexts. to further investigate the potential of llmbased agent frameworks for cryptocurrency trading under an open-source large language model setting, we introduce the agent-based single cryptocurrency trading challenge at coling 2025. this challenge focuses on two leading cryptocurrencies: bitcoin (btc) and ethereum (eth). for this task, we have curated open-source news datasets for btc and eth, enabling participants to evaluate the performance of various open-source models within the finmem @xcite framework. participants are also permitted to incorporate additional private datasets for pre-training or fine-tuning the backbone open-source llms. the goal is to optimize the generation of "buy", "sell", or "hold" decisions by the llms and achieve the possibly highest trading profits during the designated test period. this paper provides an overview of the performance of llm-based agent on cryptocurrency trading, as well as the datasets featured in the agent-based single cryptocurrency trading challenge. it summarizes participants' methodologies and evaluates their experimental results to investigate the capabilities of llms. our comprehensive evaluation highlights both the strengths and limitations of current approaches, demonstrating the effectiveness of llm-based agent frameworks in cryptocurrency trading. ## participants and results a total of 28 teams registered for the agent-based single cryptocurrency trading challenge, out of which 5 teams successfully submitted their models for evaluation. following the release of the 5 https://drive.google.com/drive/u/1/folders/ 1hg_ee-5nwsy8vda5emstqeae02w92-zs 6 https://drive.google.com/drive/u/1/folders/ 1fr0nbuhpj0bio_rukgpwa9skx4fj_fey leaderboard, three teams managed to outperform the buy-and-hold (b&h) baseline results, while two teams submitted detailed solution description papers. the rankings and performance of the participating teams are summarized in ## conclusions in this paper, the agent-based single cryptocurrency trading challenge has highlighted the efficacy and potential of llms in cryptocurrency trading. by providing a structured framework and extensive resources, the challenge has significantly contributed to advancing research in this domain. participants leveraged these resources to develop innovative strategies and models, leading to notable improvements in performance across various tasks. the experimental results from btc and eth underscore the considerable value of llmbased approaches, demonstrating their ability to navigate complex market dynamics effectively. a clear trend emerged, indicating that performance improves with increasing model size, as well as advancements in fine-tuning techniques and prompt engineering. these findings provide valuable insights for future research on financial tasks utilizing llms. moreover, the success of this challenge underscores the importance of collaborative efforts in driving forward the boundaries of ai applications in decentralized finance, offering promising directions for future innovations in the field.