C ipher B ank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challenge (2025)

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities, especially the recent advancements in reasoning, such as o1 and o3, pushing the boundaries of AI. Despite these impressive achievements in mathematics and coding, the reasoning abilities of LLMs in domains requiring cryptographic expertise remain underexplored. In this paper, we introduce CipherBank, a comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs in cryptographic decryption tasks. CipherBank comprises 2,358 meticulously crafted problems, covering 262 unique plaintexts across 5 domains and 14 subdomains, with a focus on privacy-sensitive and real-world scenarios that necessitate encryption. From a cryptographic perspective, CipherBank incorporates 3 major categories of encryption methods, spanning 9 distinct algorithms , ranging from classical ciphers to custom cryptographic techniques. We evaluate state-of-the-art LLMs on CipherBank, e.g., GPT-4o, DeepSeek-V3, and cutting-edge reasoning-focused models such as o1 and DeepSeek-R1. Our results reveal significant gaps in reasoning abilities not only between general-purpose chat LLMs and reasoning-focused LLMs but also in the performance of current reasoning-focused models when applied to classical cryptographic decryption tasks, highlighting the challenges these models face in understanding and manipulating encrypted data. Through detailed analysis and error investigations, we provide several key observations that shed light on the limitations and potential improvement areas for LLMs in cryptographic reasoning.These findings underscore the need for continuous advancements in LLM reasoning capabilities.

## introduction large language models (llms) have revolutionized artificial intelligence by achieving state-ofthe-art (sota) performance across diverse domains, from natural language understanding (nlp) @xcite @xcite to complex problemsolving @xcite . recent models, such as @xcite and claude 3.5 @xcite , have demonstrated unprecedented versatility, excelling in tasks ranging from creative writing to technical analysis. a particularly notable advancement lies in the reasoning-enhanced llms, which have emerged as a critical benchmark for evaluating llms' intelligence and now can solve mathematical problems @xcite @xcite , debug intricate code @xcite , and even engage in multi-step logical deduction @xcite with human-like proficiency. for instance, specialized architectures like o1 @xcite and @xcite have pushed the boundaries of ai reasoning, achieving breakthroughs in domains such as theorem proving @xcite and algorithmic optimization @xcite . these achievements underscore the transformative potential of llms as general-purpose reasoning engines, capable of adapting to both broad and specialized challenges. to quantify progress, the community has proposed numerous benchmarks targeting mathematical reasoning (e.g., @xcite , aime 1 , coding proficiency (e.g., @xcite , mbpp @xcite ), and general logical deduction (e.g., @xcite , mmbench @xcite , @xcite . these testbeds have become indispensable tools for assessing model capabilities. despite extensive evaluations in mathematics and coding, one critical domain remains underexplored: cryptographic decryption. cryptographic reasoning @xcite demands unique capabilities, including pattern recognition, algorithmic reverse-engineering, and contextual understanding of security constraints @xcite -skills distinct from those tested in conventional benchmarks. this gap is particularly consequential, as cryptography lies at the heart of modern digital security @xcite , with applications spanning privacy-preserving communication @xcite , secure authentication @xcite , and data integrity @xcite . the absence of a rigorous benchmark for cryptographic reasoning not only limits the true understanding of llm's reasoning ability but also hinders progress toward ai systems capable of contributing to security-critical contexts (e.g., jailbreaking @xcite ). ope-nai has scratched the surface of this challenge and put a demo foot_1 when releasing their strong reasoning model o1, but no serious efforts have been made to reveal this challenge in the committee. to address this gap, we introduce cipherbank, the first comprehensive benchmark specially designed to evaluate llms' reasoning capabilities in cryptographic decryption tasks. cipherbank is meticulously constructed to reflect real-world scenarios requiring encryption, instead of general texts that may serve as a toy testbed, with 2,358 problems derived from 262 unique plaintexts across 5 domains (e.g., personal privacy, financial information) and 14 subdomains (e.g., identity information, personal income). as for cipher algorithms, it spans 3 major cryptographic categories-substitution ciphers (e.g., rot13, vigenère), transposition ciphers (e.g., reverse, swappairs), and custom hybrid algo-rithms-encompassing 9 distinct encryption methods, covering 5 difficulty levels (from basic to expert) to ensure a diverse range of challenges. by integrating privacy-sensitive contexts and multilayered cryptographic challenges, cipherbank provides a nuanced evaluation framework that captures both the complexity and practicality of real-world decryption tasks. we evaluate cipherbank on sota llms, including general-purpose models @xcite , deepseek-v3 @xcite ) and reasoning-optimized models (o1 @xcite , @xcite ). results reveal striking limitations: even advanced models struggle with classical ciphers, achieving only 45.14 score on tasks solvable by human cryptanalysts. notably, we observe a significant performance gap between general chat llms and specialized reasoning models, suggesting that current reasoning optimizations inadequately address cryptographic challenges. besides, we also provide studies on different aspects for deep understandings, such as evaluate on noised plaintexts and different length of plaintexts. observations show the limitations of current models in decryption reasoning, with chat and reasoning models each exhibiting distinct strengths and weaknesses in cryptographic tasks. these findings highlight the need for targeted improvements in llms' cryptographic reasoning, with implications for both ai safety (e.g., adversarial robustness) and applications in cybersecurity. ## cipherbank construction cipherbank is a purpose-built benchmark designed to rigorously evaluate the reasoning capabilities of llms in cryptographic decryption tasks. it integrates three core components to ensure comprehensive coverage of real-world scenarios and cryptographic complexity: (1) diverse plaintexts meticulously constructed from multiple dimensions of real-world privacy-sensitive data, ensuring the decryption process aligns with practical requirements; ## detailed analysis in this section, we conduct a detailed analysis from the perspectives of plaintext characteristics, noise levels, testing methodologies, finer-grained evaluation metrics, and error analysis to gain deeper insights into the strengths and limitations of different llms in cryptographic decryption. ## conclusion in this work, we introduce cipherbank, a comprehensive benchmark for evaluating reasoning capabilities through cryptographic decryption. ci-pherbank includes 5 domains, 14 subdomains of plaintext data, 9 encryption algorithms, and 2,358 decryption tasks. by testing sota llms on cipherbank, we uncover significant limitations in their decryption abilities, revealing distinct strengths and weaknesses between reasoning and chat models. our analysis identifies key deficiencies in current reasoning approaches and suggests directions for improvement, positioning cipherbank as a novel benchmark for advancing structured inference and cryptographic reasoning in developing future llms.