Benchmarking Large Language Models for Cryptanalysis and Side-Channel Vulnerabilities (2025)

Abstract: Recent advancements in Large Language Models (LLMs) have transformed natural language understanding and generation, leading to extensive benchmarking across diverse tasks. However, cryptanalysis—a critical area for data security and its connection to LLMs’ generalization abilities remains underexplored in LLM evaluations. To address this gap, we evaluate the cryptanalytic potential of state‐of‐the‐art LLMs on ciphertexts produced by a range of cryptographic algorithms. We introduce a benchmark dataset of diverse plaintexts—spanning multiple domains, lengths, writing styles, and topics—paired with their encrypted versions. Using zero‐shot and few‐shot settings along with chain‐of‐thought prompting, we assess LLMs’ decryption success rate and discuss their comprehension abilities. Our findings reveal key insights into LLMs’ strengths and limitations in side‐channel scenarios and raise concerns about their susceptibility to under-generalization related attacks. This research highlights the dual‐use nature of LLMs in security contexts and contributes to the ongoing discussion on AI safety and security.

## introduction the advancement of large language models (llms) such as chatgpt @xcite , claude, mistral @xcite , and gemini @xcite has significantly transformed the field of nlp. despite these impressive capabilities, the widespread deployment of llms has raised concerns about their safety and ethical use @xcite . one pressing issue is the potential for these models to be manipulated or "jailbroken" to bypass established safety protocols @xcite . cryptanalysis, is an area of cybersecurity that focuses on analyzing encrypted information (ciphertext) without direct knowledge of the encryption process, to uncover weaknesses in the encryption system and recover the original message (plaintext) @xcite . we focus on how llms struggle to generalize across different text encodings that create opportunities for mismatched generalization attacks and exploit the long-tailed distribution of llm knowledge to increase jailbreak success @xcite . attackers might translate harmful instructions into ciphers @xcite or use different languages that are inherently learned during pre-training but safety measures may be less robust @xcite . additionally, adversarial encoding shift techniques convert the original input into alternative formats like ascii or morse code, fragment the input, or use different languages. these generalization failures are also exploited through programmatic behaviors, such as code injection and virtualization @xcite . further studies on llm jailbreak attacks, such as selfcipher @xcite , bijection learning @xcite , artprompt @xcite , changing verb tense @xcite and translation to low-resourced language @xcite have demonstrated similar behaviors using innocuous formats like ascii art, language translation and bijection encoding. while llms perform well in language understanding and generation, they face challenges with tasks that require precise numerical reasoning during inference @xcite . decrypting encrypted texts demands both linguistic understanding and intuitive mathematical reasoning, posing a significant challenge in cryptanalysis @xcite . moreover, since most encryption schemes operate at the character or block level, and llms are primarily trained on word or sub-word tokens, this representational mismatch further limits their effectiveness in cryptographic tasks. to address this research gap, we introduce a comprehensive benchmark dataset for evaluating llms' cryptanalysis capabilities. the dataset consists of diverse plaintexts from multiple domains, varying in length, style, and topic, and includes both human-generated and llm-generated content. each plaintext is paired with encrypted versions created using different cryptographic algorithms. we conduct zero-shot and few-shot evaluation of several state-of-the-art llms, assessing their decryption accuracy and discuss their comprehension abilities. additionally, we examine the safety implications of llms' partial comprehension of encrypted texts, revealing vulnerabilities that could be exploited in generalization-based attacks. our contributions are summarized as follows: ## dataset we curated a novel dataset consisting of diverse plain texts (both llm and human generated) along with its cipher-text, each of them encrypted using nine encryption algorithms. the dataset includes a total of 4,509 entries, with detailed statistics and sample dataset provided in the tables 1 3 and 2. ## experimental results and analysis we evaluate various llms on encryption methods in zero-shot (zs) and few-shot (fs) settings across diverse texts and complexities. ## discussion does a low benchmark score (em, nl, bleu) mean a better and more secure model? are lower scores preferred? the benchmark score in our analysis reflects two key aspects of model performance: comprehension and vulnerability to exploitation. lower benchmark scores generally indicate that the model struggles to understand or decrypt the transformed text, suggesting better resistance to side-channel attacks and exploitation. conversely, higher benchmark scores indicate that an unaligned model is more adept at comprehending and decrypting the transformed text, which, makes it more susceptible to jailbreak attacks. for safety aligned models, @xcite suggests that decryption scores should be high for benign texts and low for prompts that intend a harmful response. this evaluation gives directions so that partial comprehension concerns are addressed while developing llm safeguards. ## conclusion we introduced a benchmark dataset and evaluation framework for assessing the cryptanalysis capabilities of llms on encrypted texts. our analysis revealed that even when llms are unable to fully decrypt complex ciphers, they still exhibit a degree of partial comprehension, and may be susceptible to generalization based attacks. we examined the safety implications of llms' generalization abilities, and discussed how model behaviors of token inflation and partial decryption influence potential jailbreaks. our findings and evaluation methods provide directions for analyzing llm safeguardsand establish considerations that must be addressed while aligning llms towards safety.