From Billion-Scale to Million-Scale Language Models: Operational Capabilities for Embedded Post-Exploitation
Authors: Varlese, Mario, Romano, Simon Pietro, Tapiador, Juan
Type: Software
Published: 2026-08-03
Keywords: ultra-Small Language Models, AI-powered Malware, Post-exploitation, Offensive Security, Code Generation, Empirical Characterization

<h1><strong>SLM CodeEval Framework</strong></h1>
<p>An advanced and modular framework for evaluating the code generation capabilities (Python and Lua) of Small Language Models (SLMs) in offensive security scenarios (e.g., Directory Enumeration, Backdoor Generation).</p>
<p>The framework supports high-performance inference via vLLM, Constrained Decoding (via EBNF), dynamic validation pipelines in isolated sandboxes, and complex interaction paradigms such as Reflection (Feedback Loops) and Least-to-Most (Iterative Prompting).</p>
<p> </p>
<h2>Dataset</h2>
<p>The results of the executions under the different configurations described in the Paper <em>From Billion-Scale to Million-Scale Language Models: Operational Capabilities for Embedded Post-Exploitation</em> have been packaged using the datasets library in From_Billion2_Million_executions_dataset.zip</p>
<p> </p>
<h2>Setup</h2>
<p>The project uses uv for ultra-fast dependency and virtual environment management, and is optimized for Python 3.12.3.</p>
<ol>
<li><strong>System Requirements:</strong><br>Ensure you have the necessary compilers and interpreters to validate the generated code (including Lua 5.4). You can install them automatically via the Makefile:<br><code>make sys-deps</code></li>
<li><strong>Environment Initialization:</strong><br>To create the virtual environment and install the Python libraries (PyTorch, vLLM, Outlines, etc.):<br><code>make setup</code></li>
</ol>
<p> </p>
<h2>Ways to Launch Experiments</h2>
<p>The execution system is entirely managed via the Makefile, which ensures the correct configuration of environment variables (e.g., multiprocessing for vLLM).</p>
<p>All experiments save their state in a lock file (completed_*.json) to allow resuming execution in case of a crash without losing progress.</p>
<p>You can specify which configuration file to use by passing the CONFIG variable. By default, it uses experiments.json.</p>
<ol>
<li><strong>Baseline / One-Shot Interactions</strong><br>Runs standard evaluations (zero-shot, few-shot, chain-of-thought) in a single pass.<br><code>make exp-run CONFIG=backdor_experiments-python.json</code></li>
<li><strong>Reflection / Feedback Loops</strong><br>Runs the framework in self-correction mode. If the code fails sandbox validation, the interpreter's error is fed back into the model's prompt for N iterations.<br><code>make exp-feedback CONFIG=backdor_experiments-python.json</code></li>
<li><strong>Least-to-Most / Iterative Interactions</strong><br>Breaks down the task into smaller steps, maintaining the accumulated code context between iterations.<br><code>make exp-iterative CONFIG=backdor_experiments-python.json</code></li>
<li><strong>Full Execution</strong><br>Runs all modes sequentially.<br><code>make exp-all</code><br><em>Note: To reset the lock files and force a fresh execution, use the commands: make reset-exp, make reset-feedback, or make reset-all.</em></li>
</ol>
<h2>Configuration Files</h2>
<p>The framework is driven by JSON files that define the experiment matrix (grid-search). Example (dir_enum_experiments-python.json):</p>
<p><br><code>{</code><br><code>  "standard_targets": [{"language": "python", "dataset": "tasks-python.jsonl"}],</code><br><code>  "iterative_targets": [{"language": "python", "dataset": "tasks_iterative-python.jsonl"}],</code><br><code>  "models": [</code><br><code>    "HuggingFaceTB/SmolLM2-360M-Instruct",</code><br><code>    "Qwen/Qwen2.5-Coder-1.5B-Instruct"</code><br><code>  ],</code><br><code>  "strategies": ["core", "few_shot", "thinking", "few_shot_thinking"],</code><br><code>  "temperatures": [0.2],</code><br><code>  "n_samples": [10],</code><br><code>  "max_tokens": [1024],</code><br><code>  "constraints": ["none", "ebnf"],</code><br><code>  "num_runs": 10,</code><br><code>  "seeds": [42, 123, 456, 789, 1024, 2048, 4096, 8192, 16384, 32768],</code><br><code>  "max_reflexion_steps": 10</code><br><code>}</code></p>
<p> </p>
<h3>Key Parameters:</h3>
<ul>
<li><strong>strategies</strong>: Defines the prompting paradigm. The framework uses the Decorator Pattern to dynamically compose complex prompts by adding examples (few_shot) or structured reasoning instructions (thinking).</li>
<li><strong>constraints</strong>: If set to ebnf, applies Guided Decoding via the Outlines library, forcing the model to generate syntactically valid outputs for the target language even before execution.</li>
<li><strong>num_runs and seeds</strong>: Control the statistical robustness of the experiment by varying the sampling seed across multiple runs.</li>
</ul>
<p> </p>
<h2>Project Structure</h2>
<p>The code applies Clean Architecture principles (Interfaces, Dependency Injection, Decorators) to be easily extensible to new LLMs or programming languages.</p>
<p><strong>Entry Points:</strong></p>
<ul>
<li>run_experiments.py, run_feedback_experiment.py, run_iterative_experiment.py: Macro-level scripts orchestrating the grid-search over configurations.</li>
<li>main.py / main_iterative.py: Process the single experiment by instantiating vLLM and the pipeline.</li>
</ul>
<p><br><strong>Core Logic:</strong></p>
<ul>
<li>orchestrator.py: Contains the BenchmarkFacade, which hides complexity by wiring together the LLM Runner, validation, extraction, and metric reporting. Includes the Chain of Responsibility for the tests.</li>
<li>prompt_builder.py: Implements the Decorator pattern (e.g., FewShotDecorator, ThinkingDecorator) and Strategy pattern (InstructFormatter, BaseFormatter) to dynamically build prompts suitable for base or chat-tuned models.</li>
<li>language_profiles.py: Defines syntax rules (EBNF), keywords, and the isolated sandboxing environment (e.g., stdout/stderr interception and timeout) specific to Lua and Python.</li>
<li>extractor.py: Multi-strategy module (Regex, Markdown, JSON) to safely extract code from the LLM's generated text, cleaning it of Chain-of-Thought blocks.</li>
<li>validators.py: Validation pipeline (Syntax Check -> Signature Check -> Required Elements -> Sandbox Execution).</li>
<li>metrics.py: Calculates stable metrics like Pass@k and hardware telemetries (VRAM, Tokens/sec).</li>
<li>decoding_constraints.py: Bridge to vLLM and Outlines to enable code generation guided by EBNF grammars or JSON Schema.</li>
</ul>
<p><br><strong>Output:</strong></p>
<ul>
<li>results/: Contains the final structured JSONs with Pass@k metrics, sandbox logs, and generated code.</li>
<li>logs/: Temporal log files for analyzing the experimental runner's behavior and crashes.</li>
</ul>
<p> </p>
<h2>Results Analysis (Results_Analysis.ipynb)</h2>
<p>The Results_Analysis.ipynb notebook serves as the core data processing and analytical engine for this project. It leverages the raw execution data derived from the experimental setup (hosted on Google Drive) to fully reproduce the quantitative analysis and empirical findings presented in the paper.</p>
<p>The analytical workflow is methodologically structured into three distinct phases:</p>
<ol>
<li><strong>One-Shot Interaction Analysis</strong>: Evaluates the baseline experimental results, assessing the models' innate generative capabilities under single-turn constraints without retroactive corrections.</li>
<li><strong>Feedback Loop Dynamics (Reflection)</strong>: Examines the impact of self-correction mechanisms, quantifying how the models utilize interpreter error logs to progressively debug and achieve syntactic alignment.</li>
<li><strong>Iterative Interaction Strategy (Least-to-Most)</strong>: Analyzes the performance variations when complex tasks are decomposed into progressive structural steps, highlighting the systemic effects of algorithmic scaffolding and cognitive load.</li>
</ol>