Roman Urdu Crypto Sentiment Evaluation This repository contains the evaluation artifacts for the Fatima Fellowship Fall 2026 application by Abdullah Khan.
- The Blind Spot Growing up in Pakistan and building QuantAura—a crypto analytics SaaS—I've observed a massive disconnect between how frontier AI models process financial sentiment and how retail traders in South Asia actually communicate. In Pakistan and India, millions of retail investors discuss markets on Telegram, WhatsApp, and Discord using "Roman Urdu/Hindi" (Urdu/Hindi written in Latin script) mixed heavily with native crypto slang.
Standard financial benchmarks (like FiQA) are exclusively in formal English. When I attempt to use smaller open-weight models (like Llama-3.2-3B) to parse local community sentiment (e.g., "Bhai, BTC dump ho raha hai, support toot gayi, kya karun?"), they frequently fail. They hallucinate, treat the text as misspelled English, or entirely miss the underlying financial anxiety or intent. This gap is critical because it excludes a massive, growing demographic of Global South retail investors from AI-driven financial accessibility, algorithmic trading integrations, and protection against scams.
- Evaluation I systematically evaluated this gap on meta-llama/Llama-3.2-3B-Instruct (3.2B parameters) using a custom compiled dataset of Roman Urdu crypto discussions.
The evaluation script (roman_urdu_crypto_eval.ipynb) and dataset (dataset.json) are included in this repository. The model consistently struggles to capture the nuance of Roman Urdu slang, often outputting confused sentiment compared to formal English equivalents, or failing to identify the intent of the trader.
- The Path Forward To bridge this gap, I propose a targeted data curation strategy combined with cross-lingual Low-Rank Adaptation (LoRA) fine-tuning:
Data Curation: Construct a "Roman-Urdu Financial Instruct" dataset by scraping public crypto Telegram/Discord channels in South Asia. We can use a strong frontier model (like GPT-4o) to synthetically annotate the raw text for sentiment, intent (e.g., FOMO, panic selling), and entity extraction. Architectural Adjustment: Because Roman Urdu lacks standardized spelling (e.g., 'karun', 'karon', 'krun' all mean 'should I do'), traditional LLM tokenizers fragment these words inefficiently. Expanding the model's vocabulary with frequent Roman Urdu subwords before fine-tuning, or applying a character-level CNN embedding layer for out-of-vocabulary robustness, could dramatically reduce tokenization fragmentation. Fine-Tuning Paradigm: Fine-tune a lightweight model like Llama-3.2-3B on this dataset using LoRA, specifically targeting the attention and feed-forward layers to align its financial reasoning with South Asian linguistic nuances.