indicate 0.5.1
Transliterations to/from Indian languages
License: MIT
Requires Python: <3.13,>=3.11
Keywords: transliterate, indic, hindi, english

# Indicate: Transliterate Indic Languages with TensorFlow and LLMs

[![Notary Badge](https://notarypy.soodoku.workers.dev/badge/indicate/0.2.1/indicate-0.2.1-py3-none-any.whl)](https://pypi.org/integrity/indicate/0.2.1/indicate-0.2.1-py3-none-any.whl/provenance)
[![PyPI Version](https://img.shields.io/pypi/v/indicate.svg)](https://pypi.python.org/pypi/indicate)
[![Downloads](https://static.pepy.tech/badge/indicate)](https://pepy.tech/project/indicate)
[![Tests](https://github.com/in-rolls/indicate/workflows/test/badge.svg)](https://github.com/in-rolls/indicate/actions?query=workflow%3Atest)
[![Documentation](https://img.shields.io/badge/docs-github.io-blue)](https://in-rolls.github.io/indicate/)

**Indicate** provides high-quality transliteration between Indic languages and English using both traditional TensorFlow models and state-of-the-art LLMs (Large Language Models).

## 🚀 Features

- **🧠 Dual Backend Support**: Choose between TensorFlow models or LLM-based transliteration
- **🌍 Multi-Language**: 12+ Indic languages (Hindi, Tamil, Telugu, Bengali, etc.)
- **🔄 Bidirectional**: Supports both Indic→English and English→Indic transliteration
- **🛡️ Production Ready**: Safe file handling, atomic writes, backup support
- **📊 Structured Output**: Rich JSON format with metadata and error handling
- **⚡ Batch Processing**: Efficient processing of large files with progress tracking

## 🎯 Supported Languages

Hindi • Tamil • Telugu • Bengali • Gujarati • Kannada • Malayalam • Punjabi • Marathi • Odia • Urdu • Sanskrit ↔ English

## Install

We strongly recommend installing `indicate` inside a Python virtual environment (see [venv documentation](https://docs.python.org/3/library/venv.html#creating-virtual-environments))

**Requirements:** Python 3.11 or 3.12 (TensorFlow does not yet support Python 3.13)

```bash
pip install indicate
```

## 🔧 Quick Setup

### For LLM-based transliteration (recommended):
```bash
pip install indicate

# Set your API key (choose one):
export OPENAI_API_KEY=your-key
export ANTHROPIC_API_KEY=your-key  
export GOOGLE_API_KEY=your-key
```

### For TensorFlow-only usage:
```bash
pip install indicate
# No API key needed - uses pre-trained models
```

## 🎯 Usage

### 🧠 LLM-Based Transliteration (New!)

The LLM backend provides higher accuracy and supports all Indic languages:

```bash
# Simple transliteration (auto-detects Hindi)
indicate llm "राजशेखर चिंतालपति"
# Output: Rajashekar Chintalapati

# Specify languages explicitly  
indicate llm "முருகன்" --source tamil --target english
# Output: Murugan

# Between Indic languages
indicate llm "नमस्ते" --source hindi --target tamil  
# Output: நமஸ்தே

# Safe batch processing with structured JSON output
indicate llm --input names.txt --output results.json --format json --batch --backup

# Dry run to preview changes
indicate llm --input large_file.txt --dry-run
```

**Python API:**
```python
from indicate import IndicLLMTransliterator

# Initialize for any language pair
transliterator = IndicLLMTransliterator('hindi', 'english')
result = transliterator.transliterate('राजशेखर चिंतालपति')
print(result)  # Output: Rajashekar Chintalapati

# Batch processing
texts = ["राजेश", "गौरव", "प्रिया"]
results = transliterator.transliterate_batch(texts)
print(results)  # ['Rajesh', 'Gaurav', 'Priya']
```

### 🤖 TensorFlow Backend (Traditional)

```bash
# Hindi to English using TensorFlow model
indicate hindi2english "राजशेखर चिंतालपति"
# Output: rajashekar chintalapati

# From file
indicate hindi2english --input hindi.txt --output english.txt

# Batch processing
indicate hindi2english --input large_file.txt --batch
```

**Python API:**
```python
from indicate import hindi2english
result = hindi2english("हिंदी")
print(result)  # Output: hindi
```

## 📊 JSON Output Format

The LLM backend provides rich, structured output perfect for data processing:

```json
{
  "metadata": {
    "source_language": "hindi",
    "target_language": "english", 
    "timestamp": "2024-12-09T12:00:00Z",
    "total_lines": 3,
    "successful_lines": 3,
    "failed_lines": 0,
    "encoding": "utf-8"
  },
  "results": [
    {
      "line_number": 1,
      "input_text": "राजेश कुमार",
      "output_text": "Rajesh Kumar", 
      "source_lang": "hindi",
      "target_lang": "english",
      "confidence": "high",
      "processing_time": 1.2,
      "timestamp": "2024-12-09T12:00:01Z"
    }
  ]
}
```

## 🛡️ Safety Features

- **🔒 Input/Output Validation**: Prevents accidental file overwrites
- **⚛️ Atomic Writing**: Safe file operations using temporary files
- **💾 Automatic Backups**: Optional timestamped backups of existing files
- **🔄 Resume Support**: Resume interrupted batch operations
- **👁️ Dry Run Mode**: Preview operations before execution

## 🎛️ Advanced Usage

```bash
# Show few-shot examples being used
indicate llm --show-examples --source bengali --target english

# Resume interrupted batch job
indicate llm --input large_file.txt --output results.txt --resume

# Use specific LLM provider/model
indicate llm "text" --provider anthropic --model claude-3-opus

# Process JSON from previous results
indicate llm --input results.json --source english --target hindi
```

## 🔄 Backend Comparison

| Feature | TensorFlow Backend | LLM Backend |
|---------|------------------|-------------|
| **Languages** | Hindi ↔ English only | 12+ Indic languages ↔ English + Inter-Indic |
| **Setup** | No API key needed | Requires LLM API key |
| **Speed** | Very fast (local) | Moderate (API calls) |
| **Accuracy** | Good for common words | Excellent for all types |
| **Cost** | Free | Pay per API call |
| **Offline** | ✅ Works offline | ❌ Requires internet |
| **Batch Processing** | ✅ | ✅ with safety features |

## 🧪 Testing Locally

1. **Clone and install**:
   ```bash
   git clone https://github.com/in-rolls/indicate.git
   cd indicate
   uv sync  # or pip install -e .
   ```

2. **Run tests**:
   ```bash
   # All tests
   python -m pytest
   
   # Specific tests
   python -m pytest tests/test_llm_indic.py
   python -m pytest tests/test_file_safety.py
   ```

3. **Test both backends**:
   ```bash
   # TensorFlow backend
   indicate hindi2english "हिंदी"
   
   # LLM backend (set API key first)
   export OPENAI_API_KEY=your-key
   indicate llm "हिंदी"
   ```

## Data

The datasets used to train the model:

- [Indian Election affidavits](https://affidavit.eci.gov.in/CandidateCustomFilter)
- [Google Dakshina dataset](https://github.com/google-research-datasets/dakshina)
- [ESPN Cric Info](https://www.espncricinfo.com/hindi/series/pakistan-tour-of-england-2021-1239529/england-vs-pakistan-1st-odi-1239537/full-scorecard) for hindi version of the [english scorecard](https://www.espncricinfo.com/series/pakistan-tour-of-england-2021-1239529/england-vs-pakistan-1st-odi-1239537/full-scorecard)
- [IIT Bombay English-Hindi Corpus](https://www.cfilt.iitb.ac.in/iitb_parallel/)

## Evaluation

Model was evaluated on test dataset of Google Dakshina dataset, Model predicted 73.64% exact matches.
[Indic-trans](https://github.com/libindic/indic-trans) predicted 63.12% exact matches on Google Dakshina dataset.

Below is the edit distance metrics on test dataset (0.0 mean exact match, the farther away from 0.0, the difference is more between predicted text and actual text):

![Edit distance metrics of model on Google Dakshina test dataset](https://github.com/in-rolls/indicate/raw/master/images/h2e_ed.png)

## Authors

Rajashekar Chintalapati and Gaurav Sood

## Contributor Code of Conduct

The project welcomes contributions from everyone! In fact, it depends on it. To maintain this welcoming atmosphere, and to collaborate in a fun and productive way, we expect contributors to the project to abide by the [Contributor Code of Conduct](http://contributor-covenant.org/version/1/0/0/).

## License

The package is released under the [MIT License](https://opensource.org/licenses/MIT).