semvec/pypi-packages huggingface dataset:
kabyle-corpus-toolkit 2.0.0
Tools for downloading, processing, and normalizing Kabyle and Occitan language corpora from Tatoeba
License: MIT
Requires Python: >=3.8
Keywords: kabyle, occitan, corpus, nlp, tatoeba, parallel-corpus, language-processing, berber, tamazight

# Kabyle Corpus Toolkit

[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Tools for downloading, processing, and normalizing **Kabyle** (kab) and **Occitan** (oci) language corpora from Tatoeba and other sources.

## Features

- **Download Tatoeba Data**: Automated download of sentences and links from Tatoeba.org
- **Parallel Corpus Creation**: Build aligned English-Kabyle and English-Occitan sentence pairs
- **French Chain Translation**: Expand coverage by routing Kabyle→French→English translations
- **Character Normalization**: Fix encoding issues and normalize extended Latin characters
- **Language Validation**: Validate corpus quality using GlotLID FastText models
- **Stopword Generation**: Generate language-specific stopword lists from corpus statistics

## Installation

### Basic Installation

```bash
pip install kabyle-corpus-toolkit