langsegment-backup 0.3.5.post1
Multilingual text segmentation tool for zh/ja/en/ko and more. Unofficial backup of LangSegment 0.3.5.
License: BSD-3-Clause
Requires Python: >=3.6
Keywords: language detection, language identification, langid, nlp, language, multilingual, text segmentation, TTS, text-to-speech, tokenization

# LangSegment (Unofficial Backup)

[![PyPI version](https://badge.fury.io/py/langsegment-backup.svg)](https://pypi.org/project/langsegment-backup/)
[![License: BSD-3-Clause](https://img.shields.io/badge/License-BSD_3--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)

> ⚠️ **This is an unofficial backup** of LangSegment 0.3.5. The original repository has been removed or made private.

A multilingual text segmentation tool that automatically identifies and splits text by language. Particularly useful for TTS (Text-to-Speech) processing with mixed-language content.

## Attribution & History

- **Original Author**: sunnyboxs (juntaosun)
- **Original Repository**: https://github.com/juntaosun/LangSegment (now removed)
- **First Backup**: https://github.com/chameleon-ai/LangSegment-0.3.5-backup
- **This Fork**: https://github.com/MiniXC/LangSegment-0.3.5-backup (PyPI release)

This package is published to PyPI to preserve access to this useful tool after the original repository was removed. All credit for the original work goes to the original author.

## Installation

```bash
pip install langsegment-backup
```

## Supported Languages

Primary support:
- 🇨🇳 Chinese (zh)
- 🇯🇵 Japanese (ja)
- 🇬🇧 English (en)
- 🇰🇷 Korean (ko)

Experimental support:
- 🇫🇷 French (fr)
- 🇻🇳 Vietnamese (vi)
- 🇷🇺 Russian (ru)
- 🇹🇭 Thai (th)

The tool can actually support up to 97 different languages through the underlying `py3langid` library.

## Quick Start

```python
from LangSegment import LangSegment

# Basic usage - segment mixed language text
text = "你好世界！Hello World! こんにちは！안녕하세요!"
results = LangSegment.getTexts(text)

for item in results:
    print(f"[{item['lang']}] {item['text']}")
```

Output:
```
[zh] 你好世界！
[en] Hello World! 
[ja] こんにちは！
[ko] 안녕하세요!
```

## Features

### Language Filtering

You can specify which languages to detect and in what priority order:

```python
from LangSegment import LangSegment

# Set language filter (priority order: left = highest)
LangSegment.setfilters(["zh", "ja", "en", "ko"])

# Or for Chinese-English only
LangSegment.setfilters(["zh", "en"])
```

### Language Statistics

Get statistics about the languages in your text:

```python
from LangSegment import LangSegment

text = "你好世界！Hello World! こんにちは！"
LangSegment.getTexts(text)

# Get language counts (sorted by character count, descending)
counts = LangSegment.getCounts()
print(counts)  # [('zh', 10), ('en', 12), ('ja', 6)]

# Get the primary language
primary_lang, char_count = counts[0]
print(f"Primary language: {primary_lang}")
```

### Manual Language Tags

You can manually specify language regions using tags:

```python
text = "这是中文<ja>これは日本語です</ja>这又是中文"
results = LangSegment.getTexts(text)
```

### SSML Support (Chinese)

The tool includes SSML-like tags for Chinese number/date processing:

```python
# Number reading
text = "<number>12345</number>"  # → 一二三四五

# Phone number
text = "<telephone>13812345678</telephone>"  # → 幺三八幺二三四五六七八

# Currency
text = "<currency>12345</currency>"  # → 一万二千三百四十五

# Date
text = "<date>2024-08-24</date>"  # → 二零二四年八月二十四日
```

### Configuration Options

```python
from LangSegment import LangSegment

# Set Chinese/Japanese priority threshold (0-1, default: 0.89)
LangSegment.setPriorityThreshold(0.89)

# Enable/disable result merging
LangSegment.setLangMerge(True)

# Keep Chinese pinyin format
LangSegment.setKeepPinyin(False)

# Enable preview features (French, Vietnamese support)
LangSegment.setEnablePreview(True)
```

## API Reference

### Main Functions

| Function | Description |
|----------|-------------|
| `LangSegment.getTexts(text)` | Segment text and return list of `{lang, text, score}` dicts |
| `LangSegment.classify(text)` | Alias for `getTexts()` |
| `LangSegment.getCounts()` | Get language statistics as list of `(lang, count)` tuples |
| `LangSegment.setfilters(list)` | Set language filter/priority list |
| `LangSegment.getfilters()` | Get current language filters |

### Configuration Functions

| Function | Description |
|----------|-------------|
| `setPriorityThreshold(float)` | Set zh/ja disambiguation threshold (0-1) |
| `getPriorityThreshold()` | Get current threshold |
| `setLangMerge(bool)` | Enable/disable merging adjacent same-language segments |
| `getLangMerge()` | Get merge setting |
| `setKeepPinyin(bool)` | Keep Chinese pinyin format in parentheses |
| `getKeepPinyin()` | Get pinyin setting |
| `setEnablePreview(bool)` | Enable experimental language support |
| `getEnablePreview()` | Get preview setting |

## Dependencies

- `numpy >= 1.19.5`
- `py3langid >= 0.2.2`

## License

BSD 3-Clause License

Copyright (c) 2024 juntaosun. All rights reserved.

See [LICENSE](LICENSE) for full license text.

## Contributing

Since this is a backup/preservation fork, major feature additions are not planned. However, bug fixes and compatibility updates are welcome. Please open an issue or pull request at https://github.com/MiniXC/LangSegment-0.3.5-backup.

## Acknowledgments

- Original author: **sunnyboxs (juntaosun)** for creating LangSegment
- **chameleon-ai** for creating the first backup at https://github.com/chameleon-ai/LangSegment-0.3.5-backup
- The `py3langid` library by Adrien Barbaresi
- The original `langid.py` by Marco Lui