unitok 4.4.4
Unified Tokenizer
License: MIT Licence
Keywords: token, tokenizer, NLP, transformers, glove, bert, llama

# UniTok v4

Unified preprocessing for heterogeneous ML tables: text, categorical, and numerical columns in one pipeline.

- Python package: `unitok`
- Current package version: 4.4.2 (from `setup.py`)
- Legacy v3 docs: `README_v3.md`

## Why UniTok

UniTok turns raw tabular data into model-ready numeric tables while preserving:

- Consistent vocabularies across multiple datasets
- Clear feature definitions (column -> tokenizer -> output feature)
- Reproducible metadata and saved artifacts
- Simple unions across datasets via shared keys

## Core Ideas

- **UniTok**: Orchestrates preprocessing lifecycle and holds processed data.
- **Feature**: Binds a column to a tokenizer and output name.
- **Tokenizer**: Encodes objects to ids (entity, split, digit, transformers).
- **Vocab**: Global index for tokens; shared across datasets.
- **Meta**: Stores schema, tokenizers, vocabularies, and feature definitions.
- **State**: `initialized` -> `tokenized` -> `organized`.

## Install

```bash
pip install unitok
```

Requirements: Python 3.7+, pandas, transformers, tqdm, rich.

## Quickstart

```python
import pandas as pd
from unitok import UniTok, Vocab
from unitok.tokenizer import BertTokenizer, TransformersTokenizer, EntityTokenizer, SplitTokenizer, DigitTokenizer

item = pd.read_csv(
    'news-sample.tsv', sep='\t',
    names=['nid', 'category', 'subcategory', 'title', 'abstract'],
    usecols=['nid', 'category', 'subcategory', 'title', 'abstract'],
)
item['abstract'] = item['abstract'].fillna('')

user = pd.read_csv(
    'user-sample.tsv', sep='\t',
    names=['uid', 'history'],
)

interaction = pd.read_csv(
    'interaction-sample.tsv', sep='\t',
    names=['uid', 'nid', 'click'],
)

item_vocab = Vocab(name='nid')
user_vocab = Vocab(name='uid')

with UniTok() as item_ut:
    bert = BertTokenizer(vocab='bert')
    llama = TransformersTokenizer(vocab='llama', key='huggyllama/llama-7b')

    item_ut.add_feature(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid', key=True)
    item_ut.add_feature(tokenizer=bert, column='title', name='title@bert', truncate=20)
    item_ut.add_feature(tokenizer=llama, column='title', name='title@llama', truncate=20)
    item_ut.add_feature(tokenizer=bert, column='abstract', name='abstract@bert', truncate=50)
    item_ut.add_feature(tokenizer=llama, column='abstract', name='abstract@llama', truncate=50)
    item_ut.add_feature(tokenizer=EntityTokenizer(vocab='category'), column='category')
    item_ut.add_feature(tokenizer=EntityTokenizer(vocab='subcategory'), column='subcategory')

with UniTok() as user_ut:
    user_ut.add_feature(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid', key=True)
    user_ut.add_feature(tokenizer=SplitTokenizer(vocab=item_vocab, sep=','), column='history', truncate=30)

with UniTok() as inter_ut:
    inter_ut.add_index_feature(name='index')
    inter_ut.add_feature(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid')
    inter_ut.add_feature(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid')
    inter_ut.add_feature(tokenizer=DigitTokenizer(vocab='click', vocab_size=2), column='click')

item_ut.tokenize(item).save('sample-ut/item')
item_vocab.deny_edit()
user_ut.tokenize(user).save('sample-ut/user')
inter_ut.tokenize(interaction).save('sample-ut/interaction')
```

## Loading Saved Data

```python
from unitok import UniTok

ut = UniTok.load('sample-ut/item')
print(len(ut))
print(ut[0])
```

## Combining Datasets (Union)

```python
with inter_ut:
    inter_ut.union(user_ut)
    print(inter_ut[0])
```

- **Soft union** (default): links tables and resolves on access
- **Hard union**: materializes merged columns

## CLI

Summarize a saved table:

```bash
unitok path/to/data
```

Add a feature into an existing table (integrate):

```bash
unitok integrate path/to/data --file data.tsv --column title --name title@bert \
  --vocab bert --tokenizer transformers --t.key bert-base-uncased
```

Remove a feature from a saved table:

```bash
unitok remove path/to/data --name title@bert
```

## Data Artifacts

Saved directories include:

- `meta.json` with schema, tokenizers, vocabularies
- `data.pkl` with tokenized columns
- `*.vocab` pickled vocabularies

## Migration From v3

If you have v3 artifacts:

```bash
unidep-upgrade-v4 <path>
```

## Notes and Constraints

- Key feature must be **atomic** (tokenizer returns a single id, not a list).
- Shared vocabularies must match for unions.
- `truncate=None` means an atomic feature; list features must use a truncate.
- `Feature` supersedes the deprecated `Job` class.

## Repository Layout (High-Level)

- `unitok/` core library
- `UniTokv3/` legacy v3 code
- `dist/` built distributions
- `setup.py`, `requirements.txt`

## License

MIT License. See `LICENSE`.