semvec/pypi-packages huggingface dataset:
tokenkit 0.1.3
Calculate and visualize fertility and parity metrics for tokenizers
License: MIT
Requires Python: >=3.8

<h1 align="center">TokenKit</h1>

**TokenKit** is a Python library for computing **fertility** and **parity** metrics using large language model (LLM) tokenizers like LLaMA, T5, Gemma, and more. It helps you quantify and visualize how many tokens a sentence breaks into and how aligned a translation is in length across languages. **Model-agnostic** for any Hugging Face AutoTokenizer. Ideal for NLP evaluation, translation research, or just exploring how tokenizers behave across models.

---

## Installation
In the terminal:
```bash
pip install tokenkit
```

Import:
```python
from tokenkit import fertilize, paritize, TokenMetrics
# or
from tokenkit import *
```

---

## Examples

```python
# Fertility score: tokens per word
score, tokens = fertilize("I love classical music.")
print(score)
print(tokens)

# Parity score: token length ratio between original + translation
parity = paritize("Bonjour tout le monde", "Hello everyone")
print(parity)
```

```python
# Token metrics across a dataset
import pandas as pd
df = pd.DataFrame({
    "language": ["English", "French"],
    "text": ["This is a test sentence.", "Ceci est une phrase de test."],
    "translation": ["C'est une phrase de test.", "This is a test sentence."]
})

tm = TokenMetrics(data=df)
tm.fertilize(text_col="text", language_col="language")
tm.visualize_fertilities()
tm.paritize(text_col1="text", text_col2="translation")
```

Note that can use any Hugging Face AutoTokenizer:

```python
from transformers import AutoTokenizer
custom_tok = AutoTokenizer.from_pretrained("google/flan-t5-xxl")
fertilize("Tokenizing with a custom model.", tokenizer=custom_tok)
```

---

## License

MIT License © Bill & Melinda Gates Foundation 

Ada Zhang, Nihal Karim, Hamza Louzan, Victor Wei, Cody Carroll, Jessica Lundin
dbtype-checker 0.1.12
Detect mismatched column types & correct them in your database using SQLAlchemy .
License: MIT
Requires Python: >=3.7

# 🧠 dbtype_checker

A simple and lightweight tool to **detect columns stored in the wrong data type** in your database.  
It connects to your SQL database, samples data, and infers the **true data types** using intelligent pandas-based logic.

---

## 🚀 Features

- ✅ Detects `int`, `float`, `bool`, `datetime`, and `str`
- ✅ Identifies type mismatches based on actual data values (not just schema)
- ✅ Supports **MySQL**, **PostgreSQL**, and (optionally) **Oracle**
- ✅ Can **auto-fix** mismatched columns (via `ALTER TABLE`)
- ✅ Safe conversions with pre-cleaning (e.g., "true"/"false" to boolean)
- ✅ No LLMs or external APIs — **pure pandas + regex logic**

## 📦 Installation

Install via pip:

```bash
pip install dbtype_checker " OR" pip install dbtype-checker==0.1.10


""" 
## EXAMPLE USAGE

For postgres:

from dbtype_checker.checker import run_type_check_and_fix

# Detection
run_type_check_and_fix("postgresql+pg8000://user:password@localhost/database_name"dry_run=True)
# Correction
run_type_check_and_fix("postgresql+pg8000://user:password@localhost/database_name"dry_run=False)

For MySQL:

from dbtype_checker.checker import run_type_check_and_fix

# Detection
run_type_check_and_fix("mysql+pymysql://user:password@localhost:port/database", dry_run=True)
# Correction
run_type_check_and_fix("mysql+pymysql://user:password@localhost:port/database", dry_run=False)

For Oracle:
from dbtype_checker.checker import run_type_check_and_fix

db_url = "oracle+cx_oracle://username:password@host:1521/?service_name=your_service_name"
# Detection
run_type_check_and_fix(db_url, dry_run=True)

#Correction
run_type_check_and_fix(db_url,dry_run=False)   

For CLI:

# Detection
python -m dbtype_checker.checker --db_url "postgresql+pg8000://user:password@localhost/dbname" --dry_run true

#Correction
python -m dbtype_checker.checker --db_url "mysql+pymysql://user:password@localhost/dbname" --dry_run false

"""