semvec/pypi-packages huggingface dataset:
toknc 0.1.1
A Python CLI application that estimates token usage for files in a directory
License: MIT
Requires Python: >=3.7
Keywords: token, gpt, openai, cli, code-analysis

# Token Counter CLI

A Python CLI application that estimates token usage for files in a directory, helping you understand how much context window space your codebase would consume.

## Features

- 📊 Analyzes token usage for all text files in a directory
- 🤖 Supports multiple OpenAI model encodings
- 📏 Shows context window usage percentages for popular models
- 🔝 Ranks files by token consumption
- 🚫 Automatically excludes binary files and common build/cache directories
- 📋 Supports file inclusion/exclusion patterns
- 📄 JSON output option for integration
- ⚠️ Fallback token estimation when tiktoken is not available

## Installation

```bash
# Install via pip
pip install toknc

# For accurate token counting, install tiktoken:
pip install toknc[tiktoken]
```

Note: The tool will work without tiktoken using rough estimation (≈1 token per 4 characters).

## Usage

```bash
# Analyze current directory
toknc

# Analyze specific directory
toknc /path/to/project

# Use specific model encoding
toknc --model gpt-4

# Only analyze Python files
toknc --include "*.py"

# Exclude test files
toknc --exclude "test_*" --exclude "*_test.py"

# Show all files in detail
toknc --details

# Output as JSON
toknc --json

# Show top 20 files
toknc --top 20
```

## Options

- `directory`: Directory to analyze (default: current directory)
- `--model`: OpenAI model for token encoding (default: gpt-3.5-turbo)
- `--include`: File patterns to include (can be used multiple times)
- `--exclude`: File patterns to exclude (can be used multiple times)
- `--top`: Number of top files to show (default: 10)
- `--details`: Show all files in detail
- `--json`: Output results as JSON

## Example Output

```
📊 Token Analysis for: /Users/adminx/test
🤖 Model: gpt-3.5-turbo
📁 Total files: 4
🪙 Total tokens: 2,897
💾 Total size: 11.4 KB

📏 Context Window Usage:
  GPT-3.5 Turbo     4096 tokens: ██████████████░░░░░░ 70.7%
  GPT-4             8192 tokens: ███████░░░░░░░░░░░░░ 35.4%
  GPT-4 Turbo     128000 tokens: ░░░░░░░░░░░░░░░░░░░░ 2.3%
  Claude 3        200000 tokens: ░░░░░░░░░░░░░░░░░░░░ 1.4%
  Gemini Pro       32768 tokens: █░░░░░░░░░░░░░░░░░░░ 8.8%

🔝 Top 4 files by tokens:
   1. token_counter.py
      Tokens: 2,877 (99.3%)
      Size: 11.3 KB
```

## Supported File Types

The tool automatically detects text files and includes common source code extensions:
- Python (.py), JavaScript (.js, .jsx), TypeScript (.ts, .tsx)
- Web files (.html, .css)
- Config files (.json, .xml, .yaml, .yml)
- Documentation (.md, .rst)
- Shell scripts (.sh, .bash, .zsh)
- C/C++ (.c, .cpp, .h, .hpp)
- Java (.go, .rs, .php, .rb)
- And many more...

## Automatically Excluded

- Version control: .git/, .svn/, .hg/
- Dependencies: node_modules/, __pycache__/
- Compiled files: .pyc, .pyo, .pyd
- System files: .DS_Store, Thumbs.db
wiki2video 0.0.1a1
Wiki2Video: turn Wikipedia articles into short documentary-style videos
License: MIT
Requires Python: >=3.9
Keywords: ai, video, llm, wikipedia, cli, generative-ai

# 📚➡️🎬 Wiki2Video
---
![Python](https://img.shields.io/pypi/pyversions/wiki2video)
![PyPI version](https://img.shields.io/pypi/v/wiki2video)
![License](https://img.shields.io/github/license/NPgreatest/wiki2video)


> **From Wikipedia to TikTok/Shorts in One Command.**

Wiki2Video is an AI-powered, fully automated pipeline that turns any Wikipedia article into a narrated, subtitled, cinematic video — ready for YouTube, Shorts, TikTok, or Bilibili.

**No UI.
No timelines.
No manual editing.**

Just one command:

```bash
wiki2video generate https://en.wikipedia.org/wiki/Rongorongo
```

Wiki2Video will automatically:

* Fetch the article
* Summarize and rewrite it into a script
* Generate narration (TTS)
* Generate scenes (text-to-video or images)
* Build subtitles
* Render the final mp4


[English](README.md) | [简体中文](README_cn.md)

---

## 🎬 Example Output

| [Oak Island Mystery · Video Mode](https://www.youtube.com/shorts/QA5oeompLAU) | [Fermi Paradox · Image Mode](https://www.youtube.com/shorts/QU2pmhpgsU0) | [Voynich Manuscript · Landscape](https://www.youtube.com/watch?v=0eWaZLgr14M&t=153s) |
|-------------------------------------------------------------------------------|---------------------------------------------------------------------------|-------------------------------------------------------------------------------------|
| <img src="/example/video/example1.gif" width="260">                            | <img src="/example/video/example2.gif" width="260">                       | <img src="/example/video/example3.gif" width="260">                                  |

---

## 🛠️ Installation

> ⚠️ Wiki2Video is currently in **alpha** and published on **TestPyPI**.

```bash
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple wiki2video
```

### OpenAI Backend (Recommended)

Install Wiki2Video with OpenAI support:

```bash
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple wiki2video[openai]
```

---

## 🚀 Quick Start

Initialize configuration and set your OpenAI API key:

```bash
wiki2video init
```

Generate your first video:

```bash
wiki2video generate <wikipedia_url>
```


---

# License
<p>
  <img src="example/picture/W2V.png" width="80"/>
</p>

Copyright 2025 NPgreatest

Distributed under the terms of the MIT license.