README.md

metadata

title: LanguageBPE
emoji: 🌖
colorFrom: yellow
colorTo: gray
sdk: gradio
sdk_version: 4.37.1
app_file: app.py
pinned: false
license: mit

App to visualize results of BPE tokenizer trained on wikipedia and opus dataset.

Features

• Input/output text: Write some word(s) and see tokenized output
• Tokenizer: Select tokenizer

Usage

• Enter sentence in Hindi, English or both languages
• Select tokenizer model
• Hit submit and see result