metadata

title: Token Probability Analyzer
emoji: 🎲
colorFrom: blue
colorTo: red
sdk: docker
app_port: 7860

Token Probability Analyzer

This web application allows you to analyze token probabilities and predictions from various language models. It provides insights into how likely each token is according to the model, along with top predictions at each position.

Features

Analyze text using different language models (GPT-2, TinyLlama, etc.)
View token-by-token probabilities
See percentile scores for each token
Explore top-k predictions at each position
Calculate joint and average log-likelihood

Usage

Select a model from the dropdown menu
Enter your text in the input field
Click "Analyze" to see the results

Technical Details

Built with:

Flask
Hugging Face Transformers
PyTorch
JavaScript for interactive visualizations

Deployment

This app is deployed on Hugging Face Spaces using Docker.