README.md

---
title: Token Probability Analyzer
emoji: 🎲
colorFrom: blue
colorTo: red
sdk: docker
app_port: 7860
---

# Token Probability Analyzer

This web application allows you to analyze token probabilities and predictions from various language models. It provides insights into how likely each token is according to the model, along with top predictions at each position.

## Features

- Analyze text using different language models (GPT-2, TinyLlama, etc.)
- View token-by-token probabilities
- See percentile scores for each token
- Explore top-k predictions at each position
- Calculate joint and average log-likelihood

## Usage

1. Select a model from the dropdown menu
2. Enter your text in the input field
3. Click "Analyze" to see the results

## Technical Details

Built with:
- Flask
- Hugging Face Transformers
- PyTorch
- JavaScript for interactive visualizations

## Deployment

This app is deployed on Hugging Face Spaces using Docker.