Spaces:
Running
Running
| title: Token Probability Analyzer | |
| emoji: 🎲 | |
| colorFrom: blue | |
| colorTo: red | |
| sdk: docker | |
| app_port: 7860 | |
| # Token Probability Analyzer | |
| This web application allows you to analyze token probabilities and predictions from various language models. It provides insights into how likely each token is according to the model, along with top predictions at each position. | |
| ## Features | |
| - Analyze text using different language models (GPT-2, TinyLlama, etc.) | |
| - View token-by-token probabilities | |
| - See percentile scores for each token | |
| - Explore top-k predictions at each position | |
| - Calculate joint and average log-likelihood | |
| ## Usage | |
| 1. Select a model from the dropdown menu | |
| 2. Enter your text in the input field | |
| 3. Click "Analyze" to see the results | |
| ## Technical Details | |
| Built with: | |
| - Flask | |
| - Hugging Face Transformers | |
| - PyTorch | |
| - JavaScript for interactive visualizations | |
| ## Deployment | |
| This app is deployed on Hugging Face Spaces using Docker. |