Tallie 1.0

Tallie reads the thread so you don't have to.

Ask for a kind of thing and an opinion ("podcasts people recommend", "most hated books", "video games people loved") and point Tallie at thousands of forum comments. It reads every comment once, picks out each named item, works out what kind of thing it is and how the commenter feels about it, keeps the items that match your question, and tallies them into a ranked list.

"podcasts people recommend"   +   24,000 Hacker News comments
                    ↓
 1. Planet Money            16 mentions   positive 16
 2. Radiolab                13 mentions   positive 13
 3. EconTalk                13 mentions   positive 13
 4. This American Life      13 mentions   positive 13
 5. Darknet Diaries         16 mentions   positive 14, neutral 1, mixed 1

How it answers a question

1. Read. Every comment goes through the model once.
2. Extract. For each named item in the comment, the model returns the exact words, the type (book, movie, TV show, video game, board game, podcast, music, software, product, person) and the commenter's opinion (positive, negative, neutral, mixed).
3. Match. "Podcasts people recommend" becomes type = podcast, opinion = positive. Extra words in the question ("for beginners", "about space") are checked by the model's relevance head, in the context of each comment.
4. Tally. Matching items are counted across all comments (one vote per comment) and ranked.

What makes it different

• It reads; it doesn't write. Tallie began as a generative language model, but the text-writing part was removed. Each comment takes a single forward pass and nothing is generated token by token. That's why it processes 130 comments per second on one NVIDIA L4.
• Every answer comes from the text. Items are spans copied directly from the comments, so the model can't invent a book or a podcast that nobody mentioned.
• The output is structured, not prose. Each mention comes back as item + type + opinion, ready to count, filter and rank.
• It understands your question in context. The question is fed into the model alongside each comment, rather than being matched by keywords.

How it was built

Qwen3-1.7B (28 layers, generative)
   ✂  language-model head removed       → it can no longer write text
   ✂  top 8 layers pruned (20 kept)     → smaller and faster
   ↻  causal → bidirectional attention  → every token sees the whole comment and the question
   +  4 new heads, trained with LoRA:
        mention head    which words are a named item?     (BIO tags)
        relevance head  does this item fit the question?  (conditioned on the question)
        opinion head    positive / negative / neutral / mixed
        type head       book / movie / tv_show / video_game / board_game / podcast /
                        music / software / product / person / other
   =  1.33B parameters, ~2.5 GB (bf16)

Results

All numbers are on a hand-labeled test set: 150 Hacker News comments, 224 named items and 165 questions. One third of the comments come from video-game threads, a topic excluded from training.

	Tallie 1.0
Finding named items (F1)	0.86 (recall 0.94)
Item type accuracy	0.91
Right items for the question, end-to-end F1	0.68
↳ on the unseen games topic	0.65
Opinion accuracy	0.64
Speed	130 comments/s on one NVIDIA L4

How we got here

Version	Change	Names F1	End-to-end F1	Lesson
0.1	Labels from a Qwen3-8B teacher	~0.50	0.50	The teacher's own labels were only 0.55 F1 against human labels. The student can't beat its teacher.
0.2	Labels from a stronger teacher (0.88 F1), junk filters, span fixes	0.85	0.59	Clean labels raised the ceiling more than any model change.
1.0	Type head, balanced opinion loss, more "nothing matches" examples, tuned cutoff, 3 epochs	0.86	0.68	Knowing what kind of thing an item is fixed most wrong answers.

Example output (all 24,000 comments)

Podcasts people recommend: Planet Money · Radiolab · EconTalk · This American Life · Darknet Diaries · 99% Invisible · Hardcore History · Startup

Most hated books: Beacon 23 · Fall; or, Dodge in Hell · Beelzebub's Tales to His Grandson · The Glass Bead Game · 1Q84

TV shows people loved: Triumph of the Nerds · Cosmos · Planet Earth · Tales from the Loop · Taskmaster · Bluey

Usage

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("pateluday/tallie")
sys.path.insert(0, path)

from tallie import Tallie

t = Tallie.load(path)               # GPU if available; ~4 GB VRAM is enough
t.index(list_of_comment_strings)    # one pass over your texts
print(t.ask("podcasts people recommend"))
print(t.ask("most hated books"))

# raw output: every named item with its type and opinion
t.extract(["I loved Dune but Snow Crash bored me."])

Requirements: torch, transformers>=4.55. A runnable example is in example.py.

Best at / not yet good at

Best at: "kind of thing + opinion" questions: "podcasts people recommend", "most hated books", "video games people loved", "programming languages people recommend".

Not yet reliable: genre and topic questions that need outside knowledge about each item, such as "funny movies" or "books about King Arthur".

Limitations

• Domain: trained on Hacker News. Strongest on software, books and tech culture; weaker on music, TV and games.
• Opinions lean positive (the training labels did), and sarcasm is hard. Opinion accuracy is 0.64.
• Types on unseen topics are less reliable (0.80 on games vs 0.96 on familiar topics), and a nearby word can sway the type when two items share a sentence.
• English only. Very long comments are split into chunks.
• Counts reflect what a forum's users say, not ground truth about quality.

Training data

About 6,000 public Hacker News comments (collected via the Algolia HN API), labeled for names, types, opinions and topics by an LLM teacher. Training questions were generated from those labels. Raw comments are not redistributed here.

Roadmap

• Smaller variants (12–16 layers, or a ~150M encoder), with a measured size/accuracy trade-off
• Better opinion labels, more domains (Reddit, product reviews), genre-aware ranking
• 4-bit export (~0.6 GB)

Credits

Built on Qwen3-1.7B (Apache 2.0). Related ideas: GLiNER (span extraction), LLM2Vec (turning LLMs into encoders), aspect-based sentiment analysis, knowledge distillation.