jgdo/photo-query-tagger

Photo search query tagger

A small multilingual word tagger that splits a photo search query into people, places and visual description. Each of these can be wanted or excluded. It parses the search box of a self-hosted photo library: "Anna ohne Tom am Strand, nicht in Thailand" means photos of Anna, without Tom, at the beach, not taken in Thailand.

It is microsoft/Multilingual-MiniLM-L12-H384 fine-tuned for token classification on generated queries. It is exported to ONNX and quantised to int8, and its vocabulary is cut from 250k to 58k pieces.

File	Size
query_tagger.onnx	45 MB	ONNX, int8 weights; inputs input_ids, attention_mask; output logits (batch × tokens × 14)
query_tagger_tokenizer.json	4 MB	Hugging Face tokenizers file (XLM-R SentencePiece, pruned vocabulary)
labels.json		label names in logit order

Labels

Label	Meaning	Example
B-PER+ / I-PER+	a wanted person	Tom am Strand
B-PER- / I-PER-	an excluded person	Anna ohne Tom
B-LOC+ / I-LOC+	a wanted place	sunset in Rio de Janeiro
B-LOC- / I-LOC-	an excluded place	beach not in Thailand
B-TXT+ / I-TXT+	wanted visual content	Tom am Strand
B-TXT- / I-TXT-	excluded visual content	beach without people
ALONE	only these people	Anna allein, only Tom
O	operators and fillers	photos of, without, aber nicht, и

Usage

The model labels words, not subword tokens. Split the query into words, encode them as pre-tokenized input, and read the label from the first subword of each word. Words are split as follows:

• whitespace separates words;
• every CJK or kana character is a word of its own;
• runs of letters, digits and combining marks are words;
• every other character (punctuation, apostrophes) is a word of its own.

import json, numpy as np, onnxruntime as ort
from tokenizers import Tokenizer

tok = Tokenizer.from_file("query_tagger_tokenizer.json")
sess = ort.InferenceSession("query_tagger.onnx")
labels = json.load(open("labels.json"))

words = ["Anna", "ohne", "Tom", "am", "Strand", ",", "nicht", "in", "Thailand"]
enc = tok.encode(words, is_pretokenized=True)
logits = sess.run(None, {"input_ids": np.array([enc.ids]), "attention_mask": np.array([enc.attention_mask])})[0][0]
seen = set()
for i, w in enumerate(enc.word_ids):
    if w is not None and w not in seen:
        seen.add(w)
        print(words[w], labels[int(logits[i].argmax())])
# Anna B-PER+ / ohne O / Tom B-PER- / am B-TXT+ / Strand I-TXT+ / , O / nicht O / in O / Thailand B-LOC-

In the photo library, fixed rules follow the tagger. Person spans are matched against the names known to the library, and place spans are looked up in a gazetteer. A "place" the gazetteer doesn't know goes back into the visual description. The model itself only labels words.

Training data

All training data is generated; no user queries were used.

• Queries: built from templates in 13 languages (en de fr es it pt nl ru pl tr ja zh ar), with case, punctuation and typo noise.
• Names: from many cultures, including Russian and Polish case forms ("Анны", "Tomkiem").
• Places: from Natural Earth (public domain), including multi-word and localised names.
• Visual phrases: including ones that collide with place names ("nice beach" / "Nice", "turkey with gravy" / "Turkey").

There are 200k training queries. The released model was trained on all of the data for 3 epochs (batch 64, lr 5e-5).

Evaluation

The numbers are span F1 and exact match per query. The held-out set uses templates, names, places and phrases that the training data never contains.

Model (same data)	int8 size	Held-out F1	Held-out exact
Multilingual-MiniLM-L12-H384 (this model)	45 MB (pruned)	0.944	0.852
xlm-roberta-base	279 MB	0.945	0.854
distilbert-base-multilingual-cased	135 MB	0.934	0.818

On 95 hand-written test queries the photo library gets all 95 right, tagger and rules together. The data generator was shaped with those queries, so treat that as a regression test rather than a benchmark. On a Ryzen 7 5800X with 2 threads a query takes 2 ms (3.8 ms worst) and the model adds about 166 MB of memory.

Limitations

• Narrow purpose: the model is made for short photo search queries. Long sentences, other tasks or other domains are out of scope.
• Template-shaped: phrasings far from the templates can be mislabelled, for example a person tagged as visual content, or the scope of an exclusion misread.
• Other languages: they may work through the multilingual base model but are untested.