LUAR-MUD-MV-Qwen

Author Style Representations using LUAR. Embeds an author's writing so that two disjoint samples by the same person land near each other.

This is the multi-vector successor to rrivera1849/LUAR-MUD, built on Qwen3-0.6B and trained on the same Million User Dataset (1.07M Reddit authors). Two things changed relative to the original LUAR:

1. A bigger backbone. DistilRoBERTa (82M) --> Qwen3-0.6B, with last-token pooling.
2. A multi-vector head. Instead of collapsing an author's episode into one vector, the model keeps one vector per text and compares two authors with MaxSim (ColBERT-style late interaction). A single vector is still available as the mean of those vectors.

On the MUD test split this improves MRR from 0.5706 to 0.6564 over the released LUAR-MUD.

---

Quickstart

import torch
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("rrivera1849/LUAR-MUD-MV-Qwen", trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained("rrivera1849/LUAR-MUD-MV-Qwen")

# One "episode" per author: a handful of short texts they wrote.
episodes = [
    ["finally beat the final boss after nine tries", "my controller did not survive"],
    ["the amendment was struck down on procedural grounds", "worth reading the dissent"],
    ["nine tries on that boss and my thumbs hurt", "genuinely the hardest fight in the game"],
]

with torch.no_grad():
    embeddings = model.encode(episodes, tokenizer)        # [3, 2, 512], one vector per text

scores = model.similarity(embeddings[:1], embeddings[1:])
print(scores)   # tensor([[0.3802, 0.5786]]) -- episode 2, the same author as 0, wins

For GPU inference, load in bfloat16: AutoModel.from_pretrained(..., trust_remote_code=True, dtype=torch.bfloat16).cuda().

Two embedding modes, one argument

forward and encode take output_type:

output_type	Shape	What it is
"multi" (default)	[B, E, 512]	One L2-normalized vector per text; compared with MaxSim.
"single"	[B, 512]	The L2-normalized mean of those vectors; compared with cosine.
"both"	tuple	Both, from a single forward pass.

multi  = model.encode(episodes, tokenizer)                          # [B, E, 512]
single = model.encode(episodes, tokenizer, output_type="single")    # [B, 512]
multi, single = model.encode(episodes, tokenizer, output_type="both")

The single-vector option might be used for ANN indexes (FAISS, pgvector, Qdrant) where one vector per author is the only option. It costs about 4% of the multi-vector MRR (0.6283 vs 0.6564 on the MUD test split, see below). It might also be used as the first stage of a two-stage retrieval pipeline (see below).

One similarity function

model.similarity(queries, targets) dispatches on the rank of what you give it and always returns a [n_queries, n_targets] score matrix, higher meaning more similar.

model.similarity(multi_q, multi_t)    # [Nq, E, 512] x [Nt, E, 512] -> MaxSim
model.similarity(single_q, single_t)  # [Nq, 512]    x [Nt, 512]    -> cosine

Two details worth knowing:

• MaxSim is asymmetric here, because that is how the model was trained: the score averages the best match of each query text, so similarity(a, b) != similarity(b, a).T. Pass symmetric=True for the Chamfer-style average of both directions.
• Padding is inferred, not declared. Episodes in a batch may have different numbers of texts; short ones are zero-padded and those slots are ignored by both similarity and the single-vector mean. You never pass a mask.

For retrieval, model.retrieve(queries, targets, k=10) returns (scores, indices), best first.

Two-stage retrieval

Late interaction is accurate but quadratic in texts. At corpus scale, use the single vector to shortlist and the multi-vector head to rerank (pseudocode, bring your own index):

index_vectors = model.encode(corpus_episodes, tokenizer, output_type="single")   # into your ANN index
candidates = ann_index.search(query_single, k=1000)                              # cheap recall
scores = model.similarity(query_multi, corpus_multi[candidates])                 # accurate ranking

Training

Data	Million User Dataset (MUD), 1.07M Reddit authors
Objective	Supervised contrastive (SupCon) over episode-level MaxSim, temperature 0.02, asymmetric
Episodes	up to 16 texts (random size per batch, Beta(3,1)), 32 tokens each, 2 episodes per author per step
Batch	256 authors per GPU x 7 GPUs = 1792 authors per contrastive batch, via GradCache
Schedule	20 epochs, lr 2e-5 (flat, no batch scaling), 100 warmup steps, cosine decay

Results

MUD test split (using 32 tokens per text) (25,000 queries against 111,396 targets):

Model	Readout	MRR	R@1	R@8	R@64
LUAR-MUD-MV-Qwen (this model)	multi-vector, MaxSim	0.6564	0.5792	0.7854	0.9068
LUAR-MUD-MV-Qwen (this model)	single vector, cosine	0.6283	0.5481	0.7622	0.8974
LUAR-MUD (released)	single vector, cosine	0.5706	0.4904

The single vector keeps 96% of the multi-vector MRR on the same episodes, and on its own still beats the released LUAR-MUD by a wide margin.

Citation

@inproceedings{uar-emnlp2021,
  author    = {Rafael A. Rivera-Soto and Olivia Elizabeth Miano and Juanita Ordonez and
               Barry Y. Chen and Aleem Khan and Marcus Bishop and Nicholas Andrews},
  title     = {Learning Universal Authorship Representations},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year      = {2021},
}

Late interaction follows ColBERT (Khattab and Zaharia, 2020), applied over the texts of an episode rather than the tokens of a document.