Alberich-mini-v1
Hugging Face repository: wahnfried/Alberich-mini-v1. An independent adapter for Qwen3.5-2B, not an official Qwen or Alibaba release.
A compact LoRA adapter for answering multiple-choice questions about text or an image, with English and German training data. It selects an answer directly from the model's final hidden state using the existing output projection. It does not generate a free-text response or a reasoning trace.
Adapter only: 16,819,200 parameters, 67.33 MB (64.21 MiB). Download the separate Qwen3.5-2B base model to use it. No base weights, tokenizer, optimizer state, training examples, or dataset images are distributed here.
The Hub dataset links identify the training sources hosted on Hugging Face. Only selected subsets of GLUE (SST-2) and The Cauldron were used. Additional CorefUD GUM/Potsdam sources and locally generated synthetic tasks, along with all subset counts and source terms, are documented in DATA_SOURCES.md. Evaluation-only datasets are listed in the benchmark sections below.
License: the publisher releases the adapter and original accompanying code/documentation under Apache 2.0, to the extent of the publisher's rights. Third-party notices retain their original terms. Every training task is listed with counts, provider credits, available license texts, citation links and modification notices in DATA_SOURCES.md. Several sources have noncommercial or share-alike conditions; some original terms and component rights remain unresolved. The Apache license does not relicense those sources or establish third-party commercial clearance. See LICENSING.md.
What it supports
- Text context, optionally one shared image, and 1–10 questions per request.
- 2–10 explicit answer options for each question, including yes/no or an explicit "insufficient information" option if appropriate.
- Zero-based selected option index and a softmax distribution over the supplied options.
- One forward pass for a single question. For multiple questions, a shared-prefix prefill followed by one batched suffix forward. This is two model calls, not one complete forward pass for an entire question catalogue.
- No autoregressive answer-token decoding and no cross-request cache.
The adapter modifies language-backbone attention, gated-delta and MLP projections. The base weights, vision encoder and output head were frozen during training. LoRA receives the intermediate activations of each adapted linear layer, adding its low-rank update inside the model; it is not a second model receiving a completed answer.
Quick start
Download this repository into a local directory and run from that directory. Install a CUDA-enabled PyTorch build matching your system, then pip install -r requirements.txt. The tested setup used Windows, Python 3.12, PyTorch 2.11.0+cu128 and an RTX 4070 SUPER (12 GB). The helper currently requires CUDA and BF16 support. CPU, other accelerators, quantized inference and hosted inference providers are not validated.
from inference import MultiDecider
decider = MultiDecider() # downloads pinned base, then merges local adapter in RAM
answers = decider.decide(
context="Anna hat drei rote Ordner. Ben hat zwei blaue Ordner.",
questions=[
{"question": "Wie viele Ordner hat Anna?", "options": ["Zwei", "Drei", "Fuenf"]},
{"question": "Welche Farbe haben Bens Ordner?", "options": ["Rot", "Blau"]},
],
)
print(answers) # list of {"choice": zero_based_index, "probabilities": [...]}
For image questions, add a PIL image:
from PIL import Image
with Image.open("document.png") as source:
image = source.convert("RGB")
answers = decider.decide(
context="",
image=image,
questions=[{"question": "Which word is visible?", "options": ["Invoice", "Receipt", "Neither"]}],
max_image_pixels=262144,
)
Use adapter_path="/path/to/adapter" to load another local copy, or a Hub repository ID. model_path="/path/to/base" and local_files_only=True allow offline loading. With no override, the base revision is pinned to 15852e8c16360a2fea060d615a32b45270f8a8fc.
The helper enforces 1536 tokens per complete question prompt, including image tokens. This is a limit of this validated inference implementation, not the base model's native context limit. The default image budget is 262144 pixels; selected OCR tasks used up to 1048576 pixels, subject to the same token limit. Split long documents into appropriate passages and retain the source context for review.
Scores are relative to the supplied answer options, not calibrated probabilities of truth. There is no built-in abstention mechanism. Providing an uncertainty option does not guarantee reliable abstention.
Training
The released checkpoint is a fresh rank-16 LoRA trained on 100000 mixed examples, then continued for 6000 example presentations with a fresh optimizer. The continuation includes replay, so the total is not 106000 unique examples. This is supervised cross-entropy training over answer choices, not reinforcement learning.
| Setting | Main stage | Short repair |
|---|---|---|
| Examples / epochs | 100000 / 1 | 6000 / 1 |
| LoRA rank / alpha | 16 / 16 | 16 / 16 |
| Learning rate | 0.00003 | 0.00001 |
| Warmup optimizer steps | 250 | 25 |
| Gradient accumulation | 8 | 8 |
| Optimizer | AdamW | AdamW, fresh state |
BF16 backbone computation, gradient checkpointing, no LoRA dropout. Adapter tensors are stored in FP32. Targets are the language model's q_proj, k_proj, v_proj, o_proj, in_proj_qkv, in_proj_z, in_proj_b, in_proj_a, out_proj, gate_proj, up_proj and down_proj; see the exact regex in adapter_config.json.
The main stage contains 75114 English and 24886 German examples, including 21494 image examples and 2000 synthetic examples. Tasks cover natural-language inference, coreference, paraphrase, intent, sentiment, commonsense, numeric/temporal/rule reasoning, diagrams, visual relations and adapted multiple-choice OCR/document questions. See training_summary.json for exact task counts and corpus hashes, and DATA_SOURCES.md for source and licensing notes.
In-memory merging (default)
The inference helper now defaults to merge_adapter=True: it loads the separate base and adapter, then calls PEFT merge_and_unload(safe_merge=True). This removes the separate LoRA operations during inference. No base or adapter files are modified, and this repository still distributes only the adapter.
Use MultiDecider(merge_adapter=False) to reproduce the historical unmerged evaluation mode. Merging into BF16 weights is not numerically identical: in the 1000-question paired test, two answers changed, both from incorrect to correct. Individual option probabilities changed by up to 0.311; probability thresholds may therefore behave differently. This small accuracy increase does not establish a general quality improvement.
Evaluation
These are local subset results, not official benchmark leaderboard scores. The fixed comparison has 1000 questions from 27 task families, with 37 per family and 38 for AI2D: 703 text and 297 image questions. It was sampled from held-out pools used during development, after training. It is not a newly collected blind evaluation. The earlier 200 pilot comparison cases were excluded. No further model tuning or calibration was performed on this 1000-question comparison.
| System, local implementation | Correct | Accuracy | Mean ms/question | Peak allocated GPU GiB |
|---|---|---|---|---|
| This adapter + Qwen3.5-2B, merged in RAM (default) | 867/1000 | 86.7% | 63 | See paired test below |
| This adapter + Qwen3.5-2B, unmerged (original run) | 865/1000 | 86.5% | 88 | 4.32 |
| Qwen3.5-2B base, same readout without adapter | 700/1000 | 70.0% | 65 | 4.26 |
| BARD-VL-4B, local OpenJev port | 809/1000 | 80.9% | 86 | 8.43 |
| DiffusionVL-3B, local OpenJev port | 737/1000 | 73.7% | 86 | 7.24 |
| Qwen3.5-4B, LitJev | 771/1000 | 77.1% | 164 | 8.78 |
The original unmerged adapter scored 587/703 (83.5%) on text, 278/297 (93.6%) on images, and 685/815 (84.0%) after excluding synthetic tasks. The merged variant scored 589/703 (83.8%) on text, 278/297 (93.6%) on images, and 687/815 (84.3%) after excluding synthetic tasks. Original aggregate per-task results are in eval_results.json; the merge comparison is in merged_adapter_results.json.
The unchanged Qwen3.5-2B base scored 700/1000 (70.0%) using the same prompt, readout, question order and image settings, with LoRA disabled. The original unmerged adapter improves overall accuracy by +16.5 percentage points; the merged default by +16.7 points. Base-only scores: 459/703 (65.3%) text and 241/297 (81.1%) images. On paired questions, the original unmerged adapter fixes 224 base errors and introduces 59 errors on cases the base answered correctly.
The base-only speed measurement was repeated as a standalone run with no concurrent model inference, using the same 1000 questions and timing boundaries. All 1000 selected answers matched the earlier base evaluation. Peak allocated memory is per benchmark process, not total GPU usage.
All systems received the same questions, options and option order. Images were converted to RGB and resized to fit 768×768 before native processing. Native prompts/processors and readout implementations differ. This is a system comparison on one desktop, not a controlled architecture comparison. BF16, RTX 4070 SUPER, one question per request; timings include tokenization, preprocessing and inference, exclude model loading and disk reading. Desktop load was uncontrolled. These measurements do not establish ten-question batch quality or speed.
The OCR subsets are local multiple-choice adaptations with distractors, not open-ended DocVQA/TextVQA evaluations. Near-ceiling scores should not be read as general OCR accuracy. Dataset SHA256: 88cc14e7201b39b16545cd1e7578358a1747d528138d458c4351b2d27ed4e974.
Paired merge speed test
A fresh sequential comparison in the same process measured 85.9 ms/question unmerged versus 63.5 ms merged, a 26.1% latency reduction, on the same 1000 questions. Accuracy was 865 versus 867 correct; 998 answers were identical. Merging took 0.22 seconds, excluding model loading. Peak allocated memory over the benchmark and the additional ten-question tests was 4.56 GiB unmerged and 4.50 GiB merged; these peaks have a different scope from the original single-question table.
| Input | Questions per request | Unmerged median ms | Merged median ms |
|---|---|---|---|
| Text | 1 | 57.7 | 49.1 |
| Text | 10 | 141.7 | 107.1 |
| Image | 1 | 112.0 | 98.1 |
| Image | 10 | 256.7 | 235.4 |
These latency examples use one fixed text context and one fixed image with distinct questions, three warmup requests and 20 measured requests per setting. They are not a representative multiquestion accuracy benchmark. All 22 selected answers matched between modes. Default image budget: 262144 pixels; the 1000-question comparison uses its original 1048576-pixel budget after a 768-pixel thumbnail. Timings include preprocessing and inference, not image-file reading or model loading. Ten questions take roughly 2.2–2.4 times the latency of one in these merged examples; they share a prefix pass and use one batched suffix pass.
Limitations and intended use
This is a decision-support prototype for reviewing text and images with explicit answer choices. A human should inspect the source material before acting on consequential findings. It has not been validated to establish criminal intent, legal suspicion or evidentiary conclusions.
Generalization is uneven: on the comparison's small subsets, WiC was 15/37 (40.5%), WSC 20/37 (54.1%), and ANLI 24/37 (64.9%). Other models outperform it on some families. On a separate 200-case WiC subset it scored 48%. It is not uniformly better than larger models.
English and German were directly trained and evaluated. Other languages inherited from the base model are not validated by this release. Long-document coherence, adversarial inputs, handwriting and difficult scans require separate evaluation. Option wording and order can affect answers. Exact overlap checks do not exclude semantic duplicates or exposure during base-model pretraining. Repeated use of development evaluation pools may bias model selection.
Artifact integrity
adapter_model.safetensors SHA256:
dc9dbc8cd11c65e5d8dcff8df7ceeb39d8f73fe454e7fa51b5c155790c506bdd.
The weights are unchanged from the evaluated short-repair checkpoint. The adapter configuration replaces the original local base path with the public base ID and pinned revision. release_manifest.json records every distributed file's size and hash; validation.json records local loading and inference checks.
Full public benchmark splits (zero-shot product readout)
These results and their domain breakdowns were measured with the unmerged adapter (merge_adapter=False). The full MMStar/MMLU suites have not been rerun with the merged default; the paired merge validation above covers the separate 1000-question comparison.
Complete splits, original option order, 0-shot, product chat prompt, direct A-J logits in FP32, BF16 backbone, no generation. MMStar RGB thumbnail <=768x768 then image budget 262144; full prompt <=1536 tokens, no truncation. Failures counted incorrect. Not standard MMLU 5-shot or a leaderboard submission.
| Dataset | N per model | Base | Adapter | Delta pp | Errors base / adapter |
|---|---|---|---|---|---|
| mmstar | 1500 | 51.47% | 60.73% | +9.27 | 0 / 0 |
| mmlu | 14042 | 56.69% | 60.85% | +4.15 | 0 / 0 |
MMLU reports question-weighted accuracy above; subject macro accuracy is included in the JSON. MMStar categories and MMLU subjects are also reported. Inference times cover model forward/readout only, with shared preprocessing excluded; they are not directly comparable to earlier end-to-end timings.
Overlap limitations
Exact normalized question and image-file bytes only; misses reworded questions and reencoded/resized images. Broad source-dataset overlap exists. Pretraining exposure unknown. Full scores retain all cases; additionally report subset excluding detected matches.
{ "mmstar": { "question_matches": 5, "image_file_matches": 0 }, "mmlu": { "question_matches": 1, "image_file_matches": 0 } }
Sources: MMLU, MMStar. Exact revisions and input hashes are in the accompanying manifest. No training or model selection was performed on these runs. These are local evaluations, not certified leaderboard entries.
Strengths and weaknesses by domain
These are descriptive results from the complete evaluated splits, not statistical significance claims. Absolute accuracy and improvement over the base model answer different questions: a large improvement can still leave substantial errors. Small domains and comparisons across many domains require caution. Training-source overlap and the zero-shot readout limitations above still apply.
MMStar: six visual capabilities
Highest adapter accuracy: coarse perception: 72.8% (182/250); instance reasoning: 69.2% (173/250); logical reasoning: 61.6% (154/250).
Lowest adapter accuracy: science & technology: 48.8% (122/250); fine-grained perception: 56.0% (140/250); math: 56.0% (140/250).
Largest gains over base: math: +17.20 pp (n=250); science & technology: +12.40 pp (n=250); logical reasoning: +10.80 pp (n=250).
Largest declines versus base: none observed.
| Domain | N | Base accuracy | Adapter accuracy | Delta pp |
|---|---|---|---|---|
| coarse perception | 250 | 66.80% | 72.80% | +6.00 |
| fine-grained perception | 250 | 52.40% | 56.00% | +3.60 |
| instance reasoning | 250 | 63.60% | 69.20% | +5.60 |
| logical reasoning | 250 | 50.80% | 61.60% | +10.80 |
| math | 250 | 38.80% | 56.00% | +17.20 |
| science & technology | 250 | 36.40% | 48.80% | +12.40 |
MMLU: 57 subject areas
Highest adapter accuracy: marketing: 83.3% (195/234); high school psychology: 82.9% (452/545); college biology: 79.9% (115/144).
Lowest adapter accuracy: moral scenarios: 30.8% (276/895); global facts: 32.0% (32/100); abstract algebra: 35.0% (35/100).
Largest gains over base: professional medicine: +30.51 pp (n=272); college biology: +13.19 pp (n=144); formal logic: +11.90 pp (n=126).
Largest declines versus base: abstract algebra: -6.00 pp (n=100); international law: -5.79 pp (n=121); global facts: -5.00 pp (n=100).
| Domain | N | Base accuracy | Adapter accuracy | Delta pp |
|---|---|---|---|---|
| abstract algebra | 100 | 41.00% | 35.00% | -6.00 |
| anatomy | 135 | 59.26% | 60.74% | +1.48 |
| astronomy | 152 | 69.08% | 75.66% | +6.58 |
| business ethics | 100 | 61.00% | 70.00% | +9.00 |
| clinical knowledge | 265 | 64.15% | 68.30% | +4.15 |
| college biology | 144 | 66.67% | 79.86% | +13.19 |
| college chemistry | 100 | 47.00% | 49.00% | +2.00 |
| college computer science | 100 | 45.00% | 54.00% | +9.00 |
| college mathematics | 100 | 31.00% | 37.00% | +6.00 |
| college medicine | 173 | 64.74% | 66.47% | +1.73 |
| college physics | 102 | 48.04% | 48.04% | +0.00 |
| computer security | 100 | 68.00% | 73.00% | +5.00 |
| conceptual physics | 235 | 64.26% | 70.21% | +5.96 |
| econometrics | 114 | 43.86% | 53.51% | +9.65 |
| electrical engineering | 145 | 66.90% | 62.76% | -4.14 |
| elementary mathematics | 378 | 43.12% | 50.26% | +7.14 |
| formal logic | 126 | 38.89% | 50.79% | +11.90 |
| global facts | 100 | 37.00% | 32.00% | -5.00 |
| high school biology | 310 | 73.55% | 79.68% | +6.13 |
| high school chemistry | 203 | 63.55% | 62.07% | -1.48 |
| high school computer science | 100 | 69.00% | 71.00% | +2.00 |
| high school european history | 165 | 69.70% | 72.73% | +3.03 |
| high school geography | 198 | 74.75% | 76.77% | +2.02 |
| high school government and politics | 193 | 67.88% | 78.76% | +10.88 |
| high school macroeconomics | 390 | 61.54% | 64.62% | +3.08 |
| high school mathematics | 270 | 31.48% | 41.85% | +10.37 |
| high school microeconomics | 238 | 68.49% | 72.69% | +4.20 |
| high school physics | 151 | 40.40% | 48.34% | +7.95 |
| high school psychology | 545 | 76.70% | 82.94% | +6.24 |
| high school statistics | 216 | 47.22% | 52.78% | +5.56 |
| high school us history | 204 | 67.65% | 69.61% | +1.96 |
| high school world history | 237 | 72.15% | 79.75% | +7.59 |
| human aging | 223 | 60.54% | 65.92% | +5.38 |
| human sexuality | 131 | 61.07% | 67.18% | +6.11 |
| international law | 121 | 76.86% | 71.07% | -5.79 |
| jurisprudence | 108 | 70.37% | 71.30% | +0.93 |
| logical fallacies | 163 | 66.87% | 65.03% | -1.84 |
| machine learning | 112 | 43.75% | 54.46% | +10.71 |
| management | 103 | 68.93% | 75.73% | +6.80 |
| marketing | 234 | 81.62% | 83.33% | +1.71 |
| medical genetics | 100 | 63.00% | 71.00% | +8.00 |
| miscellaneous | 783 | 68.07% | 71.39% | +3.32 |
| moral disputes | 346 | 60.98% | 64.45% | +3.47 |
| moral scenarios | 895 | 25.14% | 30.84% | +5.70 |
| nutrition | 306 | 63.40% | 65.36% | +1.96 |
| philosophy | 311 | 64.31% | 68.81% | +4.50 |
| prehistory | 324 | 61.11% | 66.67% | +5.56 |
| professional accounting | 282 | 46.81% | 49.29% | +2.48 |
| professional law | 1534 | 41.33% | 39.63% | -1.69 |
| professional medicine | 272 | 38.60% | 69.12% | +30.51 |
| professional psychology | 612 | 59.97% | 61.93% | +1.96 |
| public relations | 110 | 58.18% | 67.27% | +9.09 |
| security studies | 245 | 69.39% | 68.16% | -1.22 |
| sociology | 201 | 72.14% | 79.60% | +7.46 |
| us foreign policy | 100 | 72.00% | 73.00% | +1.00 |
| virology | 166 | 45.18% | 46.39% | +1.20 |
| world religions | 171 | 69.59% | 74.85% | +5.26 |
These domains do not constitute validation of forensic, medical or legal deployment. The earlier WiC/WSC findings remain documented separately; this section does not replace those known limitations.
- Downloads last month
- 35