dev-knowledge-bullshit-detector-v0.1

dev-knowledge-bullshit-detector-v0.1 estimates how much durable, reusable technical knowledge a Polish or English comment contains. It is intended for ranking and filtering technical text for later retrieval. Despite the informal repository name, this is not a fact-checker and does not determine whether a statement is true.

Output scale

The model produces an ordered distribution over five retrieval-value classes:

Class	Meaning
0	Noise, social, or administrative text with no technical knowledge
1	Technical vocabulary or transient status without reusable knowledge
2	Potentially useful but partial, borderline, or context-dependent technical content
3	A standalone, durable unit of technical knowledge
4	A detailed diagnosis, procedure, configuration, solution, or reasoned decision

Two continuous values can be derived from the distribution:

• usefulness_score: the expected class value divided by 4, in the range 0–1;
• probability_retrieval_value_ge_3: a separately calibrated probability that the text is a standalone, reusable technical unit (class 3 or 4).

The second value is usually the more convenient operational score for filtering. Its threshold should be selected for the target corpus rather than assumed to be 0.5.

Architecture

This model is a fully fine-tuned sdadas/polish-roberta-base-8k encoder with a five-logit sequence-classification head.

Property	Value
Architecture	RoBERTa for sequence classification
Parameters	190,297,349
Transformer layers	12
Hidden size	768
Attention heads	12
Feed-forward size	3,072
Vocabulary size	128,064
Classification labels	5 ordered classes
Base encoder context capacity	8,192 tokens
Fine-tuning and recommended inference length	256 tokens
Weight format	Safetensors, float32

Although the base encoder was pretrained primarily for Polish, this checkpoint was fine-tuned and evaluated on both Polish and English text.

Training data and procedure

The source material consisted of technical comments available under free licenses. English comments were translated into Polish, and the resulting language pairs were annotated for retrieval usefulness by Qwen3.5-27B. Original and translated versions of a comment were always kept in the same data split to prevent cross-language leakage.

The training split contained 10,762 examples: 5,381 Polish and 5,381 English. Separate validation and calibration splits contained 1,793 and 1,791 text instances drawn from both languages, respectively. The held-out test set consisted of 1,794 source pairs evaluated independently in Polish and English (3,588 text instances in total).

All model parameters were fine-tuned for five epochs, with the checkpoint from epoch 3 selected on validation PR AUC for the retrieval_value >= 3 boundary. The objective combined multiclass cross-entropy with a ranked probability score weighted by 0.25, so mistakes between distant classes were penalized more strongly. Training used balanced class weights, batch size 16, a learning rate of 2e-5, weight decay 0.01, 10% warmup, mixed precision, and seed 42. Temperature scaling was fitted on a dedicated calibration split, followed by monotonic Platt calibration for the class-3-or-higher boundary.

Evaluation

These results measure agreement with Qwen3.5-27B annotations on the test split. The test texts come from the same domain and were labeled with the same annotation procedure as the training data. Therefore, this evaluation can only indicate how closely the fine-tuned encoder-only model reproduces Qwen3.5-27B's annotations in-domain. It is not an independent expert evaluation and does not demonstrate factual correctness, general reasoning quality, or performance in other domains.

The following metrics were measured after calibration on Polish and English views of the same Qwen-annotated test pairs:

Language	Examples	Accuracy	Within ±1	MAE	Spearman	QWK	PR AUC (value >= 3)
Polish	1,794	0.777	0.921	0.427	0.850	0.847	0.886
English	1,794	0.756	0.909	0.438	0.846	0.838	0.879
Combined	3,588	0.767	0.915	0.433	0.848	0.842	0.882

Synthetic examples

All texts below were written specifically for this model card and are not training or evaluation examples. Values were produced locally by this exact checkpoint. P(value >= 3) includes the saved temperature and Platt calibration. The examples were deliberately kept at comparable lengths so that score differences cannot be explained merely by short versus long inputs.

Polish

Text	Predicted class	Usefulness score	P(value >= 3)
Sprawdziłem konfigurację połączenia, wersję biblioteki i wyniki wszystkich testów integracyjnych. Nie udało mi się odtworzyć problemu na najnowszej gałęzi, więc zamykam zgłoszenie. Jeśli błąd pojawi się ponownie, proszę utworzyć nowe zgłoszenie z aktualnymi logami.	0	0.032	0.009
Przejrzałem ostatnie zmiany w module pamięci podręcznej, porównałem wyniki testów i ponownie uruchomiłem nieudane zadanie. Teraz wszystko wygląda poprawnie, nie widzę już błędów w logach, a gałąź jest gotowa do scalenia po akceptacji recenzenta.	1	0.243	0.028
Problem prawdopodobnie dotyczy pamięci podręcznej albo kolejności inicjalizacji usług. Warto wyczyścić stare wpisy, ponownie uruchomić proces i sprawdzić konfigurację połączenia. Jeśli to nie pomoże, potrzebne będą pełniejsze logi oraz informacje o wersji używanego środowiska.	1	0.395	0.199
Po zmianie schematu stare wpisy pamięci podręcznej nie zawierają pola version, dlatego deserializator odrzuca je podczas startu. Dodaj migrację ustawiającą wartość domyślną albo unieważnij wpisy z poprzednim numerem schematu przed wdrożeniem nowej wersji aplikacji.	3	0.614	0.621
Nagłówek Cache-Control: no-cache nie zabrania zapisania odpowiedzi w pamięci podręcznej, lecz wymaga ponownej walidacji przed jej użyciem. Gdy odpowiedź w ogóle nie może zostać zapisana, należy zastosować no-store; private ogranicza natomiast przechowywanie do prywatnej pamięci klienta.	3	0.788	0.897

English

Text	Predicted class	Usefulness score	P(value >= 3)
I checked the connection configuration, library version, and every integration test result. I cannot reproduce the problem on the latest branch, so I am closing the report. If it happens again, please open a new report with current logs.	0	0.038	0.010
I reviewed the recent cache changes, compared the test results, and reran the failed workflow. Everything looks correct now, no errors remain in the logs, and the branch is ready to merge after another reviewer gives final approval.	1	0.237	0.032
The issue probably involves caching or service initialization order. Try deleting old entries, restarting the process, and checking the connection configuration. If that does not help, we will need complete logs and details about the exact environment and version.	1	0.355	0.145
Old cache entries lack the version field introduced by the new schema, so deserialization fails during startup. Add a migration that supplies a default value, or invalidate entries carrying the previous schema number before deploying the updated application.	3	0.684	0.779
Cache-Control: no-cache does not prohibit storing a response; it requires successful revalidation before reuse. When a response must never be stored, use no-store instead. The private directive permits storage only in a private cache and prevents reuse by shared intermediary caches.	3	0.764	0.879

Usage

Install the runtime dependencies:

pip install torch transformers huggingface_hub

The repository includes predict.py, which applies both calibration stages and returns the class distribution, expected retrieval value, normalized usefulness score, and calibrated operational probability:

python predict.py \
  --model jansowa/dev-knowledge-bullshit-detector-v0.1 \
  "The cache entry is stale because the invalidation callback runs before the transaction commits."

For the raw five-class output, the standard Transformers API is sufficient:

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="jansowa/dev-knowledge-bullshit-detector-v0.1",
    top_k=None,
)

predictions = classifier(
    "Po zmianie schematu usuń stare pliki cache.",
    truncation=True,
    max_length=256,
)
print(predictions)

The standard pipeline returns uncalibrated class probabilities. Use the included script when the calibrated P(value >= 3) is needed.

Limitations

• This model estimates retrieval usefulness, not factual correctness, safety, code quality, or author intent.
• It was trained on technical comments; results may not transfer to documentation, chat, prose, source code, or other domains.
• Inputs longer than 256 tokens are truncated.
• English is supported through bilingual fine-tuning, but the underlying encoder is Polish-first.
• The training labels are model-generated annotations and can contain systematic errors or bias.
• Scores are not expected to be uniformly distributed across 0–1. Validate thresholds on data representative of the intended use.

License

Released under the Apache License 2.0, matching the license of the base model.