SensAI (Metadata Extraction)
Collection
Slovak-language models and datasets for extracting structured ML task requirements (TaskAttributes) from user inputs. • 7 items • Updated
You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
Access Request Terms:
By requesting access to the SensAI model, you confirm that you:
- will use the materials solely for research and non-commercial purposes;
- will cite the SensAI project and respect the CC-BY-NC-4.0 License;
- will not attempt to extract, infer, or reconstruct data from the model or dataset;
- will ensure that your downstream use complies with applicable laws, regulations, and ethical AI principles.
Log in or Sign Up to review the conditions and access this model content.
Qwen3.5-9B QLoRA Adapter
This repository contains a QLoRA adapter for Qwen3.5-9B.
Model was fine-tuned for instruction-following task ("extract metadata") using synthetic dataset called synthetic-conversations-and-ml-instructions.
Unlike the previous version, which extracted metadata from a single user query, this model extracts metadata from a multi-turn conversation between a user and an assistant in Slovak. The user may change their mind across turns and the model tracks all changes and uses only the most recent user-stated or user-confirmed preference for each attribute.
Description
- Base model: Qwen3.5-9B
- Adapter: QLoRA
- Task: extract metadata from multi-turn conversation
- Quantization: 4-bit
- GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
- Dataset: synthetic-conversations-and-ml-instructions.
LoRA Configuration
| Parameter | Value |
|---|---|
r |
32 |
lora_alpha |
16 |
lora_dropout |
0.05 |
bias |
none |
task_type |
CAUSAL_LM |
target_modules |
q_proj, k_proj, v_proj, o_proj |
Training Parameters
| Parameter | Value |
|---|---|
| Epochs | 2 |
| Learning rate | 7e-4 |
| Train batch size per device | 2 |
| Eval batch size per device | 2 |
| Gradient accumulation steps | 6 |
| Effective batch size | 12 |
| Weight decay | 0.0 |
| Optimizer | adamw_torch |
| Scheduler | cosine |
| Warmup ratio | 0.03 |
| BF16 | True |
| Save & Eval steps | 10 |
Resource Usage
| Metric | Value |
|---|---|
| Training time | 5h 44min |
| Peak memory (MiB) | 55083 / 97887 |
Evaluation Results
We evaluated the Qwen3.5-9B model and our fine-tuned QLoRA model on the test split (750 examples) of the synthetic-conversations-and-ml-instructions dataset.
Datapoint-level metrics
These metrics are computed over all test examples.
| Metric | Qwen3.5-9B | Qwen3.5-9B + QLoRA (Fine-tuned) |
|---|---|---|
| Invalidly parsed (%) | 19.467 | 0.400 |
| Complete accuracy (%) | 4.267 | 55.333 |
| Missing attributes (%) | 16.533 | 11.733 |
| Extra attributes (%) | 54.533 | 13.867 |
| Incorrect attributes (%) | 26.667 | 23.067 |
- Invalidly parsed: The percentage of examples where the model output had invalid/missing JSON format.
- Complete accuracy: The percentage of examples where all attributes in the output matched the ground truth attributes.
- Missing attributes: The percentage of examples where the model output is missing at least one attribute that is present in the ground truth example.
- Extra attributes: The percentage of examples where the model output contains attributes which are not present in the ground truth example.
- Incorrect attributes: The percentage of examples where the model output has incorrect attributes compared to the ground truth example.
The percentages for missing, extra and incorrect attributes may exceed 100% in total, since a single example can fall into multiple categories simultaneously. For instance, a model output could omit a required attribute (missing) while also adding an irrelevant one (extra).
Average field-level metrics
These metrics are computed per TaskAttributes field, only over examples whose output was validly parsed.
| Metric | Qwen3.5-9B | Qwen3.5-9B + QLoRA (Fine-tuned) |
|---|---|---|
| Accuracy (%) | 81.656 | 94.471 |
| Missing (%) | 2.417 | 1.352 |
| Extra (%) | 11.970 | 1.664 |
| Incorrect (%) | 3.534 | 2.584 |
- Accuracy percentage: The percentage of validly-parsed examples where the field's value matches the ground truth.
- Missing percentage: The percentage where the ground truth set this field but the prediction left it
null. - Extra percentage: The percentage where the ground truth left this field
nullbut the prediction set it. - Incorrect percentage: The percentage where both have a (non-null) value but they don't match.
Additional Information
This work was supported by the Výskumná Agentúra grant within the project SensAI - Morálna citlivosť a ľudské práva pre spracovanie jazykov s obmedzenými zdrojmi (Grant No. 09I01-03-V04-00100/2025/VA).
License & Attribution
This model was created within the SensAI project and is released under the CC-BY-NC-4.0 License. It is a derivative of the Qwen3.5-9B model with license: Apache license 2.0
Access Request Terms
Access to this repository is restricted. Please review and agree to the following terms before requesting access.
How to use
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch
base_model_name = "Qwen/Qwen3.5-9B"
ft_model_name ="kinit/qwen3.5-9B-extract-ml-conversations-qlora"
SYSTEM_PROMPT = """You are an ML requirements extraction specialist.
You will receive a multi-turn conversation in Slovak between a user seeking ML help and an AI assistant.
**Your task:** extract the user's FINAL ML task requirements and output them as a JSON object using the tool named `extract_metadata`
**Rules:**
- Treat USER messages as the source of truth. Assistant messages are context only.
- Read the ENTIRE conversation from start to finish before extracting.
- The user may change their mind across turns. Track ALL changes and use only the MOST RECENT user-stated or user-confirmed preference for each attribute.
- Use null when no direct or unambiguous value is supported."""
TOOL_SCHEMAS = [{
"type": "function",
"function": {
"name": "extract_metadata",
"description": "Extract structured ML task metadata from the conversation.",
"parameters": {
"type": "object",
"properties": {
"dataset_name": {"type": ["string", "null"]},
"dataset_modality": {"type": ["string", "null"], "enum": ["tabular", "text", "image", "audio", "video", "multimodal", None]},
"dataset_source": {"type": ["string", "null"], "enum": ["local", "huggingface", "zenodo", "kaggle", "web", "github", None]},
"data_format": {"type": ["string", "null"], "enum": ["csv", "json", "database", "zip", "parquet", "excel", "images_folder", "txt", None]},
"task_type": {"type": ["string", "null"], "enum": ["classification", "regression", "time_series", "clustering", "anomaly_detection", "recommendation", "ranking", "segmentation", "text_generation", None]},
"output_type": {"type": ["string", "null"], "enum": ["label", "probability_score", "numerical", "text", "ranking", "sentiment_score", "embeddings", "cluster_id", None]},
"max_runtime_secs": {"type": ["integer", "null"]},
"evaluation_metrics": {"type": ["array", "null"], "items": {"enum": ["accuracy", "precision", "recall", "f1_score", "roc_auc", "mse", "rmse", "mae", "r2", "confusion_matrix", "bleu_score", "rouge_score", "silhouette_score", "davies_bouldin_index", "ndcg", "map", "mrr"]}},
"excluded_models": {"type": ["array", "null"], "items": {"enum": ["linear_regression", "logistic_regression", "bayesian_regression", "decision_tree", "random_forest", "gradient_boosting", "xgboost", "knn", "kmeans", "naive_bayes", "neural_network", "cnn", "rnn", "lstm", "transformer"]}},
"included_models": {"type": ["array", "null"], "items": {"enum": ["linear_regression", "logistic_regression", "bayesian_regression", "decision_tree", "random_forest", "gradient_boosting", "xgboost", "knn", "kmeans", "naive_bayes", "neural_network", "cnn", "rnn", "lstm", "transformer"]}},
},
},
},
}]
tokenizer = AutoTokenizer.from_pretrained(ft_model_name)
quant_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
base_model = AutoModelForCausalLM.from_pretrained(
base_model_name,
quantization_config=quant_config,
device_map="auto",
)
model = PeftModel.from_pretrained(base_model, ft_model_name).eval()
conversation = (
"[USER]: Chcem realizovať klasifikáciu ŠPZ čísel áut pomocou CNN architektúry "
"za pomoci datasetu 'LicencePlates_ImageDataset'.\n\n"
"[ASSISTANT]: Rozumiem, použijete CNN na klasifikáciu ŠPZ. Aký formát majú "
"obrázky a koľko tried plánujete rozlišovať?\n\n"
"[USER]: Obrázky sú vo formáte JPG. Tried bude približne 30."
)
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Multi-turn conversation:\n\n {conversation}"},
]
text = tokenizer.apply_chat_template(
messages,
tools=TOOL_SCHEMAS,
tokenize=False,
add_generation_prompt=True,
enable_thinking=False,
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
im_end_id = tokenizer.convert_tokens_to_ids("<|im_end|>")
eos_token_ids = [tokenizer.eos_token_id, im_end_id]
generated_ids = model.generate(
**model_inputs,
max_new_tokens=1_024,
temperature=0.7,
eos_token_id=eos_token_ids,
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()
model_response = tokenizer.decode(output_ids, skip_special_tokens=True)
print(model_response)
- Downloads last month
- -