You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Access Request Terms:
By requesting access to the SensAI model, you confirm that you:
- will use the materials solely for research and non-commercial purposes;
- will cite the SensAI project and respect the CC-BY-NC-4.0 License;
- will not attempt to extract, infer, or reconstruct data from the model or dataset;
- will ensure that your downstream use complies with applicable laws, regulations, and ethical AI principles.

Qwen3.5-0.8B Full Fine-Tuned

This repository contains a fully fine-tuned Qwen3.5-0.8B model.

Model was fine-tuned for instruction-following task ("extract metadata") using synthetic dataset called synthetic-conversations-and-ml-instructions.

Unlike the previous version, which extracted metadata from a single user query, this model extracts metadata from a multi-turn conversation between a user and an assistant in Slovak. The user may change their mind across turns and the model tracks all changes and uses only the most recent user-stated or user-confirmed preference for each attribute.

Description

Base model: Qwen3.5-0.8B
Method: Full fine-tuning
Task: extract metadata from multi-turn conversation
GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
Dataset: synthetic-conversations-and-ml-instructions.

Training Parameters

Parameter	Value
Epochs	2
Learning rate	3e-5
Train batch size per device	4
Eval batch size per device	4
Gradient accumulation steps	6
Effective batch size	24
Weight decay	0.1
Optimizer	adamw_torch
Scheduler	cosine
Warmup ratio	0.03
BF16	True
Save & Eval steps	10

Resource Usage

Metric	Value
Training time	1h 27min
Peak memory (MiB)	75949 / 97887

Evaluation Results

We evaluated the Qwen3.5-0.8B model and our fully fine-tuned model on the test split (750 examples) of the synthetic-conversations-and-ml-instructions dataset.

Datapoint-level metrics

These metrics are computed over all test examples.

Metric	Qwen3.5-0.8B	Qwen3.5-0.8B (Fine-tuned)
Invalidly parsed (%)	42.000	0.267
Complete accuracy (%)	0.000	50.800
Missing attributes (%)	26.533	14.400
Extra attributes (%)	50.267	15.200
Incorrect attributes (%)	50.000	28.133

Invalidly parsed: The percentage of examples where the model output had invalid/missing JSON format.
Complete accuracy: The percentage of examples where all attributes in the output matched the ground truth attributes.
Missing attributes: The percentage of examples where the model output is missing at least one attribute that is present in the ground truth example.
Extra attributes: The percentage of examples where the model output contains attributes which are not present in the ground truth example.
Incorrect attributes: The percentage of examples where the model output has incorrect attributes compared to the ground truth example.

The percentages for missing, extra and incorrect attributes may exceed 100% in total, since a single example can fall into multiple categories simultaneously. For instance, a model output could omit a required attribute (missing) while also adding an irrelevant one (extra).

Average field-level metrics

These metrics are computed per TaskAttributes field, only over examples whose output was validly parsed.

Metric	Qwen3.5-0.8B	Qwen3.5-0.8B (Fine-tuned)
Accuracy (%)	41.701	93.262
Missing (%)	9.379	1.725
Extra (%)	28.046	1.858
Incorrect (%)	20.874	3.155

Accuracy percentage: The percentage of validly-parsed examples where the field's value matches the ground truth.
Missing percentage: The percentage where the ground truth set this field but the prediction left it null.
Extra percentage: The percentage where the ground truth left this field null but the prediction set it.
Incorrect percentage: The percentage where both have a (non-null) value but they don't match.

Additional Information

This work was supported by the Výskumná Agentúra grant within the project SensAI - Morálna citlivosť a ľudské práva pre spracovanie jazykov s obmedzenými zdrojmi (Grant No. 09I01-03-V04-00100/2025/VA).

License & Attribution

This model was created within the SensAI project and is released under the CC-BY-NC-4.0 License. It is a derivative of the Qwen3.5-0.8B model with license: Apache license 2.0

Access Request Terms

Access to this repository is restricted. Please review and agree to the following terms before requesting access.

How to use


from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

ft_model_name = "kinit/qwen3.5-0.8B-extract-ml-conversations"

SYSTEM_PROMPT = """You are an ML requirements extraction specialist.
You will receive a multi-turn conversation in Slovak between a user seeking ML help and an AI assistant.

**Your task:** extract the user's FINAL ML task requirements and output them as a JSON object using the tool named `extract_metadata`

**Rules:**
- Treat USER messages as the source of truth. Assistant messages are context only.
- Read the ENTIRE conversation from start to finish before extracting.
- The user may change their mind across turns. Track ALL changes and use only the MOST RECENT user-stated or user-confirmed preference for each attribute.
- Use null when no direct or unambiguous value is supported."""

TOOL_SCHEMAS = [{
    "type": "function",
    "function": {
        "name": "extract_metadata",
        "description": "Extract structured ML task metadata from the conversation.",
        "parameters": {
            "type": "object",
            "properties": {
                "dataset_name": {"type": ["string", "null"]},
                "dataset_modality": {"type": ["string", "null"], "enum": ["tabular", "text", "image", "audio", "video", "multimodal", None]},
                "dataset_source": {"type": ["string", "null"], "enum": ["local", "huggingface", "zenodo", "kaggle", "web", "github", None]},
                "data_format": {"type": ["string", "null"], "enum": ["csv", "json", "database", "zip", "parquet", "excel", "images_folder", "txt", None]},
                "task_type": {"type": ["string", "null"], "enum": ["classification", "regression", "time_series", "clustering", "anomaly_detection", "recommendation", "ranking", "segmentation", "text_generation", None]},
                "output_type": {"type": ["string", "null"], "enum": ["label", "probability_score", "numerical", "text", "ranking", "sentiment_score", "embeddings", "cluster_id", None]},
                "max_runtime_secs": {"type": ["integer", "null"]},
                "evaluation_metrics": {"type": ["array", "null"], "items": {"enum": ["accuracy", "precision", "recall", "f1_score", "roc_auc", "mse", "rmse", "mae", "r2", "confusion_matrix", "bleu_score", "rouge_score", "silhouette_score", "davies_bouldin_index", "ndcg", "map", "mrr"]}},
                "excluded_models": {"type": ["array", "null"], "items": {"enum": ["linear_regression", "logistic_regression", "bayesian_regression", "decision_tree", "random_forest", "gradient_boosting", "xgboost", "knn", "kmeans", "naive_bayes", "neural_network", "cnn", "rnn", "lstm", "transformer"]}},
                "included_models": {"type": ["array", "null"], "items": {"enum": ["linear_regression", "logistic_regression", "bayesian_regression", "decision_tree", "random_forest", "gradient_boosting", "xgboost", "knn", "kmeans", "naive_bayes", "neural_network", "cnn", "rnn", "lstm", "transformer"]}},
            },
        },
    },
}]


tokenizer = AutoTokenizer.from_pretrained(ft_model_name)

model = AutoModelForCausalLM.from_pretrained(
    ft_model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
).eval()

conversation = (
    "[USER]: Chcem realizovať klasifikáciu ŠPZ čísel áut pomocou CNN architektúry "
    "za pomoci datasetu 'LicencePlates_ImageDataset'.\n\n"
    "[ASSISTANT]: Rozumiem, použijete CNN na klasifikáciu ŠPZ. Aký formát majú "
    "obrázky a koľko tried plánujete rozlišovať?\n\n"
    "[USER]: Obrázky sú vo formáte JPG. Tried bude približne 30."
)

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": f"Multi-turn conversation:\n\n {conversation}"},
]

text = tokenizer.apply_chat_template(
    messages,
    tools=TOOL_SCHEMAS,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
im_end_id = tokenizer.convert_tokens_to_ids("<|im_end|>")
eos_token_ids = [tokenizer.eos_token_id, im_end_id]

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=1_024,
    temperature=0.7,
    eos_token_id=eos_token_ids,
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()

model_response = tokenizer.decode(output_ids, skip_special_tokens=True)

print(model_response)