Llama-3.2-3B-Code-Knowledge-Value-Eval

This model is a fine-tuned version of meta-llama/Llama-3.2-3B on the kimsan0622/code-knowledge-eval dataset. It achieves the following results on the evaluation set:

• Loss: 0.9977
• Accuracy: 0.5552

Model Description

The model trained on the Code Knowledge Value Evaluation Dataset is designed to assess the educational and knowledge value of code snippets. It leverages patterns and contextual information from a large collection of open-source code, sourced from the bigcode/the-stack repository. By analyzing these code samples, the model can evaluate their utility in teaching coding concepts, solving problems, and improving developer education.

The model focuses on understanding the structure, syntax, and logic of various programming languages, enabling it to provide insights into the learning potential and technical depth of different code samples. The dataset used for training consists of 22,786 samples for training, 4,555 for validation, and 18,232 for testing, ensuring that the model is both robust and well-generalized across different coding contexts.

Intended Uses & Limitations

Intended Uses:

1. Automated Code Review: The model can be applied in automated systems to assess the knowledge value of code during code review processes. It can help identify areas where code could be optimized for better readability, maintainability, and educational impact.
2. Educational Feedback: For instructors and educational platforms, the model can offer feedback on the effectiveness of code samples used in teaching, helping to improve curriculum materials and select code that best conveys core programming concepts.
3. Curriculum Development: The model can aid in designing coding courses or instructional materials by suggesting code examples that have higher educational value, supporting a more effective learning experience.
4. Technical Skill Assessment: Organizations or platforms can use the model to assess the complexity and educational value of code submissions in coding challenges or exams.

Limitations:

1. Narrow Scope in Knowledge Evaluation: The model is specialized in evaluating code from an educational standpoint, focusing primarily on learning potential rather than production-level code quality (e.g., performance optimization or security).
2. Language and Domain Limitations: Since the dataset is sourced from bigcode/the-stack, it may not cover all programming languages or specialized domains. The model may perform less effectively in underrepresented languages or niche coding styles not well-represented in the dataset.
3. Not Suitable for All Educational Levels: While the model is designed to evaluate code for educational purposes, its outputs may be better suited for certain levels (e.g., beginner or intermediate coding), and its recommendations might not fully cater to advanced or highly specialized learners.

How to use this model?

import torch
import numpy as np
from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Define the model name or path for loading the tokenizer and model
model_name_or_path = "kimsan0622/Llama-3.2-3B-Code-Knowledge-Value-Eval"

# Load the tokenizer from the pre-trained model
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)

# Load the pre-trained model for sequence classification and map it to the first CUDA device
model = AutoModelForSequenceClassification.from_pretrained(
    model_name_or_path, 
    device_map="cuda:0",
)

# Example code snippet to be evaluated
code = [
"""
import torch
import numpy as np
from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Define the model name or path for loading the tokenizer and model
model_name_or_path = "kimsan0622/Llama-3.2-1B-Code-Knowledge-Value-Eval"

# Load the tokenizer from the pre-trained model
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)

# Load the pre-trained model for sequence classification and map it to the first CUDA device
model = AutoModelForSequenceClassification.from_pretrained(
    model_name_or_path, 
    device_map="cuda:0",
)

# Example code snippet to be evaluated
code = ["code 1"]

# Tokenize the input code, setting max length, padding, and truncation
batch = tokenizer(code, max_length=1024, padding=True, truncation=True, return_tensors="pt")

# Perform inference with the model, without computing gradients (for faster inference)
with torch.no_grad():
    # Pass the input IDs and attention mask to the model, using the CUDA device
    res = model(
            input_ids=batch["input_ids"].to("cuda:0"),
            attention_mask=batch["attention_mask"].to("cuda:0"),
        )
    
    # Move the logits to the CPU, convert them to a numpy array
    preds = res.logits.cpu().numpy()
    
    # Get the predicted class by taking the argmax of the logits across the classification axis
    preds = np.argmax(preds, axis=1).tolist()
"""
]

# Tokenize the input code, setting max length, padding, and truncation
batch = tokenizer(code, max_length=1024, padding=True, truncation=True, return_tensors="pt")

# Perform inference with the model, without computing gradients (for faster inference)
with torch.no_grad():
    # Pass the input IDs and attention mask to the model, using the CUDA device
    res = model(
            input_ids=batch["input_ids"].to("cuda:0"),
            attention_mask=batch["attention_mask"].to("cuda:0"),
        )
    
    # Move the logits to the CPU, convert them to a numpy array
    preds = res.logits.cpu().numpy()
    
    # Get the predicted class by taking the argmax of the logits across the classification axis
    preds = np.argmax(preds, axis=1).tolist()
    print(preds)

8 Bit quantization

from transformers import AutoTokenizer, AutoModelForSequenceClassification, BitsAndBytesConfig

# Define the model name or path for loading the model
model_name_or_path = "kimsan0622/Llama-3.2-3B-Code-Knowledge-Value-Eval"

# Configure the model to load in 8-bit precision for memory efficiency
bnb_config = BitsAndBytesConfig(load_in_8bit=True)

# Load the pre-trained model for sequence classification with quantization for 8-bit precision
# This helps reduce memory usage, particularly for large models, and map it to the first CUDA device
model = AutoModelForSequenceClassification.from_pretrained(
    model_name_or_path, 
    quantization_config=bnb_config,  # Apply 8-bit quantization
    device_map="cuda:0",  # Map the model to the first CUDA device
)

4 Bit quntization

from transformers import AutoTokenizer, AutoModelForSequenceClassification, BitsAndBytesConfig
import torch

# Define the model name or path for loading the model
model_name_or_path = "kimsan0622/Llama-3.2-3B-Code-Knowledge-Value-Eval"

# Define configuration parameters for 4-bit quantization
bnb_config_params = {
    "bnb_4bit_quant_type": "fp4",  # Use FP4 for 4-bit quantization type
    "bnb_4bit_compute_dtype": torch.bfloat16,  # Use bfloat16 for computation to balance performance and precision
    "bnb_4bit_use_double_quant": False,  # Disable double quantization, which is typically used to further reduce precision
    "bnb_4bit_quant_storage": torch.bfloat16,  # Store quantized values in bfloat16 format
}

# Configure the model to load in 4-bit precision for memory and performance optimization
bnb_config = BitsAndBytesConfig(load_in_4bit=True, **bnb_config_params)

# Load the pre-trained model for sequence classification with 4-bit quantization settings
# This reduces memory usage while still maintaining reasonable accuracy, mapping the model to the first CUDA device
model = AutoModelForSequenceClassification.from_pretrained(
    model_name_or_path, 
    quantization_config=bnb_config,  # Apply 4-bit quantization configuration
    device_map="cuda:0",  # Map the model to the first CUDA device
)

Training and evaluation data

kimsan0622/code-knowledge-eval

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 8
• total_train_batch_size: 128
• total_eval_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 5.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.1115	0.9993	178	1.0997	0.4900
0.9391	1.9986	356	0.9977	0.5552
0.6717	2.9979	534	1.1785	0.5431
0.2989	3.9972	712	1.8204	0.5405
0.133	4.9965	890	2.5260	0.5482

Framework versions

• Transformers 4.44.2
• Pytorch 2.3.0
• Datasets 2.20.0
• Tokenizers 0.19.1

Test set results

Confusion matrix

y_true	pred_0	pred_1	pred_2	pred_3	pred_4	pred_5
0	940	268	107	57	5	0
1	316	344	338	221	24	1
2	132	251	510	738	135	8
3	43	128	458	2674	1565	57
4	8	15	74	1248	4229	750
5	0	2	0	29	1094	1463

Classification report

	precision	recall	f1-score	support
0	0.65	0.68	0.67	1377
1	0.34	0.28	0.31	1244
2	0.34	0.29	0.31	1774
3	0.54	0.54	0.54	4925
4	0.60	0.67	0.63	6324
5	0.64	0.57	0.60	2588
accuracy			0.56	18232
macro avg	0.52	0.50	0.51	18232
weighted avg	0.55	0.56	0.55	18232

8 Bit quantization model

Confusion matrix

y_true	pred_0	pred_1	pred_2	pred_3	pred_4	pred_5
0	933	272	111	56	5	0
1	320	333	340	224	25	2
2	129	244	502	752	139	8
3	43	119	463	2627	1604	69
4	7	15	67	1227	4191	817
5	0	2	0	30	1034	1522

Classification report

y_true	precision	recall	f1-score	support
0	0.65	0.68	0.66	1377
1	0.34	0.27	0.30	1244
2	0.34	0.28	0.31	1774
3	0.53	0.53	0.53	4925
4	0.60	0.66	0.63	6324
5	0.63	0.59	0.61	2588
accuracy			0.55	18232
macro avg	0.52	0.50	0.51	18232
weighted avg	0.55	0.55	0.55	18232

4 Bit quantization model

Confusion matrix

y_true	pred_0	pred_1	pred_2	pred_3	pred_4	pred_5
0	695	581	42	55	4	0
1	151	662	190	215	24	2
2	53	485	353	716	159	8
3	20	277	335	2446	1765	82
4	4	31	60	1104	4211	914
5	0	2	0	24	934	1628

Classification report

	precision	recall	f1-score	support
0	0.75	0.50	0.60	1377
1	0.32	0.53	0.40	1244
2	0.36	0.20	0.26	1774
3	0.54	0.50	0.52	4925
4	0.59	0.67	0.63	6324
5	0.62	0.63	0.62	2588
accuracy			0.55	18232
macro avg	0.53	0.50	0.51	18232
weighted avg	0.55	0.55	0.54	18232