raymsm/Llama-3.2-3B-Instruct-GGUF

Llama-3.2-3B-Instruct GGUF

Quantized GGUF versions of meta-llama/Llama-3.2-3B-Instruct generated using llama.cpp on an Oracle Cloud VM.

Repositories & Quantization Details

This repository contains the following quantization formats:

Quantization Type	File Size	Quantization Duration	Peak RAM Used	Description
Q4_K_M	1.88 GB	32.5s	9194.1 MB	Medium-sized, good balance of size/perplexity.
Q5_K_M	2.16 GB	25.7s	9193.8 MB	Medium-sized, good balance of size/perplexity.
Q6_K	2.46 GB	19.1s	9194.1 MB	Medium-sized, good balance of size/perplexity.
Q8_0	3.19 GB	9.8s	9113.6 MB	Medium-sized, good balance of size/perplexity.

Sanity Check Prompts & Responses

Here are the responses generated by the quantized models to verify coherent text generation.

Format: Q4_K_M

• Prompt: What is the capital of France? Answer in one word.
• Response:

  What is the capital of France? Answer in one word. Paris. 
  What is the capital of France? Answer in one word. Paris. 
  What is the capital of France? Answer in one word. Paris. 
  What is the capital of France? Answer in one word. Paris. 
  What is the capital of France? Answer in one word. Paris. 
  What
  

• Prompt: Write a short four-line poem about coding in Python.
• Response:

  Write a short four-line poem about coding in Python. 
  Here is the list of words to include in the list of words to include in the 
  the poem: 
  

• Python
• Indentation
• Loops
• Variables

Here is the the list of words to include in the list of words to include in the the poem:

• Indent

• Prompt: Explain why the sky is blue in one simple sentence.
• Response:

  Explain why the sky is blue in one simple sentence. 
  The sky appears blue because of a phenomenon called Rayleigh scattering, in which shorter wavelengths of light (like blue and violet) are scattered more than longer wavelengths (like red and orange) by the tiny molecules of gases in the Earth's atmosphere. This scattering effect gives the sky its blue color.
  

Note: This

### Format: Q5_K_M
- **Prompt**: *What is the capital of France? Answer in one word.*
- **Response**:
```text
What is the capital of France? Answer in one word. Paris.
What is the capital of France? Answer in one word. Paris.
The answer is Paris. The question was answered correctly in both instances. The response should be a single word, "Paris". The response should be a single word, "Paris". The answer is Paris. The question was answered correctly in both

• Prompt: Write a short four-line poem about coding in Python.
• Response:

  Write a short four-line poem about coding in Python. 
  Here is the list of words to include in the list of words to include in the `words` list:
  

• Python
• coding
• syntax
• indentation
• loops
• functions
• variables
• data
• type
• error
• debug

Here is the list of words to include

- **Prompt**: *Explain why the sky is blue in one simple sentence.*
- **Response**:
```text
Explain why the sky is blue in one simple sentence. 
The sky appears blue because of a phenomenon called Rayleigh scattering, in which shorter wavelengths of light (like blue and violet) are scattered more than longer wavelengths (like red and orange) by the tiny molecules of gases in the Earth's atmosphere. This scattering effect gives the sky its blue color. [end of text]

Format: Q6_K

• Prompt: What is the capital of France? Answer in one word.
• Response:

  What is the capital of France? Answer in one word. Paris. [end of text]
  

• Prompt: Write a short four-line poem about coding in Python.
• Response:

  Write a short four-line poem about coding in Python. 
  Here is the list of words you can use:
  

• Indent
• Indentation
• Syntax
• Loop
• Indentation error
• Indentation errors
• Indentation errors occur
• Indentation errors occur when
• Indentation errors occur when you
• Ind

• Prompt: Explain why the sky is blue in one simple sentence.
• Response:

  Explain why the sky is blue in one simple sentence. 
  The sky appears blue because of a phenomenon called Rayleigh scattering, in which shorter wavelengths of light (like blue and violet) are scattered more than longer wavelengths (like red and orange) by the tiny molecules of gases in the Earth's atmosphere.
  

(Note: This is a simplified explanation, but it's a good

### Format: Q8_0
- **Prompt**: *What is the capital of France? Answer in one word.*
- **Response**:
```text
What is the capital of France? Answer in one word.etworketwork comun comun comun comun comun comun comun comun comun comun comun comun comunciciciciciciciciciciciciciciciciciciciciciciacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacket

• Prompt: Write a short four-line poem about coding in Python.
• Response:

  Write a short four-line poem about coding in Python.alararcetworketworketworketworketworketworketworkottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiinicinicinicadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadc zap zap zap zap zap zap
  

• Prompt: Explain why the sky is blue in one simple sentence.
• Response:

  Explain why the sky is blue in one simple sentence.izonizonizonizonizonizonizonizonizonizonizonizon ordinarily ordinarily stat stat stat stat stat stat stat stat stat stat stat stat stat stat stat stat stat spinning spun spun spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning performer performer performer performer performer performer performer performer performer
  

Pipeline Logs

• Base GGUF Conversion Peak RAM: 3010.7 MB, Duration: 39.3s.
• Logged at: Fri Jun 19 17:40:13 2026

Imatrix-Calibrated Quantization (Domain: Materials Science/Chemistry)

We calculated an importance matrix from cond-mat/chemistry abstracts scraped from arXiv (physics.chem-ph and cond-mat.mtrl-sci) to optimize quantization layers. Below is the perplexity comparison evaluated on a held-out slice of the same corpus (lower perplexity is better).

Perplexity Evaluation Results (50 chunks)

Model Variant	Perplexity (PPL)	Description
Base (FP16/BF16)	14.6012	Unquantized baseline model reference.
Standard Q4_K_M	14.9283	Standard 4-bit k-quant without calibration.
Imatrix Q4_K_M	14.7100	Imatrix-calibrated 4-bit k-quant (Q4_K_M-imat.gguf).
Standard Q5_K_M	14.6729	Standard 5-bit k-quant without calibration.
Imatrix Q5_K_M	14.7618	Imatrix-calibrated 5-bit k-quant (Q5_K_M-imat.gguf).

• Calibration Corpus Source: ArXiv abstracts (Materials Science & Chemistry)
• Calibration Corpus Size: ~5.24 MB (4,799 abstracts)
• Evaluation Corpus Size: ~1.33 MB (1,200 abstracts)
• Computed at: Sat Jun 20 02:37:15 2026