Instructions to use micrictor/gemma-3-270m-it-ft-bash-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use micrictor/gemma-3-270m-it-ft-bash-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf micrictor/gemma-3-270m-it-ft-bash-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf micrictor/gemma-3-270m-it-ft-bash-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf micrictor/gemma-3-270m-it-ft-bash-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf micrictor/gemma-3-270m-it-ft-bash-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf micrictor/gemma-3-270m-it-ft-bash-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf micrictor/gemma-3-270m-it-ft-bash-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf micrictor/gemma-3-270m-it-ft-bash-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf micrictor/gemma-3-270m-it-ft-bash-GGUF:Q4_K_M

Use Docker

docker model run hf.co/micrictor/gemma-3-270m-it-ft-bash-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use micrictor/gemma-3-270m-it-ft-bash-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "micrictor/gemma-3-270m-it-ft-bash-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "micrictor/gemma-3-270m-it-ft-bash-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/micrictor/gemma-3-270m-it-ft-bash-GGUF:Q4_K_M

Ollama
How to use micrictor/gemma-3-270m-it-ft-bash-GGUF with Ollama:
```
ollama run hf.co/micrictor/gemma-3-270m-it-ft-bash-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use micrictor/gemma-3-270m-it-ft-bash-GGUF with Docker Model Runner:
```
docker model run hf.co/micrictor/gemma-3-270m-it-ft-bash-GGUF:Q4_K_M
```

Lemonade

How to use micrictor/gemma-3-270m-it-ft-bash-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull micrictor/gemma-3-270m-it-ft-bash-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.gemma-3-270m-it-ft-bash-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

gemma-3-270m-it-ft-bash GGUF

GGUF conversions of micrictor/gemma-3-270m-it-ft-bash, pinned to source commit 6d93b590569904932d9168e430225df634f5d09f.

Files

File	Format	Size	SHA-256
`gemma-3-270m-it-ft-bash-BF16.gguf`	BF16	551.0 MB	`b96916060157345c8ceb90698f725c4815f2d471dec244b43bda4fed0ca62098`
`gemma-3-270m-it-ft-bash-Q8_0.gguf`	Q8_0	299.7 MB	`48def0889359a6e132fb83692115bd3ec8631679f25f2a3b56cae027a9bb71b8`
`gemma-3-270m-it-ft-bash-Q4_K_M.gguf`	Q4_K_M	261.3 MB	`efaccab0eac30b900322d26f586e1a6772d58bbc6c948c6f6f6bc8f2e1da757a`
`gemma-3-270m-it-ft-bash-Q4_0.gguf`	Q4_0	249.6 MB	`5052d595049f90d61d065faa472a5dce2f103d551d7ce66da6e12d027e463a68`
`gemma-3-270m-it-ft-bash-IQ4_NL.gguf`	IQ4_NL	166.6 MB	`cd027371f56c5ee01bdbb6d6fa503621d0bcaea8ea01e8e35d7c53761e2b5945`

BF16 preserves the source model's native precision. Q8_0, Q4_K_M, Q4_0, and IQ4_NL are CPU-oriented llama.cpp quantizations made directly from that BF16 conversion. The IQ4_NL file explicitly quantizes the tied embedding/output tensor to IQ4_NL; 18 attention-value tensors use llama.cpp's automatic Q5_1 fallback because their 640-column shape is incompatible with the preferred block size.

Evaluation snapshot

On 100 rows from the fine-tune's reconstructed seed-42 evaluation split, greedy decoding produced the following results:

Format	Exact match	Token F1	64-token truncation
BF16	26%	0.6118	1%
Q8_0	28%	0.6109	1%
Q4_0	7%	0.3659	21%
IQ4_NL	10%	0.4725	2%

Q8_0 showed no detectable Token-F1 degradation relative to BF16 in this sample. IQ4_NL was faster and smaller but had a measurable quality loss. Generated shell commands were scored as text and were never executed.

Built with llama.cpp commit c1d0e7a004015f23bc0233470b747b596f29b264. See build-manifest.json for machine-readable provenance.

Use the Gemma chat template included in the GGUF metadata. Review generated shell commands before executing them.

This derivative is subject to the Gemma terms attached to the source model.

Downloads last month: 2

GGUF

Model size

0.3B params

Architecture

gemma3

Hardware compatibility

4-bit

8-bit

16-bit

Model tree for micrictor/gemma-3-270m-it-ft-bash-GGUF

Base model

google/gemma-3-270m

Finetuned

google/gemma-3-270m-it

Finetuned

micrictor/gemma-3-270m-it-ft-bash

Quantized

(1)

this model