q4_k_s: Uses Q4_K for all tensors

(Trained with Unsloth)

q5_0: Higher accuracy, higher resource usage and slower inference.

(Trained with Unsloth)

q8_0: Fast conversion. High resource use, but generally acceptable.

(Trained with Unsloth)

q6_k: Uses Q6_K for all tensors

(Trained with Unsloth)

q5_k_m: Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q5_K

(Trained with Unsloth)

q3_k_m: Uses Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else Q3_K

(Trained with Unsloth)

q4_k_m: Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q4_K

(Trained with Unsloth)

initial commit