I took the 320M-parameter model for Croatian RAG that I built in the previous article. Since it is meant to run on a CPU, this time I quantized it to see how much smaller and faster it could get—and how much accuracy it would lose.

Here is the conclusion first.

• Q6_K hits the sweet spot. It is one-fifth the size (254MB), generates about four times faster, and is just as accurate as the original model.
• Q2_K breaks down. It is fast, but starts saying “there is no answer” even when one exists.
• Surprisingly, Q3_K_M was more accurate than Q4_K_M.

What I did

I converted the PyTorch model into the format used by llama.cpp for CPU inference (GGUF), then saved it at eight different precisions.

1. Exported the model in the same format as Qwen3 and converted it to GGUF.
2. Because the tokenizer is custom, llama.cpp initially refused conversion because it was an “unknown tokenizer.” I compared its word-splitting rules and found them to be identical to Qwen2 character for character, so I registered it as Qwen2 and proceeded.
3. Verified that the conversion was correct. Tokenization matched the PyTorch version exactly for all 160 texts. For 20 questions, I generated answers by always selecting the highest-probability token; all 20 outputs matched perfectly as well.
4. Used llama-quantize to create versions from F16 through Q2_K.

That means the F32 in the table contains the same data as the PyTorch version. It serves as the baseline for comparison.

How I measured it

• The CPU was an i7-13700K. No GPU was used.
• llama.cpp was configured with 8 generation threads and 16 prompt-processing threads (this was the fastest setup I tested).
• I had each model answer 400 prompts: 100 evaluation questions × 4 context types.
  • Paragraph: A roughly 100-token passage containing the answer.
  • Long text: A roughly 3,000-token document containing the answer, mixed with unrelated articles.
  • No answer (paragraph / long text): Texts that did not contain the answer. Refusing to answer counted as correct.
• The previous article used 300 questions, so the figures differ slightly.

Results: accuracy

Format	Size	Paragraph correct	Long-text correct	Long-text refusals	Hallucinations (no-answer paragraph / long text)	Accuracy after verification
F32 (baseline)	1,233MB	77%	53%	30%	1% / 9%	80.9%
F16	617MB	77%	53%	30%	1% / 9%	80.9%
Q8_0	329MB	76%	53%	30%	1% / 8%	81.9%
Q6_K	254MB	79%	54%	30%	2% / 9%	81.3%
Q5_K_M	223MB	72%	45%	41%	1% / 6%	84.4%
Q4_K_M	193MB	68%	37%	46%	1% / 7%	75.9%
Q3_K_M	160MB	72%	48%	38%	1% / 8%	80.4%
Q2_K	129MB	39%	11%	78%	4% / 7%	72.2%

“Accuracy after verification” is the share of answered questions that were correct after a program checked whether the model’s citation actually appeared in the text and replaced unsupported answers with “no answer.”

• F16, Q8_0, and Q6_K were nearly identical to the baseline. They differed by only one or two questions.
• From Q5_K_M downward, refusals increase more than wrong answers. It seems to retain the behavior I trained in the previous article: when it loses confidence, it refuses. Hallucinations did not increase.
• Q4_K_M performed worse than Q3_K_M. It scored 37% versus 48% on long texts. There is some variance with 100 questions, but 11 points is substantial. I have not investigated the cause yet.
• Q2_K refused 78% of long-text prompts. It is effectively a different model.

Results: speed

Format	Generation speed	Average time per run
PyTorch fp32 (reference)	Paragraph 14.6 · long text 6.7 tok/s	—
F32	18.7 tok/s	2.76 seconds
F16	35.9 tok/s	2.08 seconds
Q8_0	59.0 tok/s	1.93 seconds
Q6_K	73.1 tok/s	1.33 seconds
Q5_K_M	81.2 tok/s	1.26 seconds
Q4_K_M	89.1 tok/s	1.10 seconds
Q3_K_M	103.4 tok/s	1.12 seconds
Q2_K	119.9 tok/s	1.07 seconds

The llama.cpp figures are averages over 400 runs, combining paragraphs, long texts, and no-answer cases. The PyTorch figures came from the previous article.

• Generation speed is almost inversely proportional to size. On a CPU, it is determined more by how quickly weights can be read from memory than by computation, so smaller is faster.
• Even at the same F32 precision, llama.cpp is faster than PyTorch.
• However, below Q6_K, the time per run barely falls. At this length, the initial prompt-processing time becomes the larger factor.

Video: answering the same long-context question on a CPU

I gave the model a roughly 3,200-token text and asked, “When was Need for Speed Heat released?” The correct answer is November 8, 2019.

Because I recorded the runs while the browser was also rendering the screen, the speeds in the video are lower than those in the table above. Treat this as a side-by-side comparison.

Format	Generation speed in video	Time to first token	Total	Result
PyTorch fp32	5.5 tok/s	4.8 seconds	18.5 seconds	Correct
F32	14.2 tok/s	2.8 seconds	8.1 seconds	Correct
F16	25.9 tok/s	3.5 seconds	6.4 seconds	Correct
Q8_0	38.2 tok/s	4.0 seconds	5.9 seconds	Correct
Q6_K	41.6 tok/s	2.6 seconds	4.4 seconds	Correct
Q5_K_M	46.3 tok/s	2.6 seconds	4.3 seconds	Correct
Q4_K_M	47.2 tok/s	2.3 seconds	4.7 seconds	Correct (longer answer)
Q3_K_M	56.7 tok/s	2.5 seconds	4.5 seconds	Correct (longer answer)
Q2_K	61.3 tok/s	2.7 seconds	2.8 seconds	Refused

From F32 through Q5_K_M, both the answer and the citation were identical to the PyTorch version character for character. Q4_K_M and Q3_K_M produced longer answer sentences, but their citations were correct. Q2_K said, “The text does not contain the answer,” despite an answer being present.

PyTorch fp32 (the original model)

再生

llama.cpp F32 (1,233MB)

再生

llama.cpp F16 (617MB)

再生

llama.cpp Q8_0 (329MB)

再生

llama.cpp Q6_K (254MB)

再生

llama.cpp Q5_K_M (223MB)

再生

llama.cpp Q4_K_M (193MB)

再生

llama.cpp Q3_K_M (160MB)

再生

llama.cpp Q2_K (129MB)

再生

Takeaways

• Use Q6_K to run this model on a CPU. It preserves the original accuracy, is one-fifth the size, and generates about five times faster than PyTorch. There is no reason to hesitate.
• Choose Q8_0 for extra peace of mind. It is almost identical to the original at 329MB—still plenty small.
• Small 300M-parameter models are sensitive to quantization. Larger models are said to show almost no degradation even at Q4, but this model declined visibly from Q5 onward. It has little spare capacity to begin with, so pruning it has an effect.
• It is fortunate that the failure mode is refusal. In RAG, refusing is safer than increased hallucination. The reinforcement-learning work from the previous article, which taught it how to refuse, is showing its effect here too.
• Speed differences narrow for smaller models. Below Q6_K, faster generation still leaves prompt-processing time, so the perceived difference is small. In the video, Q6_K through Q3_K_M all finished in the four-second range.
• I still do not know why Q4_K_M lost to Q3_K_M. I want to investigate with more questions and by examining how each layer is quantized.