Having built a Croatian RAG model and even quantized it, I next looked into how usable the small models (SLMs) currently available are for RAG. I tested 25 models—10 in the 1B class, 8 in the 3B class, and 7 in the 7B class—plus my own 320M model, on the same questions in Croatian and English.

Update (same day): I also tested three small models made specifically for RAG (LFM2-1.2B-RAG and Pleias-RAG-1B / 350M). The results are in the latter half of this article, under “Update: I also tried RAG-specific models.”

The conclusion first.

• At 3–4B, models are already nearly on par with 8–9B models. In overall Croatian accuracy, Qwen3.5 4B and Gemma 4 E4B scored 90%, versus 91% for the top model, Qwen3.5 9B
• The differentiator is whether they can decline when there is no answer. Phi-4-mini, which was among the top models in accuracy, fabricated an answer for 64% of answerless long texts
• More models decline even when an answer exists in English. OLMo 3 7B declined 47% of English passages, and MiniCPM5 2B declined 70%
• On CPU, 7B-class models take over 30 seconds per long-text question. 1B-class models take 4–13 seconds. My 320M model took 2.1 seconds, with accuracy roughly equal to Qwen3.5 0.8B

What I did

Questions: The same questions as in my previous article, created from Croatian Wikipedia. Each question was given four kinds of text.

• Passage: A passage of around 100 tokens containing the answer
• Long text: Roughly 3,000 tokens containing the answer, mixed with unrelated articles
• No answer (passage / long text): Text that does not contain the answer. Correct if the model declined with “There is no answer in the text.”

English version: I translated the same questions into English with EuroLLM-9B. Questions where the answer disappeared in translation or remained in Croatian rather than being translated were removed from both languages, so I compared the same 260 questions.

Instruction: In each language, I gave the instruction: “Using only the text below, answer briefly in one sentence. If there is no answer, reply only ‘There is no answer in the text.’”

Scoring: I had Qwen3.5 9B compare each response with the reference answer and classify it as “correct,” “incorrect,” or “declined.” I manually checked 65 cases; the grading was somewhat lenient, counting partial answers such as giving only the year as correct, but every judgment of whether the model declined was correct.

Overall accuracy: The total number answered correctly for passages and long texts, plus the number correctly declined for no-answer cases, divided by all questions. It measures both answering correctly and staying silent when the model does not know.

How I ran them: I measured accuracy with vLLM on an RTX 4090, using greedy generation that selects the highest-probability token each time. I turned off “thinking mode” for models that support reasoning before answering. I used FP8 for the 7B-class models so they would fit in GPU memory.

Results: accuracy (Croatian)

• The leaders are essentially tied at around 90%. Qwen3.5 9B (91%), Llama 3.1 8B, Qwen3.5 4B, and Gemma 4 E4B (90%)
• Among 1B-class models, Gemma 4 E2B stands out (85%). Despite the 2B name, its file size is relatively large at 3.2 GB
• Some models have high passage and long-text accuracy but fall in the overall score. Phi-4-mini is in the top group with 91% on passages and 89% on long texts, yet its overall score is 72%. The reason is the following “fabrication”
• The worst was LFM2.5 1.2B (34%). It could not answer questions that had answers, and fabricated answers for questions that did not

Results: accuracy (English)

• The top model was Gemma 4 E4B (88%). Gemma 4 E2B also scored 86%, so Gemma 4 was stable in both languages
• Some models rank lower in English. Qwen3.5 4B fell from 90% in Croatian to 82% in English. The cause was declining despite an answer being present (35% of passages)

What I learned 1: the difference is whether a model can decline

The percentage of fabricated answers when given a long text with no answer (Croatian).

Model	Fabrication in answerless long texts
Qwen3.5 4B	9%
My 320M model	9%
Qwen3.5 9B	10%
Gemma 4 E4B	19%
Granite 4.1 8B	30%
Phi-4-mini 3.8B	64%
SmolLM3 3B	73%
EuroMoE 2.6B-A0.6B	94%

For example, I asked, “When was Need for Speed Heat released?” and provided a long text that did not contain the answer. Qwen3.5 9B replied, “There is no answer in the text,” while Phi-4-mini replied, “August 1969.” It appears to have picked up a date from somewhere in the text.

This is the biggest problem in RAG. It is risky to choose a model based on accuracy alone. Among the answers models did give, Qwen3.5 4B and 9B were the most accurate at 91–92%, while Phi-4-mini was only 64% correct.

What I learned 2: English leads to too many declines

Conversely, many models declined in English even when the answer was clearly present. The percentage that declined when given a passage containing the answer.

Model	Croatian	English
Qwen3.5 4B	15%	35%
Qwen3.5 0.8B	17%	47%
OLMo 3 7B	8%	47%
MiniCPM5 2B	34%	70%
Gemma 4 E4B	9%	8%

For example: “When did Paul Accola last finish on the podium?” The English passage clearly says “the last time in 2000,” yet Qwen3.5 4B answered, “There is no answer in the text.” When asked the same question in Croatian, it correctly answered “2000.”

I do not yet know why. The English version is machine-translated, so the unnatural wording may be affecting it, or models may be reacting strongly in English to the instruction, “If there is no answer, reply this way.”

Results: CPU speed

Since my own use case assumes CPU inference, I converted all models into the same llama.cpp format (Q4_K_M) and measured them on an i7-13700K. Generation used 8 threads and text ingestion used 16 threads. Ministral 3 alone did not convert properly, so I used Mistral’s official Q4_K_M version.

I also estimated the time to answer one long-text question. For each model, this is the time to ingest the number of input tokens in a long text plus the time to generate an average-length answer.

• Generation speed is roughly inversely proportional to size. 7B-class models run at around 5 tokens per second, taking 34–47 seconds per long-text question
• 3B-class models take 13–23 seconds, while 1B-class models take 4–13 seconds
• Even for the same long texts, the number of input tokens differs by model, from 2,443 to 3,796. This comes down to how compactly each tokenizer can represent Croatian, and my tokenizer built specifically for Croatian is the shortest

Putting speed and accuracy side by side

Further toward the upper left means “faster and more accurate.”

• Gemma 4 E2B (11 seconds, 85%) and Qwen3.5 2B (9 seconds, 79%) offer a good balance
• If accuracy is the priority, choose Qwen3.5 4B or Gemma 4 E4B (21–23 seconds, 90%). They are over 1.5 times faster than the 7B-class models, with almost the same accuracy
• My 320M model is in a different league for speed (2.1 seconds). Its 75% overall accuracy matches Qwen3.5 0.8B, and its low fabrication rate is on par with the leading models. However, its 46% long-text accuracy is clearly weaker than Qwen3.5 0.8B (75%) and Gemma 4 E2B (88%)
• My model was not trained on English, so it fabricated answers for 91% of English answerless long texts

Caveats

• The scoring was done by an LLM, so it is not perfect. It is somewhat lenient
• The English version is machine-translated. Both the questions and their translation were created by EuroLLM-9B, which may give EuroLLM a slight advantage (it scored a middling 86%)
• Apertus 1.5B can handle only up to 4,096 tokens of context, so 23% of Croatian long texts did not fit. Cases that did not fit were counted as incorrect
• Accuracy was measured on GPU (vLLM; FP8 for the 7B class) and speed on CPU (Q4_K_M), so the test environments differ. Quantization could change accuracy slightly
• I measured speed on a PC with my usual applications still open. I measured all models consecutively under the same conditions, so I think it is still useful for comparison

Discussion

• For RAG, 4B is already enough. The leading 3–4B-class models are about as accurate as 7–9B-class models and are 1.5–2 times faster on CPU
• When choosing a model, you should always check whether it can decline. Even at similar accuracy, fabrication rates on answerless text vary enormously by model, from 1% to 99%. RAG often retrieves unrelated text, so this matters most
• The same model can behave differently depending on the language. Many models were overly cautious in English and missed answers. You cannot know without actually testing in the language you will use
• The challenge for my model is long text. Its ability to decline and its speed are sufficient; if long-text accuracy improves, it should be quite usable as a “RAG model that runs on CPU.” That is what I want to improve next

Update: I also tried RAG-specific models

I wondered, “What about models tuned specifically for RAG?” so I added three small RAG-specific models. The conditions were the same as above (the same 260 questions and the same scoring).

• LFM2-1.2B-RAG (Liquid AI): LFM2-1.2B further trained on more than one million examples of answering from supplied documents. It supports nine languages, including English and Japanese, but not Croatian
• Pleias-RAG-1B / Pleias-RAG-350M (PleIAs): Models trained on around 3.1 million synthetic RAG examples. Before answering, they write “question analysis → document analysis → draft,” then attach citations from the text to the answer. They support seven European languages, but not Croatian. Their maximum context length is 4,096 tokens
• For comparison, I also measured the base plain LFM2-1.2B model used to create LFM2-1.2B-RAG

I gave LFM2-1.2B-RAG the same instructions as the other models. Pleias accepts only its dedicated input format, which separates questions and documents with special markers, so I used the same format as the official library and graded only the final answer portion. Therefore, I could not give Pleias the instruction, “If there is no answer, reply this way.”

Model	Overall (Croatian)	Overall (English)	Fabrication in answerless long texts (Croatian / English)	Time per long-text question (CPU)
LFM2 1.2B (base model)	43%	81%	42% / 36%	7.8 seconds
LFM2 1.2B RAG	40%	80%	61% / 51%	8.1 seconds
Pleias-RAG 1B	38%	43%	57% / 73%	16.9 seconds
Pleias-RAG 350M	23%	39%	68% / 92%	6.0 seconds
Reference: my 320M model	75%	47%	9% / 91%	2.1 seconds
Reference: Qwen3.5 0.8B	75%	70%	43% / 21%	5.1 seconds
Reference: Gemma 4 E2B	85%	86%	37% / 32%	11.1 seconds

In short, I could not find anything these RAG-specific models did better than ordinary models on these questions.

LFM2-1.2B-RAG: it became more willing to answer, but also fabricated more

Compared with the base LFM2-1.2B in English, accuracy on questions with answers improved (passages: 83% → 87%; long texts: 78% → 86%). But fabrication on answerless text also increased (passages: 4% → 10%; long texts: 36% → 51%), leaving the overall score almost unchanged at 81% → 80%. The additional RAG training seems to have shifted it toward answering at all costs.

In Croatian, the base model was already almost unusable. The base LFM2-1.2B declined 93% of passages that contained answers by saying “There is no answer,” while the RAG version often returned the instruction text verbatim, such as “Odgovori kratko, jednom rečenicom. (Answer briefly in one sentence.)” Since Croatian is not among its supported languages, this is understandable.

Pleias-RAG: it adds citations, but those citations are not in the text

Pleias almost never declined even on answerless text. In English, the 1B model fabricated answers for 78% of answerless passages and 73% of answerless long texts.

Its selling point, “answers with citations,” is also questionable. I counted how often the cited sentence could be found verbatim in the text (this is an exact match that ignores only whitespace differences, so even slightly paraphrased citations count as “not found”).

	Questions with answers	Questions without answers
Pleias-RAG 1B (English)	64%	42%
Pleias-RAG 1B (Croatian)	57%	53%
Pleias-RAG 350M (English)	66%	39%
Pleias-RAG 350M (Croatian)	52%	40%

For example: “Which band originally wrote the song ‘Tourniquet’?” (The answer is Soul Embraced.)

• When given a passage without the answer, the 1B model answered, “Amy Lee wrote it,” and attached a sentence not found in the text as a citation: “Amy Lee (born Amy Leigh Lee, November 1, 1970) ...”
• Even when given a passage containing the answer, it answered, “Amy Lee wrote it.” The attached citation was “originally written ... for the Christian death metal band Soul Embraced,” so the citation was correct but contradicted the answer

Citations can make it look as if an answer is properly grounded in the text, which makes this even more dangerous.

Pleias is also slow because it writes analysis before its answer, resulting in long outputs (an average of 381–420 tokens for long texts). The 1B model took 16.9 seconds per long-text question, slower than Gemma 4 E2B (11.1 seconds). In addition, 25% of Croatian long texts did not fit in the context window (counted as incorrect; the timing averages only questions that fit, so real-world time would be somewhat longer).

Why did this happen? (Speculation)

• Language: Croatian is not among the supported languages of any of the three models. However, they also lost to ordinary models in English
• Question format: This test involved “answer briefly from one text / decline if there is no answer.” Pleias is evaluated on questions that combine multiple documents to answer (such as HotpotQA), so its strengths may suit a different format
• How to decline: Ordinary models can be instructed, “If there is no answer, reply this way,” but Pleias has nowhere to supply such an instruction. The model itself has a mechanism for deciding, “This material cannot answer the question,” but it barely worked in this test

Summary of the update

• Do not choose based on the label “RAG-specific” alone. Under these conditions, ordinary instruction models such as Gemma 4 E2B and Qwen3.5 were more accurate and declined when they should
• Answers with citations are meaningless unless you verify the citations themselves. Only 40–70% of Pleias citations could be found verbatim in the text
• My model outperforms similarly sized RAG-specific models. Compared with Pleias-RAG 350M, Croatian overall accuracy was 75% versus 23%, and time per long-text question was 2.1 seconds versus 6.0 seconds. Building it specifically for Croatian was worthwhile

I measured the speed of the four added models on the same PC under the same conditions on the evening of the same day. I also added them to the speed-and-accuracy chart.

Full results

Croatian

Model	Class	Paragraph Correct	Long-text Correct	Overall	Hallucination with no answer (paragraph/long text)	Percentage correct among answered questions
Qwen3.5 9B	7B class	84%	90%	91%	1% / 10%	91%
Llama 3.1 8B	7B class	88%	88%	90%	2% / 15%	90%
Qwen3.5 4B	3B class	82%	88%	90%	1% / 9%	92%
Gemma 4 E4B	3B class	87%	91%	90%	2% / 19%	87%
Ministral 3 8B	7B class	88%	85%	89%	3% / 14%	89%
Granite 4.1 8B	7B class	91%	90%	88%	2% / 30%	84%
Granite 4.1 3B	3B class	88%	88%	88%	2% / 29%	83%
EuroLLM 9B	7B class	87%	75%	86%	5% / 14%	88%
Gemma 4 E2B	1B class	88%	88%	85%	5% / 37%	78%
Llama 3.2 3B	3B class	80%	81%	84%	2% / 27%	81%
Apertus 8B	7B class	85%	80%	83%	8% / 29%	81%
Qwen3.5 2B	1B class	74%	74%	79%	4% / 29%	79%
Custom 320M	Custom	72%	46%	75%	3% / 9%	72%
Qwen3.5 0.8B	1B class	74%	75%	75%	11% / 43%	68%
Granite 4.0 1B	1B class	83%	77%	74%	22% / 51%	64%
MiniCPM5 2B	1B class	62%	44%	73%	1% / 6%	89%
OLMo 3 7B	7B class	80%	70%	72%	7% / 64%	62%
Phi-4-mini 3.8B	3B class	91%	89%	72%	38% / 64%	64%
LFM2 2.6B	3B class	69%	61%	71%	5% / 43%	62%
SmolLM3 3B	3B class	83%	78%	70%	20% / 73%	62%
Ministral 3 3B	3B class	64%	28%	69%	2% / 5%	90%
Gemma 3 1B	1B class	79%	62%	58%	32% / 89%	48%
Llama 3.2 1B	1B class	73%	35%	57%	43% / 33%	48%
EuroMoE 2.6B-A0.6B	1B class	73%	67%	45%	80% / 94%	42%
LFM2 1.2B	1B class	5%	15%	43%	1% / 42%	21%
Apertus 1.5B	1B class	88%	58%	40%	99% / 75%	44%
LFM2 1.2B RAG	1B class / RAG-specific	23%	25%	40%	28% / 61%	24%
Pleias-RAG 1B	1B class / RAG-specific	73%	18%	38%	63% / 57%	35%
LFM2.5 1.2B	1B class	45%	18%	34%	53% / 75%	23%
Pleias-RAG 350M	1B class / RAG-specific	54%	9%	23%	85% / 68%	21%

English

Model	Class	Paragraph Correct	Long-text Correct	Overall	Hallucination with no answer (paragraph/long text)	Percentage correct among answered questions
Gemma 4 E4B	3B class	86%	87%	88%	3% / 18%	85%
Granite 4.1 8B	7B class	85%	78%	87%	3% / 15%	89%
Gemma 4 E2B	1B class	87%	88%	86%	6% / 32%	79%
Llama 3.1 8B	7B class	78%	77%	85%	2% / 11%	89%
Phi-4-mini 3.8B	3B class	87%	90%	85%	10% / 31%	80%
Llama 3.2 3B	3B class	78%	82%	85%	2% / 20%	85%
LFM2 2.6B	3B class	77%	76%	84%	2% / 13%	88%
Qwen3.5 9B	7B class	69%	80%	84%	1% / 9%	92%
Granite 4.0 1B	1B class	87%	77%	83%	6% / 29%	81%
Ministral 3 8B	7B class	78%	63%	82%	2% / 8%	91%
Qwen3.5 4B	3B class	63%	77%	82%	1% / 8%	92%
EuroLLM 9B	7B class	83%	58%	81%	3% / 10%	90%
Granite 4.1 3B	3B class	80%	57%	81%	1% / 10%	90%
LFM2 1.2B	1B class	83%	78%	81%	4% / 36%	76%
LFM2 1.2B RAG	1B class / RAG-specific	87%	86%	80%	10% / 51%	73%
SmolLM3 3B	3B class	90%	85%	79%	16% / 52%	71%
Apertus 8B	7B class	78%	82%	78%	7% / 47%	73%
Qwen3.5 2B	1B class	58%	52%	73%	2% / 11%	87%
OLMo 3 7B	7B class	52%	47%	71%	0% / 8%	90%
Gemma 3 1B	1B class	77%	66%	71%	11% / 57%	62%
Qwen3.5 0.8B	1B class	48%	56%	70%	2% / 21%	77%
Ministral 3 3B	3B class	53%	30%	67%	1% / 6%	90%
Llama 3.2 1B	1B class	57%	35%	65%	9% / 16%	73%
LFM2.5 1.2B	1B class	39%	24%	61%	1% / 9%	76%
MiniCPM5 2B	1B class	28%	25%	60%	0% / 3%	92%
EuroMoE 2.6B-A0.6B	1B class	57%	52%	55%	27% / 66%	53%
Apertus 1.5B	1B class	89%	79%	48%	93% / 97%	47%
Custom 320M	Custom	73%	53%	47%	55% / 91%	40%
Pleias-RAG 1B	1B class / RAG-specific	75%	44%	43%	78% / 73%	40%
Pleias-RAG 350M	1B class / RAG-specific	63%	44%	39%	67% / 92%	34%

CPU speed (i7-13700K · llama.cpp Q4_K_M)

Model	Size	Generation (tokens/sec)	Reading (tokens/sec)	Input tokens for long text	Time per long-text question
Custom 320M	193 MB	90.8	1,528	2,443	2.1 sec
EuroMoE 2.6B-A0.6B	1,541 MB	65.6	778	2,920	4.3 sec
Apertus 1.5B	935 MB	28.3	634	2,810 ※	4.8 sec
Qwen3.5 0.8B	517 MB	35.9	742	3,258	5.1 sec
LFM2.5 1.2B	697 MB	33.2	673	3,468	5.5 sec
Pleias-RAG 350M	219 MB	97.2	1,685	2,853 ※	6.0 sec
Llama 3.2 1B	770 MB	26.7	636	3,511	6.2 sec
LFM2 1.2B	697 MB	31.4	675	3,468	7.8 sec
LFM2 1.2B RAG	697 MB	32.8	678	3,468	8.1 sec
Gemma 3 1B	769 MB	29.6	398	3,055	8.2 sec
Qwen3.5 2B	1,251 MB	16.4	415	3,258	9.4 sec
Granite 4.0 1B	976 MB	22.9	414	3,636	9.4 sec
Gemma 4 E2B	3,257 MB	16.0	329	3,055	11.1 sec
MiniCPM5 2B	1,489 MB	16.7	313	3,796	13.0 sec
LFM2 2.6B	1,491 MB	14.5	278	3,468	13.3 sec
Phi-4-mini 3.8B	2,378 MB	9.9	208	2,974	15.0 sec
Ministral 3 3B	2,048 MB	9.5	215	3,100	15.9 sec
Llama 3.2 3B	1,926 MB	12.0	237	3,511	16.6 sec
SmolLM3 3B	1,827 MB	12.7	238	3,544	16.8 sec
Pleias-RAG 1B	710 MB	31.6	583	2,853 ※	16.9 sec
Granite 4.1 3B	2,002 MB	11.4	202	3,614	18.6 sec
Gemma 4 E4B	5,064 MB	8.1	171	3,055	21.1 sec
Qwen3.5 4B	2,655 MB	7.6	166	3,258	22.9 sec
Apertus 8B	4,824 MB	5.1	102	3,158	34.2 sec
OLMo 3 7B	4,264 MB	5.8	108	3,642	34.9 sec
Llama 3.1 8B	4,693 MB	5.3	101	3,511	37.3 sec
Ministral 3 8B	4,958 MB	5.0	95	3,100	37.4 sec
EuroLLM 9B	5,324 MB	4.7	86	2,920	37.6 sec
Qwen3.5 9B	5,512 MB	4.5	101	3,258	38.4 sec
Granite 4.1 8B	5,100 MB	4.8	88	3,614	46.6 sec

※ For Apertus 1.5B and Pleias-RAG, the average is calculated only for long texts that fit within the 4,096-token context window.