
I Fine-Tuned a 0.8B LLM for RAG in 20 Languages: Accuracy from 74% to 94%, 7 Seconds per Question on CPU
I wanted a multilingual RAG model that runs on CPU. Give it a passage and ask a question, and it answers from the passage. If the answer is not in the passage, it says “there is no answer” instead of making something up. So I built one by fine-tuning Qwen3.5 0.8B (800 million parameters) for 20 languages.
Here is the conclusion first.
- Accuracy improved from 73.8% to 93.8%. It came within one point of the unfine-tuned Qwen3-4B (94.7%), despite being one-fifth the size
- The biggest improvement was fixing hallucination. The original 0.8B would still give some kind of answer nearly 40% of the time when given a long passage with no answer
- 7 seconds per question on CPU. That includes reading a roughly 2,500-token passage and answering it. The 4B model takes 60 seconds
- I could not get past 95%. Even after changing the learning rate, number of epochs, and data volume, it plateaued in the 94% range. This seems to be the limit of a 0.8B model
What I measured
I created questions from Wikipedia in 20 languages, including Japanese, English, Chinese, Croatian, and Hindi. For each question, I gave the model four kinds of passages.
- Paragraph: a short paragraph containing the answer
- Long context: a passage of several thousand tokens, with the answer-containing paragraph mixed into unrelated articles
- No answer (short/long): a passage that does not contain the answer. It is correct if the model says “there is no answer”
I call the average across these four “accuracy.” Qwen3-32B generated the questions and graded the answers. To check whether grading was biased toward Qwen, I regraded them with gpt-oss-120b, from a different model family; the difference was within 0.5 points.
I also tested with differently structured questions.
- XQuAD: eight-language versions of human-written questions
- hr0: a Croatian test I previously created with another model
With only questions produced in the same way as the training data, it would be impossible to tell whether the model had simply become accustomed to that format.
The original 0.8B makes things up when there is no answer
Before fine-tuning, Qwen3.5-0.8B scored as follows (average across 20 languages).
| Paragraph | Long context | No answer, short | No answer, long |
|---|---|---|---|
| 68.9 | 71.0 | 87.9 | 63.2 |
Given a long passage with no answer, it still answered something nearly 40% of the time. In Hindi, it scored only 15 points. That is a problem for RAG.
What I tried
I trained on an RTX PRO 6000 (96GB) rented through vast.ai.
1. LoRA on RAG data in 20 languages
The training data consisted of 500 questions × 20 languages. I created examples with the four types of passages, for a total of 18,025 examples. Correct answers were one-sentence responses; when there was no answer, the response used that language’s standard phrase for “there is no answer.”
One epoch with LoRA (rank 64) took 1.7 hours.
| Paragraph | Long context | No answer, short | No answer, long | |
|---|---|---|---|---|
| Original 0.8B | 68.9 | 71.0 | 87.9 | 63.2 |
| After LoRA | 91.7 | 89.0 | 98.4 | 93.4 |
That alone produced a major jump. No-answer long context went from 63 to 93, so hallucinations were almost eliminated.
Incidentally, when I first tried this with 2B, training on only three languages—Croatian, German, and Turkish—improved all 20 languages. The original model’s weakness seems not to have been knowledge or reading comprehension, but rather its tendency to follow instructions: answer when appropriate, and decline when not.
2. It declines too readily on human-written questions
When tested on XQuAD’s human-written questions, it often said “there is no answer” even when an answer was present.
For example, it may be asked, “What is another name for a rotor?” while the paragraph says only “a rotating disk.” It would decline on questions like this, where the wording differs between the question and the paragraph.
The cause was in the training data. Qwen3-32B, which generated the questions, tended to reuse the paragraph’s wording verbatim. So the model seems to have learned: “If I cannot find the exact same words, decline.”
I therefore had gpt-oss-120b generate 400 additional, more varied examples per language.
- Questions phrased differently from the paragraph
- Choppy fragments resembling search results
- Articles that are related but contain no answer, making them misleading
- Long contexts of 1K–8K tokens
This raised XQuAD paragraph performance from 79.6 to 83.6, and long-context performance from 75.5 to 79.2. This was almost the only thing that helped at this stage.
From this point onward, I trained all parameters rather than using LoRA. With the same data, the results were nearly identical either way.
3. What did not help
| What I tried | Result |
|---|---|
| Tripling the learning rate (1e-5 → 3e-5) | Validation loss clearly worsened, so I stopped early |
| Training for 2 epochs | Validation loss fell, but accuracy stayed the same. XQuAD dropped slightly |
| Doubling the more varied data | +0.1 on my own questions. XQuAD and hr0 declined |
| Adding human-written questions (SQuAD and TyDi QA) | Counterproductive. XQuAD no-answer long context fell from 99.8 to 91.2 |
I expected human-written questions to help, but the opposite happened. XQuAD is built from SQuAD paragraphs. So training on SQuAD seems to have taught the model that “a SQuAD-like paragraph has an answer.” Data that is too close to the evaluation set is risky.
4. Q4 quantization costs one point
Since the model needs to run on CPU, I converted it to llama.cpp GGUF and quantized it.
With 2B, Q4_K_M caused almost no drop. But 0.8B lost about one point. Smaller models are more sensitive to quantization. Q6_K or Q8_0 restored the score, reaching 95.4 on the Croatian test.
Results
I compared the final 0.8B model with the original and larger models. All scores are averages across the four passage types, and the 0.8B and 2B models are in their CPU-ready Q4_K_M form.
| Model | Size | Questions in 20 languages | XQuAD (human-written questions) | hr0 |
|---|---|---|---|---|
| Qwen3.5-0.8B (original) | — | 73.8 | 74.8 | 72.7 |
| 0.8B (fine-tuned) | 0.67GB | 93.8 | 89.9 | 90.0 |
| 2B (fine-tuned the same way) | 1.56GB | 96.4 | 92.4 | 93.5 |
| Qwen3-4B-Instruct-2507 (not fine-tuned) | 2.3GB | 94.7 | 94.0 | 91.2 |
- 0.8B gained 20 points and came within one point of 4B. The 2B model surpasses 4B
- On human-written questions (XQuAD), it still trails 4B by 4 points. Its weakness with differently phrased questions is not fully resolved
- Of the questions 0.8B got wrong, 64% are answered correctly by 2B. I think the rest reflects the limit of model size. Choose 2B if accuracy is the priority
Speed (laptop CPU)
I measured on 8 threads of an i7-11800H. It is a laptop CPU whose clock speed drops to about half under sustained load, so these are on the slower side.
| Reading the passage (4K tokens) | Generation | One long-context question | |
|---|---|---|---|
| 0.8B | 314 tok/s | 53 tok/s | about 7 seconds |
| 2B | 97 tok/s | 21 tok/s | about 25 seconds |
| 4B | 42 tok/s | 11 tok/s | about 60 seconds |
For a paragraph alone, it takes under one second.
Cost
The total came to about $75 on vast.ai. That includes 2B training and generating evaluation data.
Summary
- Even 0.8B can reach around 94% accuracy when fine-tuned for RAG. In particular, it almost completely fixes hallucinations when there is no answer
- What helped was increasing the variety of training data. More data, more epochs, and a higher learning rate did not help
- Choose 2B for accuracy and 0.8B for speed. I think both are more practical than an unfine-tuned 4B model



