Following the previous GPT-2, this time I built a Croatian LLM from scratch on a single RTX 4090. The goal was RAG: a model that finds answers within supplied text and responds while quoting one supporting sentence. And it is intended to run in production on a CPU.

Here is the conclusion first.

• About 68 GPU hours from scratch to completion (including trial and error). Pretraining took 51 of those hours
• 320 million parameters. 71% accuracy on paragraph questions and 44% on long texts of around 3,000 tokens
• Answers each question in 3–16 seconds using only a CPU. Text appears slowly enough to follow with your eyes

Training Time

Stage	Time	Details
Training-data generation	About 4.5 hours	45,000 Wikipedia Q&As and 50,000 translated conversations generated with EuroLLM-9B (vLLM)
Tokenizer and tokenization	About 2.5 hours (CPU)	Built a 32,000-word Croatian vocabulary from scratch
Pretraining	50 hours 46 minutes	7 billion tokens. Extended context to 4,096 for the final 10%
SFT	About 35 minutes	Taught it to answer with citations and decline when there is no answer
Reinforcement learning (GRPO)	About 11 hours	400 steps, then 300 steps after adjustments

The custom tokenizer can represent Croatian in about half as many tokens as GPT-2's tokenizer. At 4,096 tokens, it can read roughly 17,000 characters.

The Finished Model

Item	Details
Parameters	320 million (24 layers, 1,024 dimensions)
Architecture	Llama / Qwen3 family (GQA, SwiGLU, RoPE, RMSNorm + QK-Norm)
Context length	4,096 tokens
Output format	Answer + Izvor: „one sentence copied directly from the context“. If there is no answer: “U tekstu nema odgovora na ovo pitanje. (There is no answer to this question in the text.)”
Runtime	Designed for CPU use (about 1.3 GB in fp32)

I also added an external mechanism that programmatically checks whether the generated citation actually exists in the context, replacing the response with “no answer” if it does not. The key is not trusting the model completely and making the evidence verifiable.

Problems I Found While Building It

1. It fabricated an answer every time, even when none existed. At first, I taught it ordinary question answering. Accuracy improved, but when given a passage with no answer, it made up a plausible-sounding answer 100% of the time. The worst possible behavior for RAG. Teaching citations and how to decline reduced fabrication to a few percent.

2. It declined too often. I set the reinforcement-learning score to “−0.5 for a wrong answer, 0 for declining.” On difficult long texts, it learned that declining was the profitable escape route, and began declining 70% of the time. It found the loophole in the scoring rule perfectly. When I changed the penalty for declining to −0.3, it swung in the other direction: it almost never declined, and fabricated answers 40% of the time. It went back and forth like a seesaw.

3. Citation transcription errors. On long texts, the answer would be correct but it would reorder words in the citation or quote the first sentence of the article. For long contexts, only about half had fully correct citations as well.

4. Errors that bypass citation verification. It accurately quoted a sentence saying “born in XX” from a different article mixed into the context, then gave the birthplace of the wrong person. The citation was genuine, so verification could not catch it.

5. Flaws in the evaluation data. In some supposedly answer-free contexts, part of the answer remained. The model was correct but was judged to have fabricated an answer.

Final Performance

300 evaluation questions each. I used the model at step 100 of the adjusted reinforcement-learning run, where the balance between declining and answering was best.

Context	Correct	Incorrect	Declined
Paragraph (about 100–400 tokens)	71.3%	16.7%	12.0%
Long context (1,500–3,500 tokens)	44.0%	20.3%	35.7%

Context with no answer	Correctly declined	Fabricated
Paragraph	97.7%	2.3%
Long context	92.3%	7.7%

After verification, the share of answered questions that were correct was 73.8%. That is still far from the 90% target.

Incidentally, when I let it attempt each question eight times and counted it as successful if even one attempt was correct, it reached 91% on paragraphs and 74% on long texts (measured at the final checkpoint). This figure barely changed before and after reinforcement learning; what changed was whether it could solve the problem every time. The numbers made it clear that reinforcement learning does not so much add new capabilities as make problems it can occasionally solve solvable every time.

Running It on a CPU

I built a test Web UI and ran it using only a CPU (i7-13700K). The rate at which characters appear is the generation speed.

Paragraph (263 tokens): 0.3 seconds to first character, about 11 tokens/second, 7 seconds total

再生

Long context (3,215 tokens): About 5 seconds to read the text, about 5 tokens/second, 19 seconds total. The yellow text is the cited sentence; it correctly finds the right sentence from among 3,000 tokens

再生

The videos are slightly slower because they were recorded while running. Without recording, measuring 20 questions at a time gave 15 tokens/second and 3.4 seconds per question for paragraphs, and 7 tokens/second and 16 seconds per question for long contexts. Since answers are short, around 50 tokens, this seems practical even on a CPU.

Thoughts

• For a small model, the most important thing is teaching it to admit what it does not know. In RAG, reducing fabrication matters more than accuracy. The combination of citations, declining, and external verification worked very well
• At 300 million parameters, long texts are the limit. Even with eight attempts, long texts only reached 74%. To aim for 90% on long texts, I would want around 1 billion parameters
• Verification needs to do more than confirm that the citation exists. It must check whether the citation contains the answer's number or name, and whether its subject is the same person mentioned in the question
• A 300M model that runs on a CPU is ideal for Q&A over documents that cannot be put online. It can answer based only on local documents on a laptop or server without a GPU. It takes several to more than ten seconds per question, and quantization should make it faster still (untested)

Next, I want to try a 1-billion-parameter version and aim for 90% on long contexts.