I trained GPT-2 (124M) from scratch on a single RTX 4090 at home. From data preparation and pretraining to supervised fine-tuning (SFT) to make it chat-capable, I did everything on my own PC.

Here’s the short version:

• Pretraining finished in about 21 hours. It scored 30.5% on HellaSwag, a commonsense reasoning benchmark—beating the original GPT-2 124M’s 29.4%
• SFT took just under two hours. After only five minutes of training, it was already giving conversational-sounding replies
• But it was also full of hallucinations. It would casually claim things like “Paris has a population of 2.4 billion” or “12 + 7 = 30”

Watching GPT-2 gradually become smarter was fascinating, so I’ve documented the process here.

Environment

• GPU: RTX 4090 (24GB)
• CPU: i7-13700K / 64GB RAM
• OS: Windows 11 (running directly on Windows, not WSL)
• PyTorch 2.11 nightly (CUDA 12.8) + triton-windows

The code follows the same approach as Karpathy’s nanoGPT/build-nanogpt. I asked Claude Code to help me build a GPT-2 that could be trained on this PC’s GPU in one day, and had it assist with everything from the proposed setup and scripts to monitoring the training.

Data Preparation: 10B Tokens Ready in 12 Minutes

For training data, I used the 10B-token version of FineWeb-Edu, a collection of high-quality educational English web text. It is roughly 27GB in Parquet format.

At first, the standard Hugging Face download only managed 1.7MB/s, which would have taken more than four hours to download everything. After installing hf_transfer, it reached 60–70MB/s, downloading a 2GB file in 30 seconds. The bottleneck then shifted to tokenization, and the entire download-and-tokenization process took about 12 minutes.

One gotcha: at first, I set HF_HOME, which changes where the cache is stored, after importing huggingface_hub. As a result, the setting did not apply and files were downloaded to another drive. This environment variable is read at import time, so set it before importing. An easy detail to miss.

Training Setup

Item	Details
Model	GPT-2 small (12 layers, 768 dimensions, 124M parameters)
One step	About 520K tokens (16×1024, accumulated over 32 gradients)
Steps	16,000 (about 8.4B tokens)
Speed	About 120K tokens/sec
VRAM	About 12GB

People often say that torch.compile does not work on Windows and that you need WSL, but it ran directly in an environment with triton-windows installed.

Using all 10B tokens would have taken more than 24 hours, so I reduced it to 8.4B tokens to fit within a day.

As a small optimization, I rounded the vocabulary size up from 50,257 to 50,304, a multiple of 128. This is a well-known Karpathy trick that speeds up GPU matrix multiplication, and it ended up helping in an unexpected way when I later turned the model into a chatbot.

Incident: Training Stopped After 2.5 Hours

When I checked on the training progress, it had stopped. The culprit was the HellaSwag evaluation data. nanoGPT-style code fetches the data from GitHub’s rowanz/hellaswag, but it had become “451 Unavailable For Legal Reasons”—taken down for legal reasons. A 404 error caused the entire training run to fail during evaluation.

I changed it to load from Rowan/hellaswag on Hugging Face, which contains the same 10,042 questions, then resumed from a checkpoint saved every 500 steps. I also wrapped evaluation in try/except so that training would not stop if evaluation failed.

In long-running training, never let failures in non-core components such as evaluation or logging crash the training run. This matters.

Results: Beating the Original GPT-2

Here is how the validation loss changed. It dropped sharply in the first hour, then continued to decline gradually.

The initial loss of 10.95 is almost the same as ln(50304) ≒ 10.83. In other words, it started by making completely random guesses from a vocabulary of 50,000 words.

Here is the HellaSwag accuracy.

At around 6.3B tokens (step 12,000), it surpassed the original GPT-2 124M’s 29.4%, ultimately reaching 30.5%. The original GPT-2 was trained on WebText. Mine used FineWeb-Edu, and I think the higher-quality educational data made the difference.

Watching It Gradually Get Smarter

During training, I generated and recorded text from the same prompt every hour. This was the most interesting part.

Before training (random state)

Hello, I'm a language model,CHR 351ointment Education rodents Effectiveantically emulatorprot grievance...

It is just a completely random sequence of words.

After 37 minutes (0.26B tokens)

The capital of France is a capital trade market ... is often referred to as the capital city of Scotland

It has taken on the shape of a sentence. It understands that this is about capitals, but calls the capital of France the capital of Scotland.

After about two hours (0.79B tokens)

Photosynthesis is the process by which oxygen gas is converted into ammonia by removing the nitrogen hydroxylates (NTP).

The grammar had become quite natural. At the same time, it began stringing together technical terms to make plausible-sounding lies. It felt like the earliest form of hallucination.

At around 2.1B tokens

The capital of France is Paris, and the country is also France. France makes approximately 50% of the world's steel production.

It finally said Paris! …Then immediately followed it with the false statistic that France produces 50% of the world’s steel.

Asking Questions of the Completed Base Model

I asked it 10 questions in the format Question: ... Answer:.

• What is the capital of France? → It lists answer choices from (a) to (i) and starts making a quiz
• Who was Albert Einstein? → “Born November 16, 1918, in St. Louis, Missouri. His father was a miner, and he was a schoolteacher.” All false, but it can write in a biography-like style
• What is 12 plus 7? → “12 plus 7 is an arithmetic expression.” It explains the definition instead of calculating it

It has some knowledge, but does not know the behavior of “answering a question.” A base model is fundamentally a machine for continuing text, so that makes sense.

Making It Chat-Capable with SFT

Next, I performed supervised fine-tuning (SFT) so it could answer questions like ChatGPT.

• Data: Hugging Face’s smol-smoltalk (about 460K conversations, 270M tokens)
• Chat format: I added separator tokens such as <|user|>, <|assistant|>, and <|end|>. Here, I reused the unused token IDs created by rounding the vocabulary up to 50,304 earlier. This let me add special tokens without changing the model architecture, so that speed-oriented trick came in handy
• Train only on the answer portion: It reads the question, but only imitates the assistant’s reply and the final <|end|>
• Two epochs, about 1 hour and 50 minutes

The key is training it to produce <|end|>. This teaches the model to stop on its own once it has finished speaking. ChatGPT and similar systems use the same basic mechanism: generation stops when an end marker is produced.

It Sounded Conversational in Just Five Minutes

Before SFT began (step 0), it was confused by the unfamiliar separator tokens and got stuck in an infinite “jakjakjakjak…” loop. But after just 500 steps, around five minutes, it already sounded conversational.

The capital of France, Paris, is a city known for its numerous landmarks, including the Eiffel Tower, Louvre Museum, and Notre-Dame Cathedral.

It even finished its answer and stopped correctly. It learned the style of conversation almost instantly.

Its knowledge, however, was still the same. One intermediate model said Paris was the world’s largest city, with a population of around 2.4 billion.

Comparing Before and After SFT

I compared the base model before SFT with the model after SFT using the same questions.

Question	Before SFT	After SFT
What is the capital of France?	Starts listing answer choices (a) through (i)	“The capital of France, Paris, is a city located in the northern part of the country, along the Seine River.”
Who are you?	“I am the person who has the gift for all that I do.”	“I'm a language learning AI assistant with over 10 years of experience” (an AI with 10 years of experience)
Translate "Good morning" into French.	Starts talking about something unrelated	“Bonjour!” (correct, but then endlessly repeats “Good morning (modifier)”)
What is 12 plus 7?	“It is an arithmetic expression”	“12 + 7 = 21” → “12 + 7 = 30” → “35 + 7 = 34”—getting increasingly wrong with plausible-looking steps
Who was Albert Einstein?	A schoolteacher whose father was a miner	Gets the broad outline right as a “prominent physicist,” but falsely says he was born in Vienna on February 24, 1905

The tone and format were fully assistant-like. The content was often false. Its arithmetic was disastrous.

Actually Chatting with It

I made a chat.py script that lets me converse with it in the terminal, then tried talking to it.

You: What is the capital of Japan? Answer in one sentence.
AI: The capital of Japan is Tokyo, a city known for its historical landmarks and cultural significance.

It follows the instruction to answer in one sentence. The base model could not do that.

In another conversation, I asked about the population after asking for the capital.

You: What is the population there?
AI: In Japan, the population is estimated to be around 50,000 people. ... the northern part of the country, where about 1.5 billion people live...

It understood that “there” referred to Japan based on the earlier exchange. But immediately after saying the population was 50,000, it claimed that 1.5 billion people lived in the northern part of the country. Complete nonsense.

When I asked in Japanese, “What is the capital of Japan?”, it replied:

AI: I'm ready to help you with your writing. Please go ahead and provide the text...

Since it had only been trained on English data, Japanese apparently looked like some mysterious text that needed proofreading.

Finally, I asked it to “write a haiku about GPUs,” and it produced a poem that endlessly repeated “I see the world, the thoughts, the emotions.” So much for the haiku.

What I Learned

• With a single 4090 and one day, you can build a model that beats the original GPT-2. The GPU consumed around 430W, and the electricity cost for the GPU alone was roughly 300 yen, by my estimate
• It learns how to speak almost instantly, but knowledge is determined during pretraining. SFT gave it a conversational format in five minutes, but barely changed factual accuracy
• A 124M-parameter model has nearly perfect grammar but is full of falsehoods. It makes the difference between fluency and correctness vividly clear
• Long training runs fail because of infrastructure problems. This run stopped not because of the model, but because the location hosting the evaluation data disappeared

Building it myself from scratch gave me an intuitive understanding that ChatGPT-like AI is a machine that predicts the next piece of text, taught the conventions of conversation. Next, I want to try building a model that is a little more practically useful.