Hugo Vergnes trained a 3.8 billion parameter language model from scratch for $998, scoring 0.384 on the CORE benchmark after 43 hours on eight rented B200 GPUs. That number matters because of what sits next to it in the comparison table. Karpathy’s nanochat d32, a 1B model trained for roughly the same $1,000 budget, scores about 0.310. GPT-2 scores 0.2565. For the same money, a solo project landed well ahead of the reference implementation it was inspired by.
Five fixes that made the difference
The writeup is candid about failure. An earlier 858M model, trained for six days on an A100, scored worse on PIQA than GPT-2 124M, a model seven times smaller from 2019. The post-mortem identified five problems, and fixing them is most of the story.
The cosine learning rate schedule decaying to zero was wrong: the loss curve went flat after 70% of training, so the last 30% of the compute budget produced almost nothing. Switching to a trapezoidal schedule, hold flat, then linear cooldown over the final half, kept eval loss descending right to the last step. The peak learning rate was too conservative for small models. AdamW on every parameter was wasteful: Muon on the matrix parameters costs about 25% more per step but converges much faster overall, and at seven gradient accumulation steps that overhead dilutes to roughly 4% of runtime. FineWeb-Edu gave way to ClimbMix, a data change the author calls a tremendous jump in convergence speed, matching Karpathy’s own finding. And FP8 training with vocab padding to a multiple of 64 added about 33% throughput.
The architecture
The final model is a modern Llama-style stack: RMSNorm, RoPE, grouped query attention with 24 query heads and 8 KV heads, relu-squared MLPs, QK-norm, and logit softcap. Two details stand out. ResFormer-style value embeddings account for 721M parameters, a full 19% of the model, in 14 tables spread across every other layer. And the untied LM head plus token embeddings add another 309M. The transformer blocks themselves are 2.8B of the 3.85B total.
Infrastructure discipline over cleverness
The most transferable part of the post is not the model config, it is the engineering argument. little-lm is config-driven: every run is a YAML file, components self-register, and swapping an optimizer or dataset is a one-line diff. The author’s claim is that good infrastructure pays for itself the first time you hit a convergence problem, and that the best framework is one where you can read the config and know exactly what will happen, with no hidden mechanics.
He also ran the same 3.8B model at 1024 context instead of 2048 and scored 0.338 for $820. Halving context roughly doubles batch size at fixed memory with barely any throughput change per token, so shorter context is a legitimate cost lever when your use case allows it.
Why this matters
The space between toy GPT replicas and lab-scale training has been poorly documented. This post is a concrete data point in it: $1,000 now buys a model that beats GPT-2 by a wide margin, and that boundary keeps moving as the frontier advances. For anyone who has wanted to understand pretraining by doing it rather than reading about it, the full recipe, including the exact YAML and the parameters that failed, is published. That last part matters as much as the results. Knowing which schedule flattened and which optimizer was worth its overhead is the difference between reading about training and being able to run one.
The hardware economics
The choice of B200s over H100s is its own lesson. The author found them better value per unit of work, and the FP8 path through torch’s scaled matrix multiply pushed throughput further. Small things compounded: padding the vocabulary from 50,257 to 50,304 sounds trivial, but unaligned vocab sizes leave tensor cores idle. The 19% of parameters spent on value embeddings is another interesting trade. Those tables add capacity without adding depth, and the fact that more than a sixth of the budget went to them is a quiet data point in the ongoing argument about where parameters actually earn their keep in small models.
The comparison table also shows how compressed the training timeline has become. GPT-2 required a research lab’s worth of resources in 2019. nanochat put a 1B model on the board for $1,000. little-lm pushed the same budget to a 3.8B model trained on 65B tokens in under two days of wall clock time. None of this threatens frontier labs, but it steadily erodes the gap between what a curious individual can understand firsthand and what only institutions could previously touch.
Who should read the full post
Anyone planning a small-scale pretraining run should treat it as a checklist of known failure modes: conservative learning rates that waste the training tail, optimizer choices that look expensive per step but win per token, and dataset quality that matters more than almost any architectural tweak. The framework, config files, and data recipes are described in enough detail to rebuild the run, and the author is explicit about what he still does not know, which is rarer than it should be in training writeups.