Hugo Vergnes announced that he successfully trained a 3.8B parameter language model called "little-lm" to demonstrate that meaningful model training is possible with a small budget. He stated that he completed the training of 65 billion tokens in 43 hours for a total cost of $998, utilizing a rented NVIDIA B200 GPU and his own RTX 5090.
As a result of the training, the model recorded 0.384 on CORE (a benchmark measuring model capability), significantly outperforming the 1.5B GPT-2 model (0.2565) released by a large research institution in 2019. He attributed the improved convergence to the use of Muon (an optimizer for matrix parameters), acceleration via FP8 (8-bit floating point), and the adoption of the ClimbMix dataset during the training process.
Furthermore, he highlighted that infrastructure setup directly impacts development efficiency, having built a framework where the entire experimental process can be managed via YAML configuration files. This case serves as an example that individuals can develop advanced models if they have appropriate software engineering discipline and infrastructure, even without the massive computational resources available to large research labs.
Source:
- Training a 3.8B LLM to 0.384 CORE for $998 – Hugo Vergnes (Hacker News Frontpage, 2026-09-10)