TurboGPT is a tool designed for training a tiny, byte-level GPT model using CUDA C++. According to the project documentation, a 22KiB Transformer model can be trained in 13 seconds.
TurboGPT enables training of 22KiB Transformer in 13 seconds using CUDA C++
The project is released under the MIT license and supports building on Linux with Nix, Windows with Visual Studio 2022 C++ tools, and CUDA 13.4. Users can execute the training using a PowerShell script, specifying the GPU compute capability via the --CudaArch flag. The training process includes options to log results and save checkpoints, which can be resumed using the --load flag. The application also generates TensorBoard-compatible logs stored in a directory for analysis.
Sources
- Show HN: TurboGPT: train 22KiB transformer in 13s （Hacker News Frontpage, 2026-09-29）