Tabular foundation models such as TabPFN and TabICL have demonstrated the ability to predict on new datasets without undergoing any additional training, outperforming tuned XGBoost in a series of benchmarks. According to a report by Efrain Garay, these models utilize in-context learning—an approach known from large language models—to process tabular data in a single forward pass.
In a test conducted on 14 datasets from the Grinsztajn benchmark, the models showed consistent advantages. The primary benefit observed was not just in accuracy, but in computational efficiency; while tuned XGBoost requires significant time for hyperparameter searching, the foundation models provide results nearly instantaneously by using the training rows as context.
The performance advantage remains stable for datasets up to 32,000 rows. However, the models face challenges with "wide" tables. In tests involving the Bioresponse dataset with 419 columns, TabPFN's performance dropped significantly, falling below the results of an untuned XGBoost. This suggests that while the models handle long datasets well, the computational cost of the Transformer attention mechanism makes them sensitive to a high number of columns.
Sources
- TabPFN and TabICL vs. tuned XGBoost: the model that doesn't train won 14/14 （Hacker News Frontpage, 2026-09-28）