Annie Sexton, a developer at ngrok, stated in her blog that data compression and Large Language Models (LLMs) are fundamentally attempting to solve the same challenge: "data prediction."
She notes that both technologies aim to reduce data redundancy and achieve more efficient representation, and are thus linked through the mathematical framework of entropy in information theory.
In compression algorithms, an entropy coder encodes data based on probabilities presented by a model. The higher the probability of a specific symbol appearing, the better the compression ratio.
In real-world data, the probability of a symbol is heavily influenced by the surrounding context. By using an "order-N model" that considers the previous N symbols, it is possible to obtain more powerful probabilities.
LLMs predict the probability distribution of the next token based on a given context. This predictive capability is said to be directly linked to the performance of the "model" in compression.
Source: The essence shared by compression and LLMs is "data prediction" (GIGAZINE, 2026-09-06)