AI Scaling

The observation and practical application of empirical laws that relate the size of an AI model, the volume of training data, and the amount of compute used to predictable improvements in model capability. The foundational 'scaling laws' paper published by Kaplan et al. at OpenAI in 2020 demonstrated that model loss decreases smoothly and predictably as model parameters, dataset size, and compute are increased, following power-law relationships. This enabled AI labs to predict with reasonable accuracy how much a larger model would improve before training it, transforming AI development from experimental art to engineering discipline. The Chinchilla scaling laws (Hoffmann et al., 2022) refined this by showing that earlier frontier models were significantly undertrained relative to their parameter count, and that optimal training balances model size with dataset size. More recently, 'inference-time scaling' or 'test-time compute' has emerged as a complementary axis, demonstrating that allowing models more time to reason through problems before responding produces substantial capability gains without changing model weights. Example: OpenAI's o1 model family, released in late 2024, demonstrated inference-time scaling by training models to use extended internal chain-of-thought reasoning before producing answers. On complex mathematics and coding benchmarks, o1 significantly outperformed earlier GPT-4 models that had more parameters but less reasoning computation, establishing test-time compute as a distinct axis of capability improvement. Why it matters for AI: AI scaling has been the dominant mechanism behind the rapid capability advances of the past several years. Understanding scaling laws allows labs to allocate compute budgets efficiently and predict capability timelines. The emergence of inference-time scaling has extended the scaling paradigm beyond the simple 'bigger model = better' framework, suggesting that the compute investment in deploying AI (not just training it) will become increasingly important.

Explore the full Web3 Glossary — 2,000+ expert-curated definitions. Need guidance? Talk to our consultants.