The phase transition from memorization to compression during training is something I've obsreved in production models but never quite articulated this clearly. The connection to PCA is clever since most folks already get dimensionality reduction. One thing tho: the comparison between 4B and 70B models feels optimistic given current benchmarks, atleast outside very specific domains. Compression definitely wins on inference cost, but theres still edge cases where raw capacity matters (like multi-hop reasoning or handling obscure domain knowledge).
AI becomes intelligent by forgetting more.
The compression principle explains why smaller, focused implementations often outperform massive general models.
If you're thinking about what this means for your team's AI strategy: https://vivander.substack.com/p/something-shifted-when-i-read-openais
The idea that intelligence is compression, not memorization, completely reframes how we should think about AI success.
Most orgs are still measuring AI like they measure databases: how much can it hold?
The real question is how well does it forget what doesn't matter.
The phase transition from memorization to compression during training is something I've obsreved in production models but never quite articulated this clearly. The connection to PCA is clever since most folks already get dimensionality reduction. One thing tho: the comparison between 4B and 70B models feels optimistic given current benchmarks, atleast outside very specific domains. Compression definitely wins on inference cost, but theres still edge cases where raw capacity matters (like multi-hop reasoning or handling obscure domain knowledge).