Metadata
- Author: Jack Morris
- Full Title:: There Are No New Ideas in AI… Only New Datasets
- Category:: 🗞️Articles
- Document Tags:: Foundation Models,
- URL:: https://blog.jxmo.io/p/there-are-no-new-ideas-in-ai-only
- Read date:: 2025-07-08
Highlights
If you squint just a little, these four things (DNNs → Transformer LMs → RLHF → Reasoning) summarize everything that’s happened in AI. (View Highlight)
each of these four breakthroughs enabled us to learn from a new data source:
- AlexNet and its follow-ups unlocked [ImageNet](http://(https//www.image-net.org/), a large database of class-labeled images that drove fifteen years of progress in computer vision
- Transformers unlocked training on “The Internet” and a race to download, categorize, and parse all the text on The Web (which it seems we’ve mostly done by now)
- RLHF allowed us to learn from human labels indicating what “good text” is (mostly a vibes thing)
- Reasoning seems to let us learn from “verifiers”, things like calculators and compilers that can evaluate the outputs of language models (View Highlight)
As one salient example, some researchers worked on developing a new BERT-like model using an architecture other than transformers. They spent a year or so tweaking the architecture in hundreds of different ways, and managed to produce a different type of model (this is a state-space model or “SSM”) that performed about equivalently to the original transformer when trained on the same data. This discovered equivalence is really profound because it hints that there is an upper bound to what we might learn from a given dataset. All the training tricks and model upgrades in the world won’t get around the cold hard fact that there is only so much you can learn from a given dataset. (View Highlight)
And maybe this apathy to new ideas is what we were supposed to take away from The Bitter Lesson. (View Highlight)