Cal Newport has taken the position of superintelligence skeptic and his latest article – What If AI Doesn’t Get Much Better Than This – made for interesting reading. A few notes –
I appreciated his quick description of pre-training vs. post-training to improve models.
A useful metaphor here is a car. Pre-training can be said to produce the vehicle; post-training soups it up. In the scaling-law paper, Kaplan and his co-authors predicted that as you expand the pre-training process you increase the power of the cars you produce; if GPT-3 was a sedan, GPT-4 was a sports car. Once this progression faltered, however, the industry turned its attention to helping the cars that they’d already built to perform better. Post-training techniques turned engineers into mechanics.
After delving into depth into relatively linear advances of more recent models like GPT-5 vs. GPT-4, he goes onto share a possible scenario.
If these moderate views of A.I. are right, then in the next few years A.I. tools will make steady but gradual advances. Many people will use A.I. on a regular but limited basis, whether to look up information or to speed up certain annoying tasks, such as summarizing a report or writing the rough draft of an event agenda. Certain fields, like programming and academia, will change dramatically. A minority of professions, such as voice acting and social-media copywriting, might essentially disappear. But A.I. may not massively disrupt the job market, and more hyperbolic ideas like superintelligence may come to seem unserious.
And he ends with a note that recommends less hubris and more care.
Even the figures we might call A.I. moderates, however, don’t think the public should let its guard down. Marcus believes that we were misguided to place so much emphasis on generative A.I., but he also thinks that, with new techniques, A.G.I. could still be attainable as early as the twenty-thirties. Even if language models never automate our jobs, the renewed interest and investment in A.I. might lead toward more complicated solutions, which could. In the meantime, we should use this reprieve to prepare for disruptions that might still loom—by crafting effective A.I. regulations, for example, and by developing the nascent field of digital ethics.
The appendices of the scaling-law paper, from 2020, included a section called “Caveats,” which subsequent coverage tended to miss. “At present we do not have a solid theoretical understanding for any of our proposed scaling laws,” the authors wrote. “The scaling relations with model size and compute are especially mysterious.” In practice, the scaling laws worked until they didn’t. The whole enterprise of teaching computers to think remains mysterious. We should proceed with less hubris and more care.
It is early days and there’s certainly a lot of unknowns. One idea I always reflect on when I read these sorts of takes is – everyone is talking their book, skeptics included.
The truth remains that AI, even in its current form, has a lot of potential to boost productivity and change workflows. Whether we see this journey end in Superintelligence in the short-term is unclear.
But, regardless, there’s enough disruption to go around and plenty of work to be done to use these changes for good and society for the changes ahead.
