Llion Jones, a co-author of "Attention Is All You Need," appeared at the TED AI conference held in San Francisco. He pointed out that AI research is clinging too tightly to a single architecture.
Transformer Co-developer Llion Jones Expresses Concern Over Narrowing Research and Announces Departure from the Technology
This article is a translation. Read the Japanese original
Jones explained that pressure from investors and the concentration of talent are narrowing the scope of research. He stated that it has become difficult to see alternatives to the Transformer at present.
Currently serving as the CTO of Sakana AI, Jones said he decided earlier this year to significantly reduce the time he dedicates to Transformers, as he wishes to explore the next major technological leap.
He likened the current situation to the "exploration-exploitation trade-off," warning that focus is being placed almost exclusively on improving existing technology (exploitation), while unknown territories (exploration) are being neglected.
Reflecting on a period when he focused on improving RNNs (Recurrent Neural Networks), he mentioned that those efforts became meaningless with the advent of the Transformer. Similarly, he pointed out the possibility that a new breakthrough may be just around the corner.
Jones recalled that the birth of the Transformer happened organically, arising from things like casual conversations during lunch. He stated that an environment without the excessive competition and pressure seen today is what fostered creativity.
He expressed concern that researchers are currently choosing only "safe" research due to the fear of being scooped by competitors. He stated that this pressure is hindering scientific progress.
Sources: 'Attention is all you need' coauthor says he's 'sick' of transformers（HN 432pt・224コメント）（HN Search (backfill), 2025-10-24）