Beyond Transformers: An AI Pioneer's Call to Explore the Next Frontier

As a prompt engineer, my world is built on the transformer architecture. It’s the bedrock of every major large language model I interact with, the engine behind the conversational AI that has reshaped our digital landscape. So, when Llion Jones, a co-author of the seminal “Attention Is All You Need” paper and the man who literally named the transformer, says he’s “absolutely sick of them,” it’s impossible not to sit up and listen.

In a recent, strikingly candid talk reported by VentureBeat, Jones, now the CTO of Sakana AI, argues that the very success of transformers has led to a dangerous narrowing of AI research. The immense investment and competitive pressure in the field have created an environment of exploitation over exploration. Instead of venturing into the unknown to find the next paradigm-shifting architecture, labs are stuck in a cycle of incrementally improving the current one. We’re all polishing the same engine, trying to squeeze out a few more horsepower, while a completely new mode of transport might be waiting just over the horizon.

Jones’s critique is a bucket of cold water for many in the AI space, but it’s a necessary one. He recalls the pre-transformer era, where researchers were endlessly tweaking recurrent neural networks (RNNs). That work became largely obsolete overnight. Are we repeating that history? Are we so focused on scaling our current models that we’re missing the fundamental breakthrough that will make them look like quaint relics?

The pressure is immense. With billions of dollars on the line, researchers are incentivized to pursue safe, publishable, and profitable work. The freedom to explore risky, “wild ideas”—the very freedom that gave birth to the transformer at Google—has been replaced by a race for marginal gains. Jones suggests that this hyper-competitive environment is paradoxically damaging the science, leading to rushed papers and a herd mentality.

At his new venture, Sakana AI, Jones is trying to rekindle that original spirit of curiosity. He’s betting that true innovation comes not from million-dollar salaries and relentless pressure, but from giving brilliant minds the freedom to pursue research that “wouldn’t happen if you weren’t doing it.” They are exploring nature-inspired approaches and concepts like a “continuous thought machine,” deliberately stepping off the beaten path.

This isn’t to say that work on transformers should stop. The architecture will continue to deliver immense value for years to come. But Jones’s message is a powerful call for diversification. As an industry, we can afford to do more. We can walk and chew gum at the same time—we can refine the powerful tools we have while simultaneously funding and encouraging the search for what comes next.

For prompt engineers, this is a crucial reminder that the ground beneath our feet is not as solid as it seems. The models we master today could be superseded tomorrow. Our expertise must be not just in manipulating current systems, but in understanding the fundamental principles of intelligence, communication, and learning, so we are ready to adapt to whatever new architecture emerges.

Jones’s call to action is a challenge to all of us: to collectively turn up the “explore dial.” The next revolution in AI won’t be a bigger transformer; it will be something we haven’t even imagined yet. And we’ll only find it if we have the courage to look.