Elon Musk’s startup xAI has revealed its latest creation: a supercomputer named Colossus.
This massive AI training system currently boasts 100,000 Nvidia Hopper H100 processors.
Located in Memphis, Tennesse, Colossus is set to be expanded with 50,000 of Nvidia’s new, more advanced H200 series chips, roughly twice as powerful as the H100s.
That would almost certainly make Colossus the most powerful computer on the planet if it wasn’t already.
For context, Meta had announced plans to accumulate 350,000 H100 GPUs by the end of 2024, while OpenAI was believed to use just around 25,000 GPUs for GPT-4, though it could be many more.
Musk announced the launch of Colossus himself on X over the Labor Day weekend, stating, “From start to finish, it was done in 122 days. Colossus is the most powerful AI training system in the world. Moreover, it will double in size to 200k (50k H200s) in a few months.”
This weekend, the @xAI team brought our Colossus 100k H100 training cluster online. From start to finish, it was done in 122 days.
Colossus is the most powerful AI training system in the world. Moreover, it will double in size to 200k (50k H200s) in a few months.
Excellent…
— Elon Musk (@elonmusk) September 2, 2024
Powering the future of Grok
The immediate purpose of Colossus is to train xAI’s large language model (LLM) known as Grok.
Currently, Grok is only available to paying subscribers of Musk’s X social media platform. The company has already released an early beta of Grok-2, which was trained on around 15,000 Nvidia H100s.
Despite this relatively small training data set, Grok-2 is already considered among the most capable AI large language models according to competitive chatbot leaderboards.
It’s also open-source, bucking the trend of closed-source model releases from competitors OpenAI, Anthropic, and Google.
Musk’s ambitions for Grok are extravagant. He’s already looking ahead to Grok-3, slated for release in December 2024.
In a July interview with Jordan Peterson, Musk boldly claimed, “Grok-3 should be the most powerful AI in the world at that point.”
The massive increase in GPU count for training Grok-3 suggests that Musk is serious about this claim.
What about xAI’s other goals?
Founded in July 2023, xAI ultimately wants to find out “What the hell is really going on?” In Musk’s words, that means probing fundamental questions about reality, dark matter, the Fermi Paradox, and other cosmic mysteries.
xAI’s founding team was picked to pursue that vision. Igor Babuschkin, who previously worked on the Large Hadron Collider at CERN, stated that the company wants to “really make a dent in our understanding of the universe.”
Jimmy Ba, an AI researcher from the University of Toronto, explained the goal of building a “general purpose problem-solving machine” to tackle humanity’s most challenging problems.
To fuel its research and development, xAI secured $6 billion in a Series B funding round in May 2024, backed by venture capital firms like Andreessen Horowitz and Sequoia Capital, as well as deep-pocketed investors like Fidelity and Saudi prince Alwaleed bin Talal’s Kingdom Holding.
Many Tesla experts believe that Grok could eventually power the AI behind Tesla’s humanoid robot, Optimus. Musk claimed Optimus could generate $1 trillion in annual profits for Tesla. Musk has also hinted at potentially proposing a $5 billion investment from Tesla into xAI, which some shareholders have welcomed.
With Colossus now online, might xAI be poised to transition beyond creating more LLMs and attempt something bold and new?
Sure, Grok-3 will probably be very powerful. It’ll probably only be slightly more powerful than its peers, and only for a short time until something overtakes it.
You have to question the fundamental long-term goals here. AI companies are grimly competing with similar models trained with virtually identical hardware stacks. It’s expensive, and innovation risks plateauing with small, incremental gains.
This leaves the field open for one of the main players to peer beyond the era of LLMs. With Colossus, will xAI be the first one to step up as Musk hinted it would when it was founded?
