Elon Musk is reportedly planning to build a supercomputer to power the next generation of conversational artificial intelligence (AI) model Grok developed by his firm xAI, which raised $6 billion early this week in Series B funding round backed by Sequoia Capital, Andreesen Horowitz and Saudi prince Alwaleed Bin Talal.
Musk told investors that he aims to have the supercomputer with Nvidia H100 Hopper graphics processing units (GPUs) up and running by 2025, according to a report in The Information.
Nvidia H100 is a specialized GPU designed especially for generative AI models. It has helped Nvidia gain over $1.1 trillion in market valuation in February. Microsoft and Meta are the biggest consumers of the H100 GPUs.
Powered by hundreds or thousands of GPUs, supercomputers can accelerate complex calculations by providing a million times more computing power than a regular PC. The world’s fastest supercomputer Frontier runs on over 30,000 AMD GPUs, while Google’s A3 supercomputer, used for training Gemini models, is powered by 26,000 Nvidia H100 GPUs.
Large language models (LLMs) like GPT-4 are trained on massive datasets of text and codes. This involves complex calculations, which a PC will take decades to process. Supercomputers, with their thousands of central processing units (CPUs) and GPUs, can speed up the training process by carrying out billions of calculations per second, leading to creation of more advanced AI models.
Training LLMs with countless GPUs consumes a massive amount of energy, raising concerns about its impact on the planet, which is already coping with an unprecedented rise in heat waves across the world. For instance, training GPT-3 took the equivalent of running a single computer for 355 years, consuming a staggering 284,000 kWh of energy, according to nnlabs.org.
xAI was founded by Musk in July 2023 to take on OpenAI, which he had started as a non-profit with Sam Altman in 2015. Musk parted ways with the firm in 2018 citing conflict of interest. Since the release of ChatGPT and the subsequent rise of OpenAI’s valuation to $80 billion, Musk has emerged as one of its most vocal critics.
In February, Musk slammed Altman for turning OpenAI into a for-profit organization. A month later, Musk filed a lawsuit against OpenAI for violating the principles on which it was founded. OpenAI rejected Musk’s claim as frivolous.
Image credit: Wikimedia Commons