On Wednesday, Meta’s CEO Mark Zuckerberg marked a significant advancement in generative AI training, announcing that the upcoming version of the Llama model is being developed on a GPU cluster that surpasses anything previously reported.
Zuckerberg informed investors and analysts during an earnings call that the progress on Llama 4 is substantial, with a first launch anticipated in the early part of the next year. He stated, “We’re training the Llama 4 models on a cluster that exceeds 100,000 H100s, surpassing anything I’ve encountered regarding what others are achieving.” He was referencing the Nvidia chips that are commonly utilized for AI training. “I expect the smaller models of Llama 4 will be ready first,” he added.
There is a prevailing belief that expanding the scale of AI training, through increased computing power and data, is essential for creating significantly more capable AI models. While Meta currently seems to hold an advantage in this area, it’s expected that many major players in the AI space are also striving to utilize compute clusters with over 100,000 advanced chips. Earlier in the year, Meta and Nvidia provided insights into clusters that featured around 25,000 H100s, which were instrumental in developing Llama 3. In July, Elon Musk boasted that his xAI initiative had collaborated with X and Nvidia to establish 100,000 H100s, proclaiming it “the most powerful AI training cluster in the world!”
During the announcement on Wednesday, Zuckerberg refrained from disclosing specifics about the advanced capabilities expected from Llama 4. However, he alluded to “new modalities,” “stronger reasoning,” and “much faster” performance.
Meta’s strategy regarding AI is becoming an unpredictable factor in the competitive landscape. Unlike models from OpenAI, Google, and other industry giants that are only accessible via an API, Llama models can be fully downloaded free of charge. This has attracted considerable interest from startups and researchers who desire complete control over their models, data, and associated computing expenses.
While Meta promotes Llama as “open source,” the licensing does carry some limitations on its commercial utilization. Additionally, the firm does not share specifics about the models’ training processes, which restricts external analysis of their functionality. The initial version of Llama was released in July 2023, with the most recent iteration, Llama 3.2, made available this September.
Constructing Llama 4 from such an extensive collection of chips will likely present distinctive engineering difficulties and demand substantial energy resources. On Wednesday, Meta executives avoided answering an analyst’s inquiry regarding energy access constraints affecting various regions in the U.S., which have hindered companies’ endeavors to create more powerful AI technologies.
As per one estimation, a setup containing 100,000 H100 chips would necessitate 150 megawatts of electricity. In comparison, the largest supercomputer at a national lab in the U.S., El Capitan, only requires 30 megawatts. Meta is projecting expenditures of up to $40 billion this year for data centers and other infrastructure, marking a rise of over 42 percent from the previous year, with expectations of even steeper increases in spending for the upcoming year.
Meta’s total operating expenses have risen around 9 percent this year. However, overall revenue—primarily from advertisements—has skyrocketed by more than 22 percent, resulting in increased profit margins and greater earnings even as it invests billions into its Llama initiatives.
In contrast, OpenAI, regarded as the leading figure in the advancement of state-of-the-art AI, is depleting its funds despite charging developers for the use of its models. What currently remains a nonprofit endeavor has announced that it is in the process of training GPT-5, which will succeed the model that powers ChatGPT today. OpenAI asserts that GPT-5 will be more extensive than its predecessor but has not disclosed details regarding the computer cluster employed for its training. Additionally, OpenAI has mentioned that GPT-5 will not only be larger but will also integrate other advancements, including a newly devised method for reasoning.
CEO Sam Altman has stated that GPT-5 will mark “a notable advancement” compared to the earlier model. Just last week, Altman addressed reports indicating that OpenAI’s next major model could be launched by December by commenting on X, “fake news is out of control.”
On Tuesday, Google CEO Sundar Pichai announced that the company is currently developing the latest iteration of the Gemini suite of generative AI models.
Meta’s transparent strategy regarding AI has sparked debate within the tech community. Some specialists in artificial intelligence express concern that making more advanced AI models readily accessible could pose risks, as it might assist malicious actors in executing cyberattacks or developing chemical and biological weaponry. Despite the measures taken to fine-tune Llama before its public launch to prevent misuse, experts note that bypassing these limitations is quite straightforward.
Zuckerberg is optimistic about the open source approach, even amid the trend of proprietary systems advocated by companies like Google and OpenAI. “It seems pretty clear to me that open source will be the most cost effective, customizable, trustworthy, performant, and easiest to use option that is available to developers,” he stated recently. “And I am proud that Llama is leading the way on this.”
Zuckerberg also mentioned that the advanced features of Llama 4 are expected to enhance a greater variety of functionalities across Meta services. As of now, the unique service derived from Llama models is the chatbot, similar to ChatGPT, known as Meta AI. This chatbot can be found on platforms like Facebook, Instagram, and WhatsApp, among others.
According to Zuckerberg, more than 500 million individuals engage with Meta AI each month. Looking ahead, Meta anticipates revenue generation through advertising facilitated by this feature. “There will be a broader array of inquiries that users will leverage, and monetization opportunities will arise over time as we progress,” explained Meta CFO Susan Li during a recent call. With advertising potential, Meta may indeed succeed in offering Llama as a subsidized tool for everyone else.