Will Knight
Jerome Pesenti has a few reasons to celebrate Meta’s decision last week to release Llama 3, a powerful open source large language model that anyone can download, run, and build on.
Pesenti used to be vice president of artificial intelligence at Meta and says he often pushed the company to consider releasing its technology for others to use and build on. But his main reason to rejoice is that his new startup will get access to an AI model that he says is very close in power to OpenAI’s industry-leading text generator GPT-4, but considerably cheaper to run and more open to outside scrutiny and modification.
“The release last Friday really feels like a game-changer,” Pesenti says. His new company, Sizzle, an AI tutor, currently uses GPT-4 and other AI models, both closed and open, to craft problem sets and curricula for students. His engineers are evaluating whether Llama 3 could replace OpenAI’s model in many cases.
Sizzle’s story might predict a larger shift in the influence balance within AI. OpenAI has made significant impacts with ChatGPT, leading the surge in AI investments and attracting over two million developers to its cloud APIs. However, with the emergence of competitive open source models, developers and entrepreneurs might consider not investing into access for the latest models from giants like OpenAI or Google and instead make use of Llama 3 or other potent open source models.
“It’s going to be a fascinating competition,” Pesenti mentions, referring to the rivalry between open models like Llama 3 and closed ones such as GPT-4 and Google’s Gemini.
Meta’s earlier model, Llama 2, already made remarkable influences, but the company claims to have enhanced the current version by feeding it with more amounts of high-quality training data. They developed new techniques to eliminate repeated or distorted content and choose the most suitable mixture of datasets.
Pesenti mentioned that running Llama 3 on a cloud platform like Fireworks.ai only costs a fraction, about a twentieth, of the price to access GPT-4 through an API. He further states that Llama 3 can be set up to respond to queries extraordinarily fast, a crucial factor for developers at companies like his that depend on using models from various providers. “It’s a balance between latency, cost, and accuracy,” he voiced.
Open models appear to be dropping at an impressive clip. A couple of weeks ago, I went inside startup Databricks to witness the final stages of an effort to build DBRX, a language model that was briefly the best open one around. That crown is now Llama 3’s. Ali Ghodsi, CEO of Databricks, also describes Llama 3 as “game-changing” and says the larger model “is approaching the quality of GPT 4—that levels the playing field between open and closed-source LLMs.”
Llama 3 also showcases the potential for making AI models smaller, so they can be run on less powerful hardware. Meta released two versions of its latest model, one with 70 billion parameters—a measure of the variables it uses to learn from training data—and another with 8 billion. The smaller model is compact enough to run on a laptop but is remarkably capable, at least in WIRED’s testing.
Two days before Meta’s release, Mistral, a French AI company founded by alumni of Pesenti’s team at Meta, open sourced Mixtral 8x22B. It has 141 billion parameters but uses only 39 billion of them at any one time, a design known as a mixture of experts. Thanks to this trick, the model is considerably more capable than some models that are much larger.
Meta isn’t the only tech giant releasing open source AI. This week Microsoft released Phi-3-mini and Apple released OpenELM, two tiny but capable free-to-use language models that can run on a smartphone.
Coming months will show whether Llama 3 and other open models really can displace premium AI models like GPT-4 for some developers. And even more powerful open source AI is coming. The company is working on a massive 400-billion-parameter version of Llama 3 that chief AI scientist Yann LeCun says should be one of the most capable in the world.
Of course all this openness is not purely altruistic. Meta CEO Mark Zuckerberg says opening up its AI models should ultimately benefit the company by lowering the cost of technologies it relies on, for example by spawning compatible tools and services that Meta can use for itself. He left unsaid that it may also be to Meta’s benefit to prevent OpenAI, Microsoft, or Google from dominating the field.
Amanda Hoover
Andy Greenberg
Mark Andrews
Andy Greenberg
