We’ve all been astonished at how chatbots seem to understand the world. But what if they were truly connected to the real world? What if the dataset behind the chat interface was physical reality itself, captured in real time by interpreting the input of billions of sensors sprinkled around the globe? That’s the idea behind Archetype AI, an ambitious startup launching today. As cofounder and CEO Ivan Poupyrev puts it, “Think of ChatGPT, but for physical reality.”
Archetype’s foundational model is called Newton. Yes, they know about Apple’s long-lamented handheld device killed by Steve Jobs in 1997, and no, they don’t seem to care. The new Newton is designed to process data from sensors of all kinds and answer questions or provide charts or even computer code to report what’s happening in the world. For Poupyrev himself, Archetype is fulfillment of a long-held belief that the digital world can provide a means of deep engagement with the physical one. Though I didn’t know it at the time, I had my own hand in this: Soviet-born Poupyrev’s fascination for manipulating the world through tech was triggered in part by my book Hackers, which had been slipped to his father during a visit to China. “The idea that things could be hacked and their nature could be changed by inventing new technology inspired me for the rest of my life,” he tweeted about the book in 2020. He also honed his English by reading and rereading the book. You’re welcome.
Poupyrev’s journey after he left the Soviet Union and became a computer scientist included stints at Sony, Disney, and, until last March, Google’s ATAP division. That’s where he led a team working on Soli, a project that built tiny radar devices into wearable gadgets to allow them to respond to a person’s gestures and movements. The demos were impressive, but that approach had limits. “Analyzing sensors was really hard. You had read them by hand,” he says. When LLMs appeared, Poupyrev and colleagues realized that with modifications they could make sensor data more powerful by providing a way for humans to easily explore and monitor data collected across vast swaths of time and space. Instead of a large language model it would be a large behavior model. “We were excited to see how they can work with real-time data from the physical world,” he says. They were particularly excited to do it outside of Google, free of the constraints of working within a giant organization. In March last year, Poupyrev and eventually four others left to start Archetype, now funded by a $13 million seed round of investment funding.
Nick Gillian, Brandon Barbello, Ivan Poupyrev, Jaime Lien, and Leonardo Giusti left Google’s ATAP R&D group last year to cofound startup Archetype AI.
David Kushner
Andy Greenberg
Andy Greenberg
Jenna Scatena
“The physical world is where we have most of our problems, because it is so complex and fast moving that things are beyond our perception to fully understand,” says Brandon Barbello, a cofounder who is also Archetype’s COO. “We put sensors in all kinds of things to help us, but sensor data is too difficult to interpret. There’s a potential to use AI to understand that sensor data—then we can finally understand these problems and solve them.”
When Jenna visited Archetype’s founding team of five, currently working out of a cramped room in the Palo Alto office of its lead funder, venture capital firm Venrock, they showed her some illuminating demos that, they assured her, only hinted of Newton’s vast potential impact. They placed a motion sensor inside a box and prompted Newton to imagine that the container was an Amazon package with fragile cargo that should be carefully monitored. When the box was dropped, the display running the model broke the news that the package might be damaged. One can easily imagine a shipment of vaccines with motion, temperature, and GPS sensors monitored to verify whether it will arrive with full effectiveness.
One key use case is using Newton “to talk to a house or chat with a factory,” says Barbello. Instead of needing a complex dashboard or custom-built software to make sense of the data from a home or industrial facility wired with sensors, you can have Newton tell you what’s happening in plain language, ChatGPT style. “You’re no longer looking sensor by sensor, device by device, but you actually have a real-time mirror of the whole factory,” Barbello says.
Archetype’s AI model Newton processes data from various sensors, fuses these data, and converts it into simple language descriptions that depict operations in the physical world.
Notably, Amazon, equipped with some of the world’s most advanced digital logistics operations, is among Archetype’s supporters through its Industrial Innovation Fund. The belief is that this technology has the propensity to further enhance goods flow in our fulfillment centers and expedite delivery for customers, says Franziska Bossart, who manages the fund. Concurrently, Archetype is investigating possibilities in the health care sector. Stefano Bini, from the Department of Orthopaedic Surgery at UC San Francisco, has been exploring sensors that can track the progress after a knee replacement surgery. Newton might facilitate his pursuit for a singular metric extracted possibly from multiple sensors which can quantifiably measure the influence of any health care intervention.
Volkswagen, who is another early supporter of Archetype, is conducting preliminary tests of Archetype’s model. Interestingly, these do not involve autonomous driving, even though Archetype is eager for its technology to be used for that purpose. One of the experiments by Volkswagen includes a situation wherein a car’s sensors can study motions, possibly together with a sensor on the driver, to decipher when the car owner is returning from a store and requires assistance. If human intentions can be understood in such scenarios, it can automatically open the trunk and possibly store the shopping in specially maintained sections says Brian Lathrop, senior principal scientist at Volkswagen’s Silicon Valley innovation center. Lathrop is of the opinion that tasks like these are just the starting point of what becomes feasible when AI has the capabilities to convert large quantities of sensor data into insights that are human-centered. The firm’s interests envelop the safety of those inside and outside the vehicles.
The negative implications of a trillion-sensor monitoring system that provides immediate responses to queries about actions taking place at any location within its dense network is not hard to conceive. On bringing up to Poupyrev and Barbello that this could potentially be slightly dystopian, they assure me they have considered this perspective. They opine that radar and other sensor data are less controversial compared to cameras. Even though Archetype can process camera data as one of its sensor inputs, the clients they are dealing with are concentrating on addressing their specific issues with a broad range of sensors without breaching privacy rights. Lathrop from Volkswagen expresses a similar view. On using Archetype software, behavior is detected and not identity. It is entirely possible to detect potentially threatening behavior without identifying the person involved. However, there is evidence suggesting that a person’s unique walking pattern, which can be detected by a high-quality radar, is as unique as a fingerprint.
David Kushner
Andy Greenberg
Andy Greenberg
Jenna Scatena
Archetype’s vision isn’t unique. Robotics companies in particular are looking at using generative AI and sensors; one company is literally called Physical Intelligence. Poupyrev acknowledges the competition but says that Archetype is distinguished by its breadth. “We address a much more generic market where pretty much anybody who has sensor data, irrespective of use case, should be able to make it useful and functional for them,” he says. Barbello adds that Archetype’s technology is more powerful because of its breadth. “Our models are able to learn from more examples of how the physical world works than those foundations served by more narrow physical models,” he says. “We think our approach is the best way to tackle this problem of understanding the entire physical world.” Got it. But after that, maybe we can tackle the new problems that come when Archetype’s mission is accomplished.
Until a year ago, Archetype’s team all worked together at ATAP, an R&D lab inside Google, embedded within the Motorola division acquired in 2011. Its DNA was a strange mixture of mobile-directed research, the values of the US Department of Defense’s Advanced Research Project Agency, and Google’s own fixation on making bets on exotic breakthroughs. I talked about ATAP in 2013 when writing about an underappreciated, and sadly discontinued, animation project called Spotlight Stories, and its first sensor-driven mini-movie called Windy Day.
Windy Day came from Motorola’s in-house moonshot division, Advanced Technology and Products (ATAP). This research group, begun in May 2012 (the same month that Google’s $12.5 billion purchase of Motorola Mobile became official), shares the high ambitions of its parent company’s own long-term research group, Google X. But ATAP has a different model: DARPA, the US Defense Department’s Advanced Research Projects Agency. DARPA’s accomplishments include game-changers like lasers, stealth bombers, and that little thing we call the internet.
David Kushner
Andy Greenberg
Andy Greenberg
Jenna Scatena
Over the years, DARPA has developed a well-honed process to develop its breakthroughs, a regimen Motorola Mobility’s new CEO Dennis Woodside thought was worth emulating. So he hired DARPA’s charismatic director, Regina Dugan, as a senior vice president. Making the jump with Dugan was her deputy, Ken Gabriel.
Motorola immediately initiated a mini-DARPA internally. Much like DARPA, ATAP hires researchers for two-year periods, urging them to embark on a project as soon as new technologies make it feasible to make a significant advancement. Project leaders have the freedom to contract external help – recruiting some of the world’s most brilliant minds for innovative research with explicit aims. Some of these objectives are exactly what you’d envisage from a corporate R&D division. Just as DARPA takes on certain projects with clear military applications, ATAP targets obvious things such as accelerating graphics, developing thinner materials, or enhancing the longevity of your phone’s battery. If the outcome isn’t at least five to ten times better, a project isn’t considered. But ATAP’s work isn’t restrained to catering to popular demand. “It’s also our job at ATAP to do things that they don’t know to ask for,” comments Ken Gabriel.
Joe inquires, “How does battery-powered garden equipment like mowers, snow blowers, and leaf blowers stack up against their gas-powered equivalents?”
That’s a great question, Joe. Are you asking in relation to environmental impact? Because we all know the answer – it’s electric all the way. When it comes to deciding which equipment does the best job, after comprehensive research (including Google, ChatGPT, and consulting BJ, the person who actually clears my driveway), there seems to be a unanimous agreement. Battery-powered tools demand less maintenance, provide ease of use, and create significantly less noise. However, their ability to handle heavier tasks is limited, partially because the charge doesn’t last very long and partially due to their lower power. For someone residing in Buffalo, a battery-powered snow blower might not suffice. If you own a vast property like Saltburn, using an electric mower might require an exorbitant amount of recharging. Also, BJ told me an interesting incident recently where the local fire department had to extinguish a massive fire, caused by a leaf blower’s battery charger.
David Kushner
Please also note that the links, images, iframes, form, and style attributes are not provided in the original content, so they’re not included in the rewritten content.
Andy Greenberg
Andy Greenberg
Jenna Scatena
But, people, how long are we going on this cursed path of burning energy to tend our gardens? Google once hired a bunch of goats to eat its grass! Leaves can get gathered by good ol’ rakes. Or let them mulch in place! And the best way to clear a driveway piled with snow is to hire some neighborhood ragamuffins with snow shovels, pay them some cash, and give them a belt of hot chocolate. Mother Nature will honor you!
You can submit questions to mail@wired.com. Write ASK LEVY in the subject line.
Taiwan’s earthquake this week hit a magnitude of 7.4. Naturally, people are asking: Will this slow down Nvidia?
While our LLMs talk to sensors, the chips that run them might one day communicate via light.
Two years ago a lone hacker took down North Korea’s internet. Now he’s revealing his identity. Brave soul.
David Kushner
Andy Greenberg
Andy Greenberg
Jenna Scatena
Turkey’s love affair with crypto was symbolized by Faruk Özer, who is now beginning a 11,196-year prison term for fraud. What will Bitcoin be worth then?
For some female scientists, the cold was not the most treacherous part of Antarctica.
Don’t miss future subscriber-only editions of this column. Subscribe to WIRED (50% off for Plaintext readers) today.
