Some significant AI companies believe that keeping AI models locked away in labs minimizes the potential for chaos and mischief. However, Nathan Lambert and Tom Zick, two scientists in the industry, advocate for a different approach. They have founded a nonprofit, Trillium Labs, aimed at conducting AI research transparently, especially in areas known for their risks, like recursive self-improvement (RSI) and agent behavior.
Lambert argues that the current secrecy surrounding frontier AI research limits scrutiny and innovation from the broader community. He contends that allowing external experts to observe and analyze model configurations can help mitigate risks associated with AI. “The current closed trajectory of frontier AI development is taking us a step backwards,” he says, emphasizing the importance of the scientific method in fostering advancements.
Presently, powerful models from companies like OpenAI and Anthropic are primarily accessible through apps or APIs, often lacking transparency regarding their construction and behavior. In contrast, certain international competitors, notably from China, allow users to download capable models for personal use, presenting a stark contrast in methodologies and raising questions about the best practices for AI development.
As debates continue about the appropriate access to AI models, Lambert and Zick draw attention to the potential dangers posed by models that can identify software vulnerabilities and hack systems autonomously. While some advocate limited access to maintain control, Lambert and Zick argue that shared understanding benefits everyone.
Lambert’s background includes work at Ai2, which has adopted an open approach to AI research, while Zick has expertise in responsible AI guidelines. Recognizing a gap between industry research and academic work during their graduate studies at UC Berkeley, they launched Trillium Labs focusing on post-training and the implications of RSI, which could lead to models developing independently of human oversight—a worrying prospect in the AI community.
With initial funding from Schmidt Sciences and Halcyon Futures, the lab has plans to raise $40 to $100 million overall, earmarking $30 million for training in the coming 18 months. Their research aims to publish findings on reinforcement learning methods, which can enhance model capabilities but also induce unpredictable behavior. As Zick notes, careful experimentation could reveal valuable insights when scrutinized by outside researchers.
Ultimately, Lambert and Zick aim for Trillium Labs to contribute a necessary perspective to the ongoing discourse on AI development, advocating for greater transparency and scientific rigor. “We believe that the scientific method and careful measurement of recent events is the best way to understand new behaviors of AI models,” Lambert asserts.
