Language has long been considered a unique trait of humanity, a perspective supported by thinkers like Aristotle. However, the emergence of large language models (LLMs) such as ChatGPT has sparked questions about whether these AI systems can truly grasp the complexities of language in a way comparable to human reasoning.
Some linguists, including renowned figure Noam Chomsky, have argued that while LLMs can mimic fluent speech, they lack genuine reasoning about language. Chomsky and co-authors recently expressed skepticism in a New York Times article, suggesting that these models merely “marinate in big data” rather than develop a deep understanding of linguistic principles.
Contrary to this viewpoint, a recent study led by Gašper Beguš from UC Berkeley, in collaboration with linguists Maksymilian Dąbkowski and Ryan Rhodes, challenged these assumptions. They tested various LLMs on linguistic tasks, including the ability to generalize rules from a constructed language. While many models struggled, one model, OpenAI’s o1, demonstrated exceptional capabilities, akin to a graduate linguistics student. It was able to analyze and diagram sentences, navigate ambiguities, and utilize complex features like recursion, a hallmark of human language.
Tom McCoy, a Yale computational linguist, emphasized the criticality of this research as society increasingly depends on technology, necessitating a clearer understanding of its capabilities and limitations.
Conducting rigorous linguistic assessments posed challenges, primarily ensuring models didn’t simply recall memorized data from their vast training sets. To address this, the researchers crafted a four-part linguistic test focused on innovative sentence structures using tree diagrams and recursion.
Recursion, a key feature allowing phrases to be embedded within others, was a focal point. Strikingly, the o1 model not only parsed sentences featuring recursion but also provided deeper analyses by identifying additional layers within complex sentences, showcasing a higher-level metalinguistic ability.
David Mortensen, another computational linguist not involved in the study, commented on the significance of these findings, hinting at a potential invalidation of claims that LLMs only predict the next word without true understanding.
Moreover, the researchers examined ambiguity—defined in sentences that can be interpreted in multiple ways. The o1 model accurately generated various syntactic interpretations of sentences, outperforming earlier computational models.
The study also explored phonology, looking into how LLMs could infer rules from newly created languages without prior exposure. Impressively, o1 showcased an understanding of phonological processes, solidifying its advanced capabilities.
This research provokes critical questions about the future of language models. Could they eventually exceed human proficiency in language tasks through sheer computational scaling, or are the nuances of human language a product of our unique evolutionary history?
The results present a compelling narrative that these language models are capable of sophisticated linguistic analysis, yet they have yet to generate original insights about language itself. As improvements continue, while some experts remain cautious, the prospect of AI models achieving a better linguistic understanding than humans seems increasingly feasible.
In conclusion, the gap between AI language capabilities and human linguistic abilities may be closing. As Beguš notes, our uniqueness in this domain might not be as prominent as previously believed.
