Is AI Finally Getting Smarter? MIT Researchers Discover ‘Brain Regions’ in Large Models!

AIbase基地

Published inAI News · 3 min read · Oct 30, 2024

242

AI is starting to "grow a brain"?! The latest research from the Massachusetts Institute of Technology reveals that the internal structures of large language models (LLMs) are astonishingly similar to the human brain!

This study used sparse autoencoder technology to conduct an in-depth analysis of the activation space of LLMs, uncovering three levels of structural features that are truly remarkable:

Firstly, at the microscopic level, researchers discovered the existence of structures akin to "crystals." The faces of these "crystals" are composed of parallelograms or trapezoids, similar to familiar word analogies, such as "man:woman::king:queen."

Even more astonishing is that after removing some irrelevant interference factors (such as word length) through linear discriminant analysis, these "crystal" structures become clearer.

Secondly, at the mesoscopic level, researchers found that the activation space of LLMs exhibits modular structures similar to the functional zoning of the human brain.

For example, features related to mathematics and coding cluster together, forming a "lobe" akin to a functional lobe of the human brain. Through quantitative analysis with multiple indicators, researchers confirmed the spatial locality of these "lobes," indicating that co-occurring features are also more spatially concentrated, far exceeding the expectations of random distribution.

At the macroscopic level, researchers found that the overall structure of the LLM feature point cloud is not isotropic but exhibits a power-law eigenvalue distribution, most evident in the intermediate layers.

Researchers also quantified the clustering entropy at different levels, finding that the clustering entropy in the intermediate layers is lower, indicating more concentrated feature representation, while the early and late layers have higher clustering entropy, indicating more dispersed feature representation.

This research provides a new perspective for understanding the internal mechanisms of large language models and lays the foundation for developing more powerful and intelligent AI systems in the future.

ZhiYuan and Tencent Launch Long Text Understanding Benchmark Model LongBench v2

On December 19, 2024, at a press conference, ZhiYuan Research Institute and Tencent announced the launch of LongBench v2, a benchmark designed to evaluate the deep understanding and reasoning capabilities of large language models (LLMs) in real-world long text multi-task scenarios. The platform aims to advance long text models in understanding and reasoning, addressing the current challenges faced by large language models in practical applications.

AI Unicorn Cohere Deeply Collaborates with Palantir: Enterprise-level AI Applications Emerge

As one of the most watched AI startups outside of OpenAI and Anthropic, Cohere was valued at $5.5 billion in July. One of the company's co-founders is an author of the paper 'Attention Is All You Need,' which is considered critical to sparking the revolution of large language models (LLMs). Headquartered in Toronto and San Francisco, Cohere focuses on providing AI solutions for enterprise clients, rather than pursuing a similar path as other companies.

Meta Launches 'Large Concept Models' (LCMs)! Breaking Through LLM Limitations and Leading a New Direction in AI Language Understanding

Large Language Models (LLMs) have made significant progress in the field of Natural Language Processing (NLP), shining in applications such as text generation, summarization, and question answering. However, the reliance of LLMs on token-level processing (predicting one word at a time) presents some challenges. This method contrasts with human communication, which typically operates at higher levels of abstraction, such as sentences or ideas. Token-level modeling also struggles in tasks that require understanding long contexts and may produce inconsistent outputs.

Meta AI Launches Conceptual Model: A Breakthrough Beyond Traditional Language Models

In recent years, large language models (LLMs) have made significant progress in the field of natural language processing (NLP), widely applicable in scenarios such as text generation, summarization, and question answering. However, these models rely on a token-level processing method that predicts word by word, which struggles with contextual understanding and often leads to inconsistent outputs. Moreover, when scaling LLMs to multilingual and multimodal applications, the computational costs and data requirements tend to be relatively high. To address these issues, Meta AI has proposed a novel approach.