AI is starting to "grow a brain"?! The latest research from the Massachusetts Institute of Technology reveals that the internal structures of large language models (LLMs) are astonishingly similar to the human brain!
This study used sparse autoencoder technology to conduct an in-depth analysis of the activation space of LLMs, uncovering three levels of structural features that are truly remarkable:
Firstly, at the microscopic level, researchers discovered the existence of structures akin to "crystals." The faces of these "crystals" are composed of parallelograms or trapezoids, similar to familiar word analogies, such as "man:woman::king:queen."
Even more astonishing is that after removing some irrelevant interference factors (such as word length) through linear discriminant analysis, these "crystal" structures become clearer.
Secondly, at the mesoscopic level, researchers found that the activation space of LLMs exhibits modular structures similar to the functional zoning of the human brain.
For example, features related to mathematics and coding cluster together, forming a "lobe" akin to a functional lobe of the human brain. Through quantitative analysis with multiple indicators, researchers confirmed the spatial locality of these "lobes," indicating that co-occurring features are also more spatially concentrated, far exceeding the expectations of random distribution.
At the macroscopic level, researchers found that the overall structure of the LLM feature point cloud is not isotropic but exhibits a power-law eigenvalue distribution, most evident in the intermediate layers.
Researchers also quantified the clustering entropy at different levels, finding that the clustering entropy in the intermediate layers is lower, indicating more concentrated feature representation, while the early and late layers have higher clustering entropy, indicating more dispersed feature representation.
This research provides a new perspective for understanding the internal mechanisms of large language models and lays the foundation for developing more powerful and intelligent AI systems in the future.