Against the backdrop of the global generative AI wave in 2022, YuanShi Intelligence (RWKV) completed an angel round financing of tens of millions of RMB in December 2023, with investment from Tianji Capital. Following this financing, the company's valuation doubled, and the funds will be used for team expansion, new architecture research and development, and product commercialization.
The emergence of RWKV represents a strong challenge to the traditional Transformer architecture. With the development of large language models (LLMs), although the parameter scale of models is increasingly large, issues such as hallucinations and accuracy have remained difficult to resolve. Therefore, the founding team of RWKV decided to explore a brand-new architecture in hopes of achieving higher efficiency and flexibility.
The design philosophy of RWKV is entirely different from that of Transformers. Co-founder Luo Xuan stated that traditional Transformer models need to "read" the previous context every time they generate a token, while RWKV does not need to record the state of each token, significantly reducing the computational load. RWKV achieves breakthroughs in efficiency and language modeling capabilities by combining the advantages of RNNs (Recurrent Neural Networks).
The advantage of this innovative architecture is that RWKV can process information within a limited state space. Through reinforcement learning methods, the model can automatically determine when to revisit previous context, thereby enhancing its memory capabilities. Compared to traditional models, RWKV has shown superior performance in multiple benchmark tests, proving its improvement in language learning efficiency.
Currently, RWKV has completed model training from 0.1B to 14B parameters and has released a 32B preview model in overseas communities. In the future, YuanShi Intelligence plans to launch the RWKV-7 with 70B parameters or more by 2025 and explore new reasoning frameworks and chips to further enhance model performance.
In terms of business, RWKV not only offers open-source projects but is also actively engaged in commercial layouts, involving AI music generation and collaborations with enterprises. It has already partnered with several companies, including the State Grid. With the advancement of technology and the push for commercialization, RWKV aims to become the "Android and Linux" of the large model field.
