Researcher Flood Sung from the Dark Side of the Moon recently published a lengthy article detailing the development approach of the k1.5 model for the first time, and reflecting deeply on the technical insights brought by the OpenAI o1 model.
According to Flood Sung, the importance of Long-CoT (Long Chain of Thought) was validated over a year ago by Tim Zhou Xinyu, co-founder of the Dark Side of the Moon. By training small models on multi-digit calculations and transforming the fine-grained calculation processes into Long Chain of Thought data for SFT (Supervised Fine-Tuning), significant results can be achieved.
However, due to cost considerations, the Dark Side of the Moon previously focused on optimizing Long Context (long text input). Flood Sung explained that Long Context mainly deals with the input side, utilizing Prefill and Mooncake technologies to effectively control costs and speed. In contrast, Long-CoT focuses on the output side, requiring higher costs and longer processing times.
But the release of OpenAI o1 prompted the team to rethink the priority of their technological direction. "Performance is the most important thing," Flood Sung stated, "Costs and speed will continue to be optimized with technological advancements; the key is to first achieve breakthrough performance." Based on this understanding, the Dark Side of the Moon has begun to comprehensively advance Long-CoT research, aiming to enable the model to achieve a level of free thought closer to that of humans.
The release of this technical decryption article marks the beginning of a systematic alignment with the o1 model by the Dark Side of the Moon, initiating substantial research in related fields.
In-depth article on decrypting the o1 cracking process:https://mp.weixin.qq.com/s/sJmT-tM3A-mglZ1d4OI80A