Recently, a paper by DeepMind on the application of ultra-large-scale Transformers in the field of chess has sparked extensive discussions within the AI community. Researchers utilized a novel dataset called ChessBench to train a Transformer model with up to 270 million parameters, exploring its capabilities in complex planning problems like chess.
The ChessBench dataset includes 10 million human chess games collected from the Lichess platform, annotated with the top chess engine Stockfish16, providing up to 15 billion data points, including win rates, optimal moves, and value assessments of all legal moves for each game state.
Researchers employed supervised learning methods to train the Transformer model to predict the value of each legal move in a given game state. Experimental results show that even without using any explicit search algorithms, the largest models can make quite accurate predictions in new game states, demonstrating strong generalization capabilities.
Surprisingly, the model achieved an Elo rating of 2895 when playing blitz chess against human players on the Lichess platform, reaching the level of an international chess master.
The researchers also compared the model with chess engines like Leela Chess Zero and AlphaZero, which are based on reinforcement learning and self-play training. The results indicate that while supervised learning can distill an approximation of the Stockfish search algorithm into the Transformer model, achieving perfect distillation remains challenging.
This study shows that ultra-large-scale Transformer models have significant potential in solving complex planning problems and provides new insights for future AI algorithm development. The release of the ChessBench dataset will also offer researchers a new benchmark platform to explore AI planning capabilities.