Google researchers recently introduced a new model called TransNAR, which combines Transformer with Neural Algorithm Reasoning (NAR) to achieve outstanding performance in algorithmic tasks.
Traditional Transformers have shortcomings in algorithmic reasoning, while NAR excels in handling structured data and possesses strong generalization capabilities. Through cross-attention mechanisms, TransNAR deeply integrates Transformer and NAR, allowing the model to handle both textual representations of algorithmic problems and graph representations, thereby achieving superior algorithmic reasoning capabilities.
The training strategy of TransNAR is also quite unique, employing a multi-level training approach. In the pre-training phase, NAR is trained independently, learning intrinsic logic and computational steps through executing various algorithmic tasks. During the fine-tuning phase, TransNAR receives dual inputs of textual descriptions and graph representations, utilizing the node embedding information provided by the pre-trained NAR to adjust its own token embeddings through cross-attention mechanisms.
This process enables TransNAR to outperform the baseline Transformer model significantly in algorithmic tasks, especially in its generalization capabilities outside of the distribution, with TransNAR showing over 20% improvement in optimization.
Key Points:
⭐ Google introduces the TransNAR model, combining Transformer with NAR to enhance algorithmic reasoning capabilities
⭐ TransNAR uses cross-attention mechanisms to deeply integrate Transformer and NAR, excelling in both textual and graph representations
⭐ The multi-level training strategy makes TransNAR significantly better than the baseline Transformer in algorithmic tasks, particularly shining in its generalization abilities