The release of PaddlePaddle Framework 3.0 features a core upgrade to unified automatic parallelism technology, aimed at simplifying the distributed training process for large models and improving development efficiency. The new version supports mixed parallelism techniques from four-dimensional to five-dimensional, employing various parallel methods such as data parallelism, tensor model parallelism, pipeline parallelism, and grouped parameter slicing parallelism, effectively enhancing the training efficiency of large models. The automatic parallelism technology uses tensor slicing syntax to automatically infer distributed slicing states and add communication operators, reducing the difficulty of development. The principles of automatic parallelism include distributed tensor representation and slicing inference.

Scientists successfully trained a model at ChatGPT level using only 8% of the computing power. The research team utilized innovative technology on the Frontier supercomputer, training a trillion-parameter language model with just thousands of AMD GPUs. They achieved 100% weak scaling efficiency for 175 billion-parameter and trillion-parameter models while utilizing only 8% of computing capacity. The results indicate that training large language models with trillion parameters remains challenging.