PaddlePaddle, the open-source deep learning framework, has released version 3.1.0, marking a significant step toward scalable and efficient large model training. The release focuses on refining the auto-parallel architecture, introducing FP8 low-precision training, and lowering the barrier for heterogeneous hardware support.
Auto-Parallel Architecture
The core focus of version 3.1.0 is the auto-parallel architecture. PaddlePaddle has optimized the core mechanism to improve usability and dynamic graph performance. Key enhancements include:
- Splitting Rules: New splitting derivation rules have been added for multiple operators. The framework now supports scenarios where a single dimension of a distributed tensor is split across multiple mesh dimensions.
- Strategies: Support for dynamic graph parallel strategies such as Pipeline Parallelism (PP), Context Parallelism (CP), Sequence Parallelism (SEP), and Tensor Parallelism for Convolution (TP-CONV).
- Performance: Vendor claims indicate that performance on models like Llama2, Qwen, and Baichuan is essentially on par with manual parallelism.
FP8 Low-Precision Training
To accelerate large model training, PaddlePaddle 3.1.0 introduces blockwise FP8 GEMM operators. This feature supports low-precision training with accuracy comparable to BF16. According to the vendor, this results in a 10-20% speedup for large model training.
Heterogeneous Hardware Adaptation
A significant development in this release is the support for heterogeneous multi-core chips. PaddlePaddle provides a “CUDA-like” operator reuse mechanism. Users can register kernels to reuse existing CUDA kernels, significantly reducing the cost of adapting to hardware that is similar to CUDA. This allows for a plugin-style integration of non-NVIDIA hardware backends.
Framework Stability and API Updates
Beyond performance, the release addresses stability and usability:
- Stability Fixes: The framework fixes calculation errors for operators in 0-Size and large dimension scenarios.
- New APIs: A new
paddle.randn_likeAPI has been added. There are also fixes fortensordotandatleastAPIs. - Deprecation: PaddlePaddle 3.1.0 stops maintenance for Python 3.8 environments.