PaddlePaddle 2.4.0: New Dynamic Graph, Sparse Computing, and Distributed Training Upgrades

by

PaddlePaddle 2.4.0: New Dynamic Graph, Sparse Computing, and Distributed Training Upgrades

PaddlePaddle has released version 2.4.0, marking a substantial architectural shift with the official rollout of a new dynamic graph framework. The release also introduces specialized modules for sparse computing and graph learning, alongside significant enhancements to distributed training capabilities.

New Dynamic Graph Architecture

The most prominent change is the official activation of a new dynamic graph framework. According to the release notes, this overhaul has significantly improved scheduling performance, with over 90% of APIs seeing a boost of more than 50%. The framework is described as having a clearer functional architecture and lower coupling, which the vendor claims will significantly enhance secondary development capabilities and the user experience for extension modules like Hooks and PyLayer.

Sparse Computing and GNN Engine

To address high-sparse scenarios, PaddlePaddle has added 55 new APIs under the paddle.sparse.* namespace. These APIs support mainstream sparse computing scenarios, including 3D point cloud target detection and Sparse Transformers. Vendor claims indicate that in high-sparse scenarios, these APIs offer a speedup of 105.75% compared to using dense tensors and 4.01% to 58.55% faster than similar products.

Additionally, a new large-scale graph neural network (GNN) GPU training engine has been introduced. By utilizing heterogeneous hierarchical storage (SSD, memory, and GPU memory), this engine aims to break through memory bottlenecks for super-large-scale graphs. The vendor claims it delivers a training speed more than 10 times faster than traditional distributed CPU solutions under the same cost.

Domain Expansion and Distributed Training

The framework now includes new modules for specific domains:

  • Audio: A new paddle.audio module provides feature extraction APIs (MFCC, Spectrogram) and datasets, with GPU implementations offering over 15x performance improvements over CPU.
  • Graph Learning: The paddle.geometric module adds message passing and high-performance graph sampling APIs, with reported speedups of 32x for GraphSage sampling and 12x for model training.

Distributed training has also been enhanced with the introduction of an adaptive parallel interface system. This allows users to easily obtain automatic distributed training capabilities based on single-machine networking, supporting data, model, pipeline, and hybrid parallelism. New features include tensor parallelism support and auto-tuning for Sharding stages.

Deployment and Hardware Support

Deployment capabilities have been unified. The training and inference pre-compiled packages now default to supporting TensorRT, reducing switching costs. The Inference library now supports native Automatic Mixed Precision (AMP) and has been optimized for large models, including INT8 implementations for fused multi-transformer operators.

Hardware support has been expanded to include Hygon DCU, Kunlunxin 2nd generation chips, Cambricon MLU, Graphcore IPU, and Intel CPUs with oneDNN optimizations.

Breaking Changes and Deprecations

Users upgrading to 2.4.0 should be aware of several breaking changes:

  • Environment: Pre-compiled packages for CUDA 10.1 have been removed. The framework now defaults to CUDA 11.7 support and supports Ubuntu 22.04.
  • Python: Support for Python 3.6 will be deprecated in version 2.5.
  • API Behavior: In dynamic graphs, gradients for non-leaf nodes are no longer retained by default. Users must explicitly call Tensor.retain_grads() if they need to preserve specific gradients.
  • PyLayer: Inputs to paddle.autograd.PyLayer can no longer be tuples; they must be passed as a list of Tensors.