PaddlePaddle 2.2.1: Experimental CUDA Graphs and Performance Optimizations

by

PaddlePaddle has released version 2.2.1, focusing on refining existing functionality, introducing experimental performance features, and stabilizing core APIs. The update addresses performance bottlenecks in training and inference while adding tools for graph learning and linear algebra.

Experimental CUDA Graphs Support

A notable addition in this release is the experimental support for NVIDIA's CUDA Graphs. The framework now exposes the paddle.device.cuda.graphs.CUDAGraph API. According to the vendor, this feature allows capturing GPU computation into a single graph to eliminate framework overhead during repeated calls, potentially improving runtime performance.

Important: The vendor explicitly states that this API is currently in an experimental stage and has not been stabilized.

New Linear Algebra and Graph Learning APIs

The release introduces several new APIs to support specialized workloads:

  • Linear Algebra: A new paddle.linalg.triangular_solve API has been added to solve linear systems with triangular coefficient matrices.
  • Graph Learning: The paddle.incubate.graph_send_recv API is available for graph learning tasks. It aims to reduce memory overhead during message passing by supporting four update modes: SUM, MEAN, MIN, and MAX.
  • ResNet Unit: A new paddle.incubate.operators.ResNetUnit API has been introduced to fuse convolution, batch normalization, and shortcut/bottleneck operations within ResNet networks.

Performance Optimizations

Several optimizations target training efficiency:

  • paddle.top_k: The implementation now selects between a CUB implementation and a hand-written kernel based on the value of k relative to input_width.
  • LarsMomentumOptimizer: Performance has been improved by integrating optimizer operator fusion with CUDA Cooperative Groups.
  • FusedTransformerEncoderLayer: Added support for pure fp16 and src_mask=None.

Bug Fixes and Stability Improvements

The release addresses numerous issues affecting core tensor operations and model training:

  • Tensor Operations: Fixed calculation errors in paddle.nn.ELU and paddle.nn.functional.elu (specifically when alpha is negative). Fixed issues with paddle.slice during reverse execution and paddle.scatter index out-of-range errors.
  • Gradient Support: Clarified that paddle.shape and paddle.arange do not support backward propagation; gradients are explicitly set to True for these operations.
  • Dropout and RNN: Fixed gradient calculation issues in paddle.nn.Dropout during evaluation mode and corrected backward calculation errors for multi-layer RNNs (LSTM, GRU, SimpleRNN) in CPU training.
  • Inference: Fixed memory optimization policies that were causing incorrect prediction results or crashes, and resolved scale calculation errors in Transformer MultiHead structures.

Inference Improvements

In the Paddle Inference module, the release includes fixes to memory and video memory optimization policies to prevent crashes. Additionally, redundant debug logs have been reduced when logging is disabled, and all inference operators have been registered for int8 quantization to resolve historical registration gaps.