If there is a significant disparity in computational power between different data centers, will it lead to a situation of load imbalance?
Excellent question. CDynP is specifically designed to handle computational heterogeneity between data centers and actively works to prevent severe load imbalance. The framework incorporates two key mechanisms for this purpose.
First, during the initial heuristic pre-partitioning phase, the model layers are coarsely allocated to each DC proportionally to its computational power (measured by available GPUs), establishing a balanced starting point. Second, and most importantly, the core CDynP algorithms optimization objective is to minimize the maximum stage time (Titer=(M+S−1)×maxsTstages). This objective function inherently penalizes any stage that becomes a bottleneck. During the chain-based dynamic programming process, each DC makes local partitioning decisions that consider the computational load of its assigned layers and the cumulative state from upstream DCs. This allows CDynP to dynamically adjust the layer boundaries between DCs, assigning more layers to powerful DCs and fewer to weaker ones, thereby striving to equalize the stage times across the pipeline and avoid load imbalance. The results in Figure 4, showing minimal performance degradation when scaling to multiple heterogeneous DCs, empirically validate its effectiveness.
