How does the inter-DC latency magnify the bubble time in pipeline parallelism (PP)?
In pipeline parallelism (PP), the bubble time refers to the idle periods in the training pipeline caused by the sequential nature of stage execution, where some devices wait for others to complete computation or communication. Inter-DC latency exacerbates this bubble time nonlinearly because the higher latency in metropolitan area networks (MANs) compared to intra-DC links increases the communication time between stages. Specifically, in the per-iteration time formula Titer=(M+S−1)×maxsTstages, each stage time Tstagesdepends on the maximum of computation time Tcompsand communication time Tcomms−1,s. When inter-DC latency is high, Tcommdominates due to factors like propagation delays, leading to elongated stage times. This magnification occurs because the bubble—idle time proportional to S−1—scales with the slowest stage, and latency-driven communication bottlenecks make this worst-case stage longer, reducing overall throughput. As illustrated in Figure 1, the inter-DC delay stretches the communication phases, creating larger gaps between micro-batches and amplifying idle bubbles. This effect is particularly pronounced in cross-DC environments, where latency can be milliseconds compared to sub-millisecond intra-DC links. Consequently, strategies like CDynP incorporate latency awareness into optimization to mitigate this by balancing stage loads and overlapping computation with communication, but the fundamental issue stems from the nonlinear impact of latency on pipeline efficiency.
