ICACT20260329 Question.1
Questioner: xueyy@dsp.ac.cn 2026-02-04 오후 1:39:50 |
ICACT20260329 Answer.1
Answer by Auhor yangyn@dsp.ac.cn 2026-02-04 오후 1:39:50
Chrome  Click!! |
| In this paper, privacy is preserved by exchanging compressed state summaries (SS) between data centers instead of full resource and topology information.
Could this compression lead to the loss of critical information that may affect the global optimality of the generated parallel strategy?
Have the authors analyzed the trade-off between privacy strength and optimization performance? |
The compression of state summaries (SS) is carefully designed to conceal detailed resource and topology information while retaining the minimal maximum stage execution time necessary for global optimization. Our theoretical analysis in Section IV.B.3 proves that CDynP achieves optimality equivalent to the exhaustive enumeration method under privacy constraints, as it leverages the optimal substructure of dynamic programming and ensures no critical information loss through chain-based interactions. Specifically, the SS compression captures the cumulative state from upstream data centers, allowing each DC to make locally optimal decisions without exposing internal data.
We have explicitly analyzed the trade-off between privacy strength and optimization efficiency. The pre-defined overlapping areas in the heuristic pre-partitioning phase limit the search space to balance computational cost and solution quality, ensuring that privacy preservation does not compromise global optimality in practice. Experimental results in Section V validate that CDynP maintains leading performance across various scenarios, underscoring its robustness. Thus, the compression mechanism is both effective and privacy-compliant. |
ICACT20260329 Question.10
Questioner: zhaoyb@dsp.ac.cn 2026-02-09 오전 11:57:18 |
ICACT20260329 Answer.10
Answer by Auhor yangyn@dsp.ac.cn 2026-02-09 오전 11:57:18
Chrome Click!! |
| If there is a significant disparity in computational power between different data centers, will it lead to a situation of load imbalance? |
Excellent question. CDynP is specifically designed to handle computational heterogeneity between data centers and actively works to prevent severe load imbalance. The framework incorporates two key mechanisms for this purpose.
First, during the initial heuristic pre-partitioning phase, the model layers are coarsely allocated to each DC proportionally to its computational power (measured by available GPUs), establishing a balanced starting point. Second, and most importantly, the core CDynP algorithms optimization objective is to minimize the maximum stage time (Titer=(M+S−1)×maxsTstages). This objective function inherently penalizes any stage that becomes a bottleneck. During the chain-based dynamic programming process, each DC makes local partitioning decisions that consider the computational load of its assigned layers and the cumulative state from upstream DCs. This allows CDynP to dynamically adjust the layer boundaries between DCs, assigning more layers to powerful DCs and fewer to weaker ones, thereby striving to equalize the stage times across the pipeline and avoid load imbalance. The results in Figure 4, showing minimal performance degradation when scaling to multiple heterogeneous DCs, empirically validate its effectiveness.
|
ICACT20260329 Question.11
Questioner: shil@dsp.ac.cn 2026-02-09 오전 11:57:59 |
ICACT20260329 Answer.11
Answer by Auhor yangyn@dsp.ac.cn 2026-02-09 오전 11:57:59
Chrome Click!! |
| How does CDynP handle privacy concerns while coordinating distributed training across multiple data centers, and what specific information is shared between data centers in the compressed state summary (SS)? |
CDynP addresses privacy concerns through a decentralized, chain-based coordination mechanism that avoids exposing any internal data center information. The specific information shared between DCs is the compressed state summary (SS), which is mathematically defined as SSk−1[s][l]=min1≤g≤Gk−1dpk−1[s][l][g].
This SS contains only the minimal maximum stage execution timefor partitioning a given number of layers (l) into a given number of stages (s). It is a highly aggregated cost metric that deliberately excludes all sensitive internal details, such as:
The specific GPU allocation (g) that achieved that minimal time.
Intra-DC topology or bandwidth (Bkintra).
Detailed workload distribution.
During the "Forward CDynP" phase, this SS is passed sequentially from one DC to the next. Each DC uses the received SS to initialize its local dynamic programming process, makes a local optimization decision for its segment of the model, and then compresses its results into a new SS for the next hop. This process ensures that no DC ever has visibility into anothers internal resources or configuration, achieving privacy-preserving coordination. The theoretical analysis in the paper proves that this method can achieve a solution quality equivalent to a full-information approach under the privacy constraints.
|
ICACT20260329 Question.12
Questioner: shil@dsp.ac.cn 2026-02-09 오전 11:58:46 |
ICACT20260329 Answer.12
Answer by Auhor yangyn@dsp.ac.cn 2026-02-09 오전 11:58:46
Chrome Click!! |
| What is the role of the chain-based interaction mechanism in CDynP, and how does it help in generating a near-optimal global parallel strategy without centralized control? |
The chain-based interaction mechanism is the core of CDynPs decentralized optimization. It enables the assembly of a near-optimal global strategy by having each data center (DC) in a logical chain perform a local, privacy-preserving optimization and pass only a compressed summary to the next hop.
The process, illustrated in Figure 3, works as follows: The first DC in the chain performs dynamic programming (DynP) on its expanded layer range and compresses the results into a State Summary (SS) containing only minimal stage times, hiding internal GPU allocations.
The next DC receives this SS, uses it to initialize its local DynP process (accounting for the upstream pipeline cost), and then performs its own optimization. This "inherit-plan-compress-forward" cycle repeats along the chain. Finally, the tail node identifies the best global strategy, and a backtracking process propagates the optimal cut-points upstream. This mechanism eliminates the need for a central controller with full system knowledge. Each DCs local decision, guided by the inherited SS, contributes to a globally efficient solution, as proven to be equivalent to a full-information enumeration under the privacy constraints. |
ICACT20260329 Question.2
Questioner: xueyy@dsp.ac.cn 2026-02-04 오후 2:02:18 |
ICACT20260329 Answer.2
Answer by Auhor yangyn@dsp.ac.cn 2026-02-04 오후 2:02:18
Chrome Click!! |
| For the single-DC case, the time complexity of the DynP algorithm is
𝑂
(
𝑆
𝐿
2
𝐺
2
)
O(SL
2
G
2
).
As the model depth and the number of GPUs continue to grow, do the authors anticipate scalability challenges in practical deployments of CDynP?
Are there any plans to incorporate pruning techniques, approximate dynamic programming, or heuristic methods to further reduce the computational cost? |
Yes, as model depth (L) and GPU count (G) grow, the O(S L² G²) complexity of the single-DC DynP algorithm could pose challenges for extremely large-scale deployments. However, CDynP mitigates this through its chain-based decentralized approach, which partitions the problem across K data centers, effectively reducing the search space to O(S (L/K)² G²) per node and avoiding the combinatorial explosion of a global optimization. Our evaluations in Section V.B.3 demonstrate that CDynP maintains practical search times—e.g., 39.27 seconds for a 32-GPU setup—which is 13.4× faster than SPP, as it leverages compressed state summaries to limit computational overhead. While the current framework prioritizes privacy and near-optimality, we acknowledge that further scalability may require enhancements. In future work (Section VI), we plan to explore adaptive techniques, such as heuristic pruning for layer splits or approximate dynamic programming, to reduce costs without compromising solution quality. These methods could integrate with CDynP’s chain structure to handle exponential growth in model parameters, ensuring robustness for next-generation AI training. Thus, although complexity is a consideration, CDynP’s design already provides a balanced trade-off, and ongoing efforts aim to enhance efficiency for larger deployments.
|
ICACT20260329 Question.3
Questioner: xueyy@dsp.ac.cn 2026-02-04 오후 2:03:42 |
ICACT20260329 Answer.3
Answer by Auhor yangyn@dsp.ac.cn 2026-02-04 오후 2:03:42
Chrome Click!! |
| CDynP adopts a logical chain structure for state inheritance and backtracking across data centers.
If a data center in the chain experiences performance degradation or network instability, could this have a cascading effect on the overall planning process?
Do the authors consider extending the framework to more robust topologies, such as tree-based or multi-path coordination schemes? |
Thank you for this insightful question regarding the robustness of CDynPs chain-based structure. The logical chain in CDynP is designed for privacy-preserving state inheritance, where each data center (DC) sequentially processes compressed state summaries (SS) from its predecessor. While a performance degradation or network instability in one DC could theoretically cause delays, CDynP mitigates cascading effects through its decentralized design: each node performs local optimization independently based on received SS, minimizing direct dependencies. Our evaluations in Section V.B.2 demonstrate CDynPs resilience under degraded network conditions, as it incorporates inter-DC latency into cost models, allowing adaptive adjustments to partitioning. For instance, as shown in Figure 8, CDynP maintained the slowest increase in iteration time under varying bandwidth/delay levels, highlighting its ability to absorb disruptions without severe cascading impacts. Regarding topology extensions, we acknowledge that the linear chain may have limitations in highly volatile environments. In the future work section (VI), we plan to explore more robust schemes, such as tree-based or multi-path coordination, to enhance fault tolerance. This could involve dynamic re-routing or integrating emerging network architectures like ICN for hop-by-hop optimization, ensuring scalability and adaptability. Thus, while the current chain is practical for privacy-aware MAN scenarios, we are committed to evolving CDynP for greater resilience in heterogeneous deployments. |
ICACT20260329 Question.4
Questioner: tanghr@dsp.ac.cn 2026-02-04 오후 2:12:20 |
ICACT20260329 Answer.4
Answer by Auhor yangyn@dsp.ac.cn 2026-02-04 오후 2:12:20
Chrome Click!! |
| How does the inter-DC latency magnify the bubble time in pipeline parallelism (PP)? |
In pipeline parallelism (PP), the bubble time refers to the idle periods in the training pipeline caused by the sequential nature of stage execution, where some devices wait for others to complete computation or communication. Inter-DC latency exacerbates this bubble time nonlinearly because the higher latency in metropolitan area networks (MANs) compared to intra-DC links increases the communication time between stages. Specifically, in the per-iteration time formula Titer=(M+S−1)×maxsTstages, each stage time Tstagesdepends on the maximum of computation time Tcompsand communication time Tcomms−1,s. When inter-DC latency is high, Tcommdominates due to factors like propagation delays, leading to elongated stage times. This magnification occurs because the bubble—idle time proportional to S−1—scales with the slowest stage, and latency-driven communication bottlenecks make this worst-case stage longer, reducing overall throughput. As illustrated in Figure 1, the inter-DC delay stretches the communication phases, creating larger gaps between micro-batches and amplifying idle bubbles. This effect is particularly pronounced in cross-DC environments, where latency can be milliseconds compared to sub-millisecond intra-DC links. Consequently, strategies like CDynP incorporate latency awareness into optimization to mitigate this by balancing stage loads and overlapping computation with communication, but the fundamental issue stems from the nonlinear impact of latency on pipeline efficiency.
|
ICACT20260329 Question.5
Questioner: tanghr@dsp.ac.cn 2026-02-08 오후 2:24:32 |
ICACT20260329 Answer.5
Answer by Auhor yangyn@dsp.ac.cn 2026-02-08 오후 2:24:32
Chrome Click!! |
| The paper states that CDynP achieves 8.3% iteration-time increase when scaling from one to four DCs. Is this increase compared to the single-DC baseline or to other baseline methods like SPP? |
Thank you for your question. The 8.3% iteration-time increase for CDynP when scaling from one to four DCs refers to the comparison with the single-DC baseline, not other methods. This is explicitly highlighted in the paper: CDynPs increase is substantially lower than competing baselines, such as SPP (16.0%) and PipeDream (18.2%), demonstrating its efficiency in multi-DC settings. |
ICACT20260329 Question.6
Questioner: tanghr@dsp.ac.cn 2026-02-08 오후 2:26:00 |
ICACT20260329 Answer.6
Answer by Auhor yangyn@dsp.ac.cn 2026-02-08 오후 2:26:00
Chrome Click!! |
| How does the compressed state summary (SS) in CDynP prevent privacy leaks while still enabling optimal global strategy generation? Does it truly not expose any internal information? |
The compressed state summary (SS) in CDynP prevents privacy leaks by design: it only transmits the minimal maximum stage execution time for each stage-layer combination, computed as SSk−1[s][l]=mingdpk−1[s][l][g], which aggregates cost metrics without exposing internal GPU allocations, topology, or resource details. This compressed data enables global optimization by allowing each data center to perform local dynamic programming based on inherited SS from predecessors, thus coordinating strategy generation along a chain.
The papers theoretical analysis confirms that CDynP achieves near-optimal solutions equivalent to full-information methods under privacy constraints, as SS excludes sensitive intra-DC data like bandwidth or GPU mappings. Thus, it truly exposes no internal information while maintaining training efficiency. |
ICACT20260329 Question.7
Questioner: wangsy@dsp.ac.cn 2026-02-08 오후 2:27:13 |
ICACT20260329 Answer.7
Answer by Auhor yangyn@dsp.ac.cn 2026-02-08 오후 2:27:13
Chrome Click!! |
| What exactly does the "compressed state summary (SS)" mentioned in the abstract consist of? How is it transmitted between DCs? |
The compressed state summary (SS) in CDynP is a privacy-preserving data structure that consists of the minimal maximum stage execution time for each stage-layer combination, computed as SSk−1[s][l]=min1≤g≤Gk−1dpk−1[s][l][g], where dpis the dynamic programming array representing the optimal cost for partitioning layers. This SS aggregates only the essential cost metrics, excluding sensitive details like GPU allocations, intra-DC bandwidth, or topology, thus preventing information leakage.
SS is transmitted sequentially along a logical chain of data centers during the "Forward CDynP" phase. Each DC receives the SS from its predecessor, performs local optimization using dynamic programming, and then compresses and forwards the updated SS to the next DC.
This chain-based mechanism ensures decentralized strategy generation without exposing internal data, maintaining privacy while achieving near-optimal performance through hop-by-hop coordination. |
ICACT20260329 Question.8
Questioner: wangsy@dsp.ac.cn 2026-02-08 오후 2:27:59 |
ICACT20260329 Answer.8
Answer by Auhor yangyn@dsp.ac.cn 2026-02-08 오후 2:27:59
Chrome Click!! |
| What is your optimization goal? (Minimum iteration time? Also including partitioning + mapping?) |
The optimization goal in our paper is explicitly to minimize the per-iteration training time (Titer), as defined in Section III (Problem Definition and System Modeling). This objective encompasses co-optimizing two critical aspects: (1) partitioning the model into contiguous stages, and (2) mapping these stages to data centers with specific GPU replica counts. The formulation Titer=(M+S−1)×maxsTstagesintegrates pipeline parallelism costs, where Tstagesdepends on stage computation/communication times influenced by partitioning and mapping decisions. Thus, the goal holistically addresses partitioning and mapping to reduce latency while adhering to privacy constraints, ensuring efficient multi-DC training. |
ICACT20260329 Question.9
Questioner: zhaoyb@dsp.ac.cn 2026-02-09 오전 11:55:25 |
ICACT20260329 Answer.9
Answer by Auhor yangyn@dsp.ac.cn 2026-02-09 오전 11:55:25
Chrome Click!! |
| The current experiments were conducted on a simulation platform. However, in a real-world Metropolitan Area Network (MAN), inter-DC bandwidth and latency are subject to fluctuations (jitter). How does the strategy generated by CDynP address sudden network congestion during the training process? |
Your question rightly points out the challenges of network volatility in real-world MANs. CDynP addresses this by incorporating inter-DC bandwidth and latency costs into its dynamic programming model during offline strategy generation, enabling it to produce plans resilient to expected network variations. For instance, experiments show CDynP maintains low iteration time under degraded inter-DC conditions by optimizing stage partitioning to overlap computation and communication, mitigating latency impacts.
However, the current strategy is static and does not adapt in real-time to sudden jitter mid-training. Future work will focus on dynamic scheduling to handle such volatility, as noted in the conclusions. |
