 ICACT20260329 Slide.01
[Big Slide]
[YouTube] |
Chrome  Click!! |
 |
Good morning, everyone. Thank you for joining me today. I am [Your Name], and I will be presenting our work titled A Distributed Training Adaptive Parallel Strategy Generation Algorithm for Multi-DCs in the MAN: Privacy-Preserving and Network-Aware Scheduling for Large Language Models. This research addresses the challenges of distributed AI training across metropolitan area networks, and I am excited to share our findings with you. |
| ICACT20260329 Slide.02
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
To guide our discussion, I will first cover the background and motivation behind this work. Then, I will define the problem statement and system modeling, followed by a detailed explanation of our methodology—including intra-DC and inter-DC optimization. After that, I will present our experimental evaluation and results, and finally, conclude with key takeaways and future work. Lets dive in. |
| ICACT20260329 Slide.03
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
The trend in large language models, or LLMs, shows exponential growth in parameters—from GPT-3s 175 billion parameters to even larger models today. This creates a significant resource gap, as single data centers struggle to meet the computational demands. |
| ICACT20260329 Slide.04
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
The widening gap between model size and GPU memory necessitates distributed training across multiple data centers in metropolitan area networks, or MANs. These networks connect fragmented compute resources, like edge and cloud DCs, using optical fiber. However, MANs introduce challenges such as high latency—1 to 3 milliseconds—and limited bandwidth of 1-10 Gbps, which are far from ideal for distributed training. Our goal is to efficiently orchestrate these resources into a cohesive computing fabric. |
| ICACT20260329 Slide.05
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Lets now examine the basics of parallelism strategies. Data parallelism replicates models but requires high-bandwidth All-Reduce operations, while tensor parallelism splits operations but needs ultra-low latency. Pipeline parallelism, or PP, is more suitable for inter-DC settings due to lower communication volume, as it only transfers activations between stages.
Here is a comparison table of parallelism techniques: |
| ICACT20260329 Slide.06
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
However, key challenges include heterogeneity in DC resources, network constraints like bandwidth limits, and privacy concerns—since DCs belong to different organizations and cannot share internal topologies. |
| ICACT20260329 Slide.07
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Existing approaches like PipeDream assume homogeneous, high-bandwidth clusters, which fail in real-world MAN environments. This gap motivates our work. |
| ICACT20260329 Slide.08
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Our methodology is built on a two-level architecture: inter-DC synchronous pipeline parallelism and intra-DC hybrid parallelism. |
| ICACT20260329 Slide.09
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
To optimize this, we developed a cost model that minimizes per-iteration training time by balancing computation, communication, and pipeline bubbles.
where stages accounts for both communication and computation times. Decision variables include layer partitioning and device mapping, but we must preserve privacy by not exposing internal DC details. |
| ICACT20260329 Slide.10
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Now I will introduce our proposed method, CDynP. CDynP stands for Chain-based Distributed Dynamic Programming.
The core idea is to decompose a global parallel strategy problem into locally solvable sub-problems, and then coordinate them through a chain structure.
Firstly, each data center performs local optimization independently, using its own cost model and internal information. Then, instead of sharing raw topology or resource details, data centers only exchange high-level summaries.
By doing so, CDynP enables collaborative optimization while strictly preserving organizational privacy. |
| ICACT20260329 Slide.11
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Inside each data center, we apply Local Dynamic Programming.
The scenario is this: given a segment of the model, say layers S to E, how does one DC process it optimally using its local, possibly heterogeneous, resources?
The method is a Local Dynamic Programming algorithm. It automatically balances computation and intra-DC communication costs. We define a DP state, D(s, i, j), which represents the minimum cost to assign the first i layers to j devices. The algorithm recursively explores all possible splits to find the configuration that minimizes the maximum stage time.
The result of this local computation is a Cost function—a black-box performance estimator for that DC. This function encapsulates the DCs heterogeneous capabilities into a simple interface: you give it a layer range, and it returns the minimum possible execution time, all without exposing any internal topology. |
| ICACT20260329 Slide.12
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Now, we transition to the global, multi-DC challenge. We have local cost functions, but two critical questions remain: First, which DC gets which layers? And second, how do we account for the costly inter-DC transmission?
Our solution is a two-phase distributed coordination. |
| ICACT20260329 Slide.13
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Phase 1 is Heuristic Pre-partitioning. Its purpose is to reduce the massive search space. The coordinator node collects aggregate computational capacity, like total FLOPS, from each DC and roughly assigns layers proportionally. |
| ICACT20260329 Slide.14
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Phase 2 is where the core of CDynP executes: Chain-based Dynamic Programming. The DCs are organized into a logical chain. The process begins with the first DC in the chain.
It runs its local DP over its assigned search interval and then compresses the results into a compact State Summary, which contains only layer indices and bottleneck times—no internal details. This State Summary is sent to the next DC.
The next DC inherits this summary, which represents the optimal solution up to that point. It then integrates this with its own local DP, considering the inter-DC communication cost in its state transitions. It finds the best way to extend the global solution using its local resources, compresses the new global state, and passes it downstream.
This process repeats until the tail node has a near-optimal global strategy. The key benefit is distributed execution with strict privacy preservation, as only results, not raw data, are exchanged. |
| ICACT20260329 Slide.15
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
To summarize the complete algorithm flow: It starts with pre-partitioning to define search bounds. Then, each DC performs its local DP. This is followed by the chain propagation phase, where State Summaries are passed forward. Once the tail node finishes, a backtracking process identifies the global optimum and notifies all nodes to finalize their strategies.
Through this elegant flow, CDynP simultaneously handles heterogeneity via pre-partitioning and local DP, adapts to network constraints by factoring communication costs into the DP, and guarantees privacy through the State Summary mechanism. |
| ICACT20260329 Slide.16
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Now, lets move to evaluation. We simulated a 3-DC MAN environment with heterogeneous GPUs—V100s in varying counts—and MAN links with 1-10 Gbps bandwidth and 1-3 ms latency. Models included GPT-1.9B and BERT-Large. We compared CDynP against baselines like standard data parallelism, PipeDream, and synchronous pipeline planning (SPP). Metrics focused on training time per iteration and strategy search time. |
| ICACT20260329 Slide.17
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Our results show that CDynP consistently outperforms baselines. For example, when scaling from one to four DCs, CDynP limited the iteration time increase to just 8.3%, compared to 16% or more for others.As seen in Figure 1, CDynP adapts better to DC scaling. Similarly, for larger models like GPT-1.9B, it reduced training time by 10.1% over SPP by optimizing stage counts and bubble overhead.Figure 2 highlights this model-scale adaptability.CDynP also excels under network variations. For instance, with degraded inter-DC bandwidth, it maintained the slowest increase in iteration time.Figure 4 shows CDynPs stability across bandwidth levels. In terms of algorithm efficiency, CDynP achieved a search time of 39.27 seconds—13.4 times faster than SPP—due to its chain-based approach.Figure 6 illustrates this speed advantage. |
| ICACT20260329 Slide.18
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In summary, CDynP provides a privacy-preserving, network-aware solution for multi-DC training in MANs. It handles heterogeneity, adapts to weak networks, and preserves privacy through hierarchical optimization. Our results demonstrate consistent gains in training efficiency and scalability. |
| ICACT20260329 Slide.19
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Looking ahead, we plan to deploy CDynP on physical MAN testbeds, extend it to support dynamic runtime adaptation and advanced paradigms like mixture-of-experts, and explore integration with emerging network architectures. This will further enhance its practicality for large-scale AI training. |
| ICACT20260329 Slide.20
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Thank you very much for your attention. I am happy to take any questions. |
