|
Inside each data center, we apply Local Dynamic Programming.
The scenario is this: given a segment of the model, say layers S to E, how does one DC process it optimally using its local, possibly heterogeneous, resources?
The method is a Local Dynamic Programming algorithm. It automatically balances computation and intra-DC communication costs. We define a DP state, D(s, i, j), which represents the minimum cost to assign the first i layers to j devices. The algorithm recursively explores all possible splits to find the configuration that minimizes the maximum stage time.
The result of this local computation is a Cost function—a black-box performance estimator for that DC. This function encapsulates the DCs heterogeneous capabilities into a simple interface: you give it a layer range, and it returns the minimum possible execution time, all without exposing any internal topology. |