ICACT20260115 Slide.24
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Now, I will summarize the evaluation results.
・In the single-flow scenario, RL-HKD achives uo to a 1.39% improvement over NewReno and up to a 0.88% improvement over RL-TCP.
・In the throughput convergence scenario, RL-HKD recovers unused bandwidth faster than CUBIC.
・In the fair coexistence scenario, RL-HKD achieves better fairness than NewReno when coexisting with CUBIC, and better fairness than CUBIC when coexisting with NewReno.
|
|
ICACT20260115 Slide.23
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・This table shows Jain’s fairness index under different bandwidth and RTT settings.
・When comparing the minimum values of Jain’s fairness index between NewReno versus CUBIC and RL-HKD versus CUBIC, the values are 0.9949 and 0.9965, respectively.
・This indicates that RL-HKD has better coexistence behavior with CUBIC than NewReno.
・Similarly, when comparing the minimum values of Jain’s fairness index between NewReno versus CUBIC and RL-HKD versus NewReno, the values are 0.9949 and 0.9992, respectively.
・Thus, RL-HKD shows better coexistence behavior with NewReno than CUBIC.
|
|
ICACT20260115 Slide.22
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Now, I will present the results of fair coexistence scenario.
・We evaluate throughput and Jain’s fairness index, for both NewReno, CUBIC, and RL-HKD.
・Jain’s fairness index measures how evenly bandwidth is shared among multiple flows.
・The evaluation interval is from 20 to 50 seconds, as highlighted in the figure.
・The figure also shows the simulation topology used in this scenario.
|
|
ICACT20260115 Slide.21
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Next, I will explain the cwnd dynamics in the throughput convergence scenario.
・Figure (a) shows the cwnd behavior when both flows use CUBIC.
・The yellow line represents the competing flow n1, which stops during the simulation.
・After the competing flow stops, the cwnd of the n0 flow increases slowly.
・In contrast, Figure (b) shows the cwnd dynamics when RL-HKD is used for the n0 flow.
・We can see that RL-HKD rapidly increases its cwnd after the competing flow stops.
・This indicates that RL-HKD can quickly adapt to network changes, leading to faster throughput recovery.
|
|
ICACT20260115 Slide.20
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・This table shows the average throughput of the n0 flow after the competing flow n1 stops.
・We compare RL-HKD with CUBIC under different bandwidth and RTT settings.
・As we can see, RL-HKD consistently achieves higher throughput than CUBIC after the competing flow n1 stops.
・This indicates that RL-HKD can recover throughput faster when network conditions change.
・To better understand this behavior, we next analyze the cwnd dynamics.
|
|
ICACT20260115 Slide.19
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Now, I will present the results of throughput-convergence scenario.
・We evaluate how quickly throughput converges after the competing flow n1 stops, for both CUBIC and RL-HKD.
・The evaluation interval is from 35 to 50 seconds, as highlighted in the figure.
・The figure also shows the simulation topology used in this scenario.
|
|
ICACT20260115 Slide.18
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Backup slide
|
|
ICACT20260115 Slide.17
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Thank you
|
|
ICACT20260115 Slide.16
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Finally, I will summarize our work and discuss future directions.
・Conventional rule-based algorithms have limitations in adapting to unseen network environments.
・To address this issue, we proposed RL-HKD, a basic RL-based congestion control algorithm.
・We defined four actions with different cwnd growth rates.
・As key results, RL-HKD achieves up to 1.39% improvement over NewReno and up to 0.88% improvement over RL-TCP in bandwidth utilization.
・RL-HKD also achieves faster throughput recovery than CUBIC and better fairness than both CUBIC and NewReno.
・For future work, we consider two main directions.
・1st, we will optimize the action space and neural network for broadband network.
・2nd, we will evaluate the proposed algorithm under more dynamic network conditions, such as wireless environments.
|
|
ICACT20260115 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・This table summarizes the average bandwidth utilization of NewReno, RL-TCP, and RL-HKD under different network parameters.
・Overall, RL-HKD consistently achieves high bandwidth utilization across all evaluated environments.
・In particular, RL-HKD achieves up to a 1.39% improvement over NewReno in the case of bandwidth of 500 Mb/s and RTT of 15 ms.
・Furthermore, RL-HKD achieves up to a 0.88% improvement over RL-TCP in the case of bandwidth of 100 Mb/s and RTT of 15 ms.
|
|
ICACT20260115 Slide.14
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・This slide shows the throughput results in the single-flow scenario, with bandwidth of 100 Mb/s and an RTT of 15 ms, extracted from five trials.
・From left to right, the graphs correspond to NewReno, RL-TCP and the proposed RL-HKD.
・NewReno shows insufficient bandwidth utilization, and RL-TCP shows unstable oscillations.
・In contrast, RL-HKD achieves high and stable throughput.
・The average bandwidth utilization is shown on the next slide.
|
|
ICACT20260115 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Now, I will present the results of single-flow scenario.
・We evaluate throughput and bandwidth utilization for NewReno, RL-TCP, and RL-HKD.
・The evaluation interval is from 20 to 50 seconds, as highlighted in the figure.
・The figure also shows the simulation topology used in this scenario.
|
|
ICACT20260115 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Next, I will explain the evaluation three scenarios:
・In the single-flow scenario, we evaluate throughput and bandwidth utilization under a steady-state condition.
・In the throughput convergence scenario, we evaluate how quickly the throughput recovers after competing flow stops.
・In the fair coexistence scenario, we evaluate fairness when RL-HKD coexists with other congestion control algorithms.
・All results are averaged over five trials, since RL-HKD used DQN and includes random elements.
・Due to time constraints, I only show the result of the single-flow scenario. For more details, please look at the paper in the proceedings.
|
|
ICACT20260115 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Next, I will explain the evaluation environment setup.
・We use the ns3-ai which is an extension of the ns-3 network simulator for integrating machine learning algorithms.
・The reason for choosing ns3-ai is to call PyTorch libraries from the ns3 environment.
・The system configuration is summarized in this table.
|
|
ICACT20260115 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Next, I will explain a neural network of the Deep Q-Network framework.
・The architecture and training parameters of the neural network are summarized in the table.
・The neural network design is inspired by the model used in RL-TCP, which is one of the previous studies.
|
|
ICACT20260115 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Next, I will explain the reward design of the proposed algorithm.
・The reward is defined using a utility function U, which consists of throughput and RTT.
・This design assigns a positive reward for an increase in throughput and a negative reward for an increase in RTT compared to the previous step.
・Based on the parameter tuning, β and γ are both set to 1.0, and ε is set -10.
|
|
ICACT20260115 Slide.08
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Next, I will explain the action of the proposed algorithm.
・We define four actions with different CWND growth rates controlled by the parameter alpha.
・Action 0 corresponds to NewReno’s congestion avoidance behavior, while other actions allow faster CWND growth.
・The agent dynamically selects acitons depending on the network condition.
|
|
ICACT20260115 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Now, I will explain the state of the proposed algorithm.
・We select five important features as the state variables to represent network conditions, including segment acked, bytes in flight, slow start threshold, cwnd, segment size.
・These features are inspired by the state design used in RL-TCP which is one of the previous studies.
|
|
ICACT20260115 Slide.06
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・We propose a congestion control algorithm based on reinforcement learning named RL-HKD.
・RL-HKD is based on reinforcement learning with Deep Q-Networks.
・The CWND increase is determined by the DQN.
・Packet loss is detected by timeouts and duplicate acks similar to NewReno.
|
|
ICACT20260115 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Here, I introduce several existing studies on machine learning–based congestion control algorithms.
・The first example is Remy, which is one of the earliest machine learning–based congestion control algorithms.
・However, since Remy relies on offline training, it has difficulty adapting to unseen network environments.
・The second example is QTCP, which applies reinforcement learning and enables online learning in real time.
・However, its cwnd update amount is fixed at the byte level, leading to an action space with limited flexibility.
・From these observations, it can be seen that reinforcement learning–based algorithms capable of adapting to changing environments remain insufficient.
|
|
ICACT20260115 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Based on the characteristics of conventional congestion control algorithms, two major limitations can be identified.
・First, adapting to unseen network environments is difficult because rule-based algorithms are manually designed for specific network environments.
・Second, rule-based approaches have reached their limitations.
・Therefore, drastic approach is required.
・In this study, we propose an AI-based approach, specifically reinforcement learning.
|
|
ICACT20260115 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・Here, looking back on the history of congestion control algorithms
・Traditionally all existing algorithms have been evolved as rule-based algorithms.
・Many congestion control algorithms have been developed for specific network environments
・However, an important question arises:
・Can rule-based algorithms adapt to unseen network environments?
|
|
ICACT20260115 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
・First, I would like to explain the background of this study.
・Today, the Internet has become an indispensable part of modern society.
・There exist quality-of-service degradations such as high latency, low throughput & packet loss.
・Therefore, efficient congestion control is essential for stable and high-performance internet communication.
|
|
ICACT20260115 Slide.01
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Thank you very much for the introduction. I am Kazuma Aoki from Hokkaido University, and I will present our study, "Performance Enhancement of Adaptive Network Congestion Control Using Reinforcement Learning."
|
