Now 5214 visitors
Today:0 Yesterday:0
Total: 146365
215S 51P 5R
2026-10-21, Week 43 |
| Member Login |
| TACT Journal Homepage |
| Call for Paper |
| Committee |
| Find My Paper |
| Author Homepage |
| Statistics & History |
| Welcome Message |
| Paper Procedure |
| FAQ |
| Paper Archives |
| Outstanding Papers |
| Author Homepage |
| - Paper Procedure |
| - Journal Procedure |
| - Presentation Tips |
|
| Program & Proceedings |
| Program with Papers |
| Plenary Session |
| Tutorial Session |
|
| Presentation Platform |
| Hotel & Travel Info |
| About Korea |
| Accommodation |
| Transportation |
| VISA |
| Other Infomation |
|
| Photo Gallery |
| Scheduler Login |
| Seminar |
| Archives Login |
|
| Sponsors |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
main_email:shahram@ieee.org
| Save the slide and description |
|
*** You can edit any slide by selecting the Slide number below***
|
ICACT20210131 Slide.16
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
Question and Answers.
|
|
ICACT20210131 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
For future work, it would be more practical to have a
complex reward function returning multiple values as opposed
to -1 or +1. In doing so, we will be able to theoretically
determine the rate of congestion, and which links have the
worst congestion based on rewards as currently congestion is
binary. Furthermore, this environment specifically focuses on
re-routing one TCP stream, which at the same time can be
scaled which should be explored further. As this algorithm
was engineered with the mindset of the core data center, this
algorithm can run one TCP-CA/RL algorithm per leaf switch,
and look at TCP statistics from all flows which are considered
high priority that are leaving the switch. In doing so, re-routing
will be done locally, whilst allowing for easy scaling.
|
|
ICACT20210131 Slide.14
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In this paper, we presented a TCP-CA/RL based solution
of mitigating congestion caused by saturation at the network
layer, by using statistics from end hosts to approximate congestion
using RL in a Software-Defined environment and we
were able to show that in our tests, we were completely able
to minimize congestion caused by re-routing traffic in realtime.
We also noticed the results were reproducible when run
multiple times, with some variation which was expected due to
the nature of our testing. Furthermore, we also confirmed that
the longer we have TCP-CA/RL learning about the topology
and if the knowledge carries over from iteration to iteration,
data transfer time is kept consistently low even in situations
where 3 out of the 4 uplinks are congested. The next steps
would be to test the algorithm in larger environments, with a
greater amount of data exchange to analyze the response of the
algorithm when run for a longer period of time. Theoretically,
the longer the algorithm is run, the better understanding it will
have of the environment so we should see an optimal transfer
time as if there was no congestion to begin with.
|
|
ICACT20210131 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Comparing specifically, we can see that the time taken to
transfer 100 Megabytes of data varies by approximately 60
seconds. Furthermore, on average the number of packets being
transferred is hovering between 80 and 100 packets, except
for the dip down to 20 when both priority and congested flow
paths are on the same path, which is significantly greater in
Figure 3 which is purely random. The drop in packet transfer
significantly lessens with time a s well, which shows that the
RL algorithm is able to learn which path is congested and
actively avoid that, further proving our algorithm. In this case,
the RL algorithm is able to correctly choose the path 750
seconds into the data transfer, which is completed by around
1000 seconds, meaning if the data transfer was for a longer
period of time, we would have seen far superior transfer times
compared to random path selection.
|
|
ICACT20210131 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
To compare, the highest transfer time witnessed was with
for a = 70 and E THRESHOLD = 0.1. in this scenario,
it would take TCP-CA/RL 700 path changes to converge into
a purely greedy model where TCP-CA/RL would make all
the decisions, which is both inefficient not needed as the data
transfer would have completed well before 700 path changes
were completed.
Lastly, the comparison for the I/O data transfer can be
seen in the figures on this slide and the next where we are visualizing a single
TCP stream of 100 Megabytes of data transfer via iperf
using RL, Random path selection every five seconds, and
using standard TCP congestion control when both priority and
background flows are on the same path, ( which i s provided
to provide a baseline), respectively.
|
|
ICACT20210131 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Through manual analysis, it was seen that when the
background traffic and the priority traffic are taking the
same link, the throughput as per the information for iperf,
was significantly lower which was expected. The algorithm
provides a 20,000 byte buffer for the as this amount of
data can be considered negligible from a data transfer point
of view, and as a result a less than this should not be
counted as a negative reward. This is further described in Table 1, outlining background flow, b, and priority flow,
p. Furthermore, in Table 1 we outline a baseline of the
environment and the time taken to transfer 100 Megabytes
of data. Another key data point to note is when random
paths were used to send traffic for the priority traffic, every
five seconds, we calculated it took an average 17 and a
half minutes, this will be compared to the time taken by
TCP-CA/RL.
|
|
ICACT20210131 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We used Mininet to simulate an SDN environment,
and the framework defined in OpenAI Gym for RL. The
topology for our experiment is shown in Figure 1. In this
environment, three hosts (H1, H2, H3), connect to three leaf
switches respectively. The leaf switches upstream, connect to
4 spine switches upstream, using the spine and leaf topology
which is the latest trend inside the data-center allowing for
east-to-west traffic to move efficiently. From an expansion
point of view, new leaf switches are connected to the spine as
the path length from any host is always just three hops so this
environment can grow without disruption to existing traffic,
making it ideal for the core data center.
Background traffic was generated on one of the four uplinks
to simulate congestion. As the links connected in Mininet
are set to unlimited bandwidth by default, the bottleneck, or
congestion which will be artificially induced will be on the
uplinks.
|
|
ICACT20210131 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Once the reward is calculated, it is fed into Equation (1)
which is used to calculate how ’good’ of a decision was made
by the RL algorithm. The value that is calculated is known
as the Q-Table value for that particular state/action transition
(e.g. going from path x to path y )which is stored in a matrix,
with the best decision having the highest value. After the for
loop completes iteration as defined by , then E is decreased
by the variable E THRESHOLD so in the next iteration,
there is a greater chance that TCP-CA/RL will select the
path, eventually having the algorithm adopt a greedy model,
selecting the Q-value at each state transition that has yielded
the highest reward based on past iterations.
|
|
ICACT20210131 Slide.08
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The idea that is being proposed in this paper, has some
common aspects to the aforementioned paper, however we
will be addressing a slightly different problem. In the data
center, although there are equal-cost multipath (ECMP) routing
enabled which allow for multiple paths to be used to send
traffic simultaneously, there is still an inefficiency there as the
ECMP path selected is based on a hash of source/destination
IP/port. This means although there are multiple paths, traffic
may continuously take a congested path if that is the path
which is selected by ECMP routing. In our research, we are
utilizing OpenAI Gym for RL, which will take input from TCP
statistics in a Software-Defined Network to make decisions on
routing over ECMP paths.
|
|
ICACT20210131 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Deep TCP-RL for congestion control has been explored in
[5], employing the Aurora algorithm against TCP to increase
the throughput by trying to distinguish between packet loss
and congestion, utilizing Neural Networks. This was done by
developing a framework utilizing Open AI Gym Environment.
The two most important parameters which were considered
was history length - how much in the past the agent would
go based on data retention, and discount factor - which
showed that with a higher discount factors, good results are
achieved quickly. Simulation results using Mininet showed
that Aurora was comparable to or outperformed many state
of the art congestion control algorithms such as BBR, PCCVivace,
RemyCC, and Copa. The reward function utilized was
a linear function that rewarded throughput and penalized loss
and latency with coefficients used to force the model to balance
throughput and latency. In the analysis, it was found that TCP
Cubic achieves higher throughput at the expense of higher
latency, however it is plagued with bad performance in many
different scenarios [6]. In [7], the best Initial Window value
and Congestion Control scheme is determined to be used by
RL, based on network characteristics. In [8] both Supervised
Learning and RL is used, with received ACKs, state (e.g. RTT)
as the RL feature input.
In [9], a framework was also presented utilizing Long-Short
Term Memory (LSTM) in a Deep RL (DRL) framework for
MPTCP, in a physical network, which provides benefit to
user traffic by utilizing multiple network connectivity (e.g.
Ethernet, Wi-Fi, and LTE) to load balance a single TCP flow.
The results of this research shows that the use of this type of
framework outperforms traditional MPTCP congestion control
algorithms. While this work does not focus on data centres, it
shows the promise of TCP-RL and further confirms the use of
Q-Learning as the datacenter environment is more predictable
than utilizing multiple network connectivity points, but still
very critical to increasing throughput.
Other works have been done to optimize TCP for data center
networking [10]. In this paper, D-TCP estimates the variable
bandwidth, and based on end-host information, quickly adapts
to changing environments where traditional TCP is not sufficient
for the modern era of network traffic demands.
Work in Omniscient TCP [11] further demonstrates the
inadequacy of TCP to operate in low-latency, high throughput
environments. OTCP in particular achieves a higher level of
performance than standard TCP by configuring TCP congestion
parameters on a route by route basis based on network
characteristics.
Lastly in [12], which utilizes the TCP-RL algorithm, it was
proven that QTCP operated better than TCP NewReno, further
giving momentum to our research.
|
|
ICACT20210131 Slide.06
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Among AI techniques, Reinforcement Learning (RL), due
to its operational characteristics and its continuous interactions
with its environment to learn via trial-and-error, can be a viable
option in network control and management, especially due to
its ability to respond to a changing environment
Comparing this to Supervised and Unsupervised Learning the algorithm must be fully trained and then applied to a
dataset for testing. If the algorithm is not able to respond as
predicted, it would require further re-training of the algorithm,
which is not scalable for networking due to the continuous
learning required. Such techniques can also be used for
translating intents into network policies, which can be done
in conjunction with AI models to monitor performance and
detect network problems
|
|
ICACT20210131 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
AI is one of the fastest growing areas of research, and its
applications have been tried and tested in areas such as stock
markets, real estate, big data, and more. However, adoption
in the area of networking has been slow due to the everchanging
nature of network traffic, making it very hard to
predict and distinguish between normal and abnormal traffic
conditions. Also, due to the characteristics of network traffic,
it is not possible to have realistic and accurate traffic datasets
which can be used for training in every scenario, as the AI
will learn the intricacies of the training data, but the training
models will not translate well to other situations, especially
into production networks. The main reason is that as traffic
patterns are unpredictable, network traffic from one day to
another day can vary greatly which may inaccurately classify
traffic as problematic when that may not be the case.
|
|
ICACT20210131 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In a software-defined network, network functions can be
controlled from one or more central location(s) that are not
on the data path. This would allow deployment of crosslayer
software-based modules with access to global network
information to detect a network scenario (e.g. congestion) and
allow re-routing to be done, based on the TCP parameters
of a particular flow. Such modules can employ a variety of
techniques, including artificial intelligence (AI).
|
|
ICACT20210131 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
TCP is natively built to handle congestion; however, this
is done through controlling the traffic it sends while in a
congested state, as network conditions that attribute to congestion
over the network are not controllable by the Transport
layer. Some mechanisms to deal with congestion include slowstart,
where the network is slowly probed with data so to
not overwhelm the network, however in doing so the full
capacity of the network is not used. Routing
algorithms such as OSPF do not consider available bandwidth
on the link, so traffic will continually be routed over a link
either because that is the only path available, or because the
specific flow has been hashed to take a particular link. But
changes made on the network layer can directly impact TCP
performance, which will affect user experience. To add on,
if a queue on the router overflows because it is still waiting
to send traffic over the link, the packets will be dropped or
severely delayed resulting in a negative user experience.
|
|
ICACT20210131 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
TCP natively handles congestion through congestion windows and slow-start. Datacenter traffic is quite different from Wide-Area Network (WAN) and the same methods of mitigating congestion do not apply because data-center networks are high bandwidth, low latency environments, while WAN traffic is typically low bandwidth, high latency. In the datacenter, TCP operates at a much higher throughput as the situation allows for it, meaning that once congestion is detected, throughput is decreased at a higher rate than in WAN environments.
|
|
ICACT20210131 Slide.01
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Introduction
|
| |
