 ICACT20240375 Slide.01
[Big Slide]
[YouTube] |
Chrome  Click!! |
 |
Hi everybody, my name is Yuan Sun. The topic of my speech today is Deep Reinforcement Learning Based Beamforming in RIS-assisted MIMO System Under Hardware Loss. |
| ICACT20240375 Slide.02
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
I will cover the following seven sections below. |
| ICACT20240375 Slide.03
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
First, I will introduce the background of this paper. Reconfigurable intelligent surface (RIS) is an emerging technology that is widely recognized as a foundational technology for the forthcoming generation of wireless communications. Considering the increasingly intricate radio environment, traditional optimization algorithms have become impractical for their complexities. However, with the continuous advancements in hardware design, computers now possess significantly enhanced computing capability. Consequently, a new alternative algorithm called reinforcement learning (RL) has emerged as a main method for machine learning (ML) based RIS-assisted communication.
|
| ICACT20240375 Slide.04
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In this paper, a deep reinforcement learning (DRL) approach is used for the RIS beamforming design in the RIS-assisted multiple input multiple output (MIMO) communication to maximize the system sum rate in the downlink transmission. The main contributions of this paper are summarized as follows. We consider the hardware loss of RIS and propose a phase shift design for RIS-assisted MIMO communication. We design a DRL based RIS beamforming algorithm for the above scheme, by introducing interference signals and interfering with action selection to prevent from falling into local optimal solutions, and the simulation results show that the proposed algorithm has a significant improvement over the typical algorithm.
|
| ICACT20240375 Slide.05
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
For the RIS-assisted wireless communication system, the system diagram is shown in Figure 1, with 𝑘 single antenna users and one RIS with 𝐿 elements,
|
| ICACT20240375 Slide.06
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
The channel model between the base station (BS) and the user 𝑘 can be expressed as the formula, where 𝐡_(𝑏𝑠,𝑘)∈ℂ^(𝑁_𝐵×1), 𝐡_(𝑏𝑠,𝑙)∈ℂ^(𝑁_𝐵×1), and ℎ_(𝑙,𝑘) respectively denote the channel vector from the BS to the user 𝑘, from the BS to the RIS, and from the RIS to the user 𝑘. Considering the conditions of hardware loss, the phase-independent amplitude model of RIS can be obtained as the formula.
|
| ICACT20240375 Slide.07
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
The single-user channel is aggregated to obtain the multi-user channel, which is denoted as the formula.
|
| ICACT20240375 Slide.08
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In the uplink stage, the BS simultaneously receives the pilot signals transmitted by the k users. The pilot signal vector received by the BS is expressed as the formula, where 𝐏∈ℂ^(𝑁_𝑘×𝑁_𝑘 ) is the pilot pattern. The BS uses minimum mean square error (MMSE) to estimate the channel matrix of the received pilot signals, that is the formula.
When P is a unitary matrix, the above estimated channel matrix can be transformed.
|
| ICACT20240375 Slide.09
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In the downlink transmission, data transmission with zeroforcing (ZF) precoding is performed with the following expression, where 𝐏𝑛𝑜𝑟𝑚 is the normalized power matrix and written as this formula.
The signal received by the k-th user is this formula, where 𝑥𝑘 is the signal sent to the user 𝑘, 𝑛𝑘 is additive gaussian white noise.
|
| ICACT20240375 Slide.10
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
For the RIS-assisted wireless communication MIMO system, we will use the sum downlink rate as an indicator to measure the performance of the system. Thus, the problem can be expressed as this formula. The signal to interference plus noise ratio (SINR) is given as the formula.
|
| ICACT20240375 Slide.11
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In contrast to typical DRL algorithms, we use the advanced SAC algorithm. The training of the algorithm involves a trade-off between the maximized expected returns and the maximized entropy. The SAC agent consists of three networks: two Q networks and a random policy network (or actor network). The Q networks take the states provided by the environment and actions produced by the actor network as inputs and produce Q-value estimates, which are scalar values. Given a network of actors 𝜋_𝜓 parameterized by 𝜓, the Q network is trained jointly in the SAC algorithm as the formula.
|
| ICACT20240375 Slide.12
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Similarly, the policy network takes state vectors from the environment as inputs and produces numerical action vectors. The loss function for the policy network in the SAC algorithm is calculated as the formula. For the update of the policy network, we use the stochastic gradient algorithm for the policy gradient ∇_𝜓 𝐽(𝜓) and the gradient ascent algorithm to update the parameter of the policy network as the formula. Finally, the regularization factor of entropy, 𝛼 is a hyper parameter, and a larger value of 𝛼 means that the agent will have a larger exploration space, which enables agent to explore more possibilities instead of being limited to the current action.
|
| ICACT20240375 Slide.13
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In RL, environments are usually distinguished into episodic and non-episodic tasks. An episode can terminate when a terminal condition is met, e.g., the fall of a walking humanoid or reaching a pre-specified number of time steps. Non-episodic (continuing) tasks, in contrast, do not have a certain ending point, and the agent roams the environment on an infinite horizon. Since the BS performs beamforming and configures RIS elements continuously, it is evident that this is a continuing task by nature. While it is possible to specify a set of terminal conditions such as a sum rate threshold or a maximum number of steps, it might introduce bias and mislead the learning agents. Therefore, we adopt the continuing task framework to develop our approach.
|
| ICACT20240375 Slide.14
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Three important covariates of reinforcement learning we define as follows.
Action: For the optimization problem we have constructed, the action of the agent is the beamforming design of the RIS. Since the neural networks cannot handle complex numbers, we need to separate the real and imaginary parts of the RIS phases, and the action space contains 2 𝐿 elements.
State: To take full advantage of the information obtained by the agent interacting with the environment, we define the state space as 𝐇 and 𝚽, the equivalent wireless channel matrix and RIS beamforming. In order to reduce the influence of the correlation of the channel state matrix, we perform a whitening operation on the channel matrix after each environmental interaction.
Reward: For each episode, we will use the sum rate of the downlink transmission to determine the reward in order to choose the right action to get a higher reward for the agent.
|
| ICACT20240375 Slide.15
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
For the non-episode task, we can use average reward instead of the instantaneous reward to allow the agent to learn from the environment as the formula.
The agents to enter the state-action space where Q prediction is difficult to perform, thus allowing them to correct the prediction error of the unknown or less-selected actions as this formula, where ⊙ is the Hadamard product and the 𝜆∈(0,1] is used to restrict the explorer network from influencing the policy network to choose the wrong action.
|
| ICACT20240375 Slide.16
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
At each update step, perturbed actions are sampled from the experience replay buffer instead of the original actions generated by the policy network, and then the loss functions for the Q network and the actor network are as follows.
|
| ICACT20240375 Slide.17
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
We define the loss function of the explorer network by maximizing the sum of the absolute temporal-difference (TD) errors corresponding to the two Q networks as the formula.The deterministic exploration policy gradient is then computed by the deterministic policy gradient algorithm.
|
| ICACT20240375 Slide.18
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
We set the simulation parameters as follows.The BS is equipped with a uniform linear array (ULA) that is placed at [1, 0, 0] (i.e., x-axis), and RIS is a ULA, and located at [0, 1, 0] (i.e., y-axis), and the user is equipped with single antenna and is randomly generated in a circular range. We set the number of the BS antennas as 4, the number of users as 4, and the number of RIS elements as 32. One channel episode includes 20 channel blocks. The parameters of the Q network and the parameters of the RIS hardware loss model are given in Table I.
|
| ICACT20240375 Slide.19
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In Fig. 2, we analyze the influence of different algorithms on the system sum rate. Compared to the random algorithm, the RIS beamforming design can significantly improve the system sum rate, and the proposed algorithm achieves significant improvement over the SAC algorithm.
|
| ICACT20240375 Slide.20
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
The performance of the proposed algorithm for different 𝜁is investigated in Fig. 3. For the convenience of observation, we set the typical SAC algorithm as a benchmark. Comparing with Fig. 2, it can be seen that the proposed DRL algorithm with 𝜁=0.6has a significant improvement in the system sum rate over the SAC algorithm, and it is also observed that the system sum rate 𝑃_𝑚 of the proposed algorithm increases as the value of 𝜁increases.
|
| ICACT20240375 Slide.21
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
We conclude with a summary of the work done on the thesis.
We study the RIS beamforming design in the RIS-assisted MIMO communication and propose a modified SAC-exploration algorithm to solve the RIS beamforming design considering the RIS hardware loss.
The simulation results show that the proposed algorithm is able to realize better RIS beamforming and has a significant improvement in the downlink system sum rate compared with the typical SAC algorithm.
Considering the case of multiple RISs, we will consider the multi-agent algorithm to solve the beamforming design problem in the future.
|
| ICACT20240375 Slide.22
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Thank you to all the experts, teachers and students for listening! |
