ICACT20220153 Slide.23
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
This paper designed the environment and agent to optimize ESS SoC and reduce user satisfaction.
Through this, it was confirmed that the optimal policy for two dilemmas was learned.
This paper conducted RL through a single agent.
However, in this simulation, several EVs were used. Therefore, there is a problem that the state-action dimension increases by the number of EVs.
Because of this, it took quite a long time to learn.
Therefore, there is a need for a method of applying the MARL for allocating an agent for each EV.
|
|
ICACT20220153 Slide.22
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The next is EV's SoC.
Similarly, it can be seen that the EV was randomly discharged in Episodes 100 and 2200, while the power was discharged in advance in 3100 considering the time the EV started.
|
|
ICACT20220153 Slide.21
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The following is a table summarizing the values for the average ESS SoC and the shifted amount of power previously identified.
We confirmed in episode 2200 that the reward has a maximum value and a value that is repeated afterward.
The average value of ESS SoC in Episode 100 is 6.5 and 6.33 at 3100.
In this case, the decrease in the average value of ESS SoC decreased to 97.4%.
The shifted power decreased significantly to about 80% from 94.41 near Episode 100 to 75.57 at 3100.
Our goal is to optimize ESS SoC optimization and minimize user satisfaction reduction.
To this end, the RL was modeled.
In this simulation, ESS SoC should have a higher value and shifted power should have a lower value.
From the above results, it can be seen that even if the average ESS SoC is similar, a policy to significantly reduce shifted power has been learned near episode 3100.
|
|
ICACT20220153 Slide.20
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Next is the shifted amount of power.
The shifted power near 2200 has a maximum value.
|
|
ICACT20220153 Slide.19
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The following is a graph of ESS SoC.
Each graph is output for each episode, and the unit is a step.
Reward is the maximum at episode 2200 and the average ESS SoC is also the maximum.
The neighboring episode with a low reward value in the section repeated after 2200 confirmed in the previous figure is 3100.
In this case, the average ESS SoC also decreases.
|
|
ICACT20220153 Slide.18
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The following is a graph of the reward for the episode.
As a characteristic of the result, the reward reaches a maximum at 200 and then repeats from 2200.
The reason for this will be explained by checking other result graphs.
|
|
ICACT20220153 Slide.17
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The collected power supply and demand data are shown in the following figure.
Data is characterized by more demand than supply.
|
|
ICACT20220153 Slide.16
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The setting of each parameter for simulation was set as follows.
|
|
ICACT20220153 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The following table is the data collected by the ACN dataset.
Each EV is managed as an id and includes connection time, disconnection time, charging time, and power requirement data.
|
|
ICACT20220153 Slide.14
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The following table shows the data set of power information collected at Smart Start.
Each data was divided into groups of RG, IL, and NIL.
|
|
ICACT20220153 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The HW and SW specifications used are as follows.
In agents, state and action generally use the box format of OpenAI Gym.
|
|
ICACT20220153 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Next is Agent.
Because the agent uses DQN, it initializes the related MLP and DQN.
It also initializes observation space and action space to interact with the environment.
Learning agent is an agent for learning.
It receives state, return, and done information from the environment and selects an action.
At this time, the search method for selecting action is the epsilon greedy method.
This is to solve the problem of exploring only the initially learned q-value value.
The learned q-value is stored and managed.
Test agent is an agent to check the learned results.
The difference from the Learning agent is the use of the greedy method to check the best results.
Also, instead of learning and saving q-value, it brings up the stored value and stores the final result.
|
|
ICACT20220153 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Calculate return is an algorithm that calculates the final reward.
It calculates the final reward through each reward returned by the EV Controller, Run Action, and ESS SoC Update algorithms described earlier.
It returns the next state, reward, and done.
|
|
ICACT20220153 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
ESS SoC update is finally an algorithm that manages changes in ESS SoC.
It also performs IL to optimize ESS SoC.
It returns the shifted amount of power.
|
|
ICACT20220153 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Run Action is an algorithm that determines EV and power emission through the selected action.
This returns the value for the power, EV misselection emitted by the EV for optimization of ESS SoC.
|
|
ICACT20220153 Slide.08
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Next is the RL design for the DR system.
EV Controller is an algorithm that manages when EVs leave and arrive.
This returns the insufficient amount of EV SoC.
|
|
ICACT20220153 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The following is the formation for RL.
The first is the state.
The state contains the following elements.
Time and the remaining time of EV are the main information related to patterns such as power demand, PV power generation, and EV use.
ESS SoC should be observed for ESS SoC optimization, one of the purposes of this paper.
EV SoC, IL, and NIL should be observed for user satisfaction, one of the purposes of this paper.
Action is the determination of the EV index and supply of EVs to supply power to ESS.
Therefore, the action has a range for the discharge amount of EV. Rewards include rewards for the optimization of ESS SoC.
It also includes penalties for reducing user satisfaction by IL and V2G and misselection of EVs.
|
|
ICACT20220153 Slide.06
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The overall system architecture is shown in the following figure.
The structure of the system is environment and agent.
Environment is divided into getting data, EV data, home data, and steps.
Get data collects data.
Data on PV's power output and requirements are collected from the Smartstar project.
This is managed by the home database after importing data in CSV form.
Data related to EVs are collected by the ACN dataset.
This is managed by EV Dataset by collecting data in the form of HTTP requests and parsing JSON format data.
The step is the unit of change in the environment.
It takes data from EV dataset, one dataset.
It also receives an action from the agent, updates the state, and calculates return and done.
Each state, reward, and done is delivered to the agent. Agents are divided into initiation, test agent, and learning agent.
initiation initializes the agent.
In this paper, we use DQN.
The learning agent uses the epsilon greedy search method as an agent for learning.
The test agent uses the greedy search method as an agent to check the learned results.
|
|
ICACT20220153 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Studies have been proposed to solve these problems.
One study is a paper we previously proposed that optimizes ESS SoC through DR.
This was modeled on a limited environment, such as the usage pattern of EVs and the capacity of each EV SoC.
Methods of learning DR using RL have also been proposed, but ESS SoC and user satisfaction have not been considered.
When learning, we perform considering ESS SoC and user satisfaction.
|
|
ICACT20220153 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
ESS SoC needs to be optimized.
When this is high, if supply power exceeds demand and ESS capacity, the remaining power is wasted.
Conversely, when this is low, peak load response performance decreases.
From this point of view, ESS SoC should be optimized.
It also affects battery life and electricity bills.
Lastly, ESS Have a minimal SoC to maintain self-power
DR affects user satisfaction.
IL delays the power to be supplied to the user.
V2G reduces the SoC of EVs.
Therefore, it is necessary to minimize user satisfaction by DR.
|
|
ICACT20220153 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The balance between demand and supply of power is one of the important issues.
Power information can be used to solve this problem.
An energy storage system is a system that stores power.
This can store power when the supply of power is greater than the demand, and resupply the stored power when the demand exceeds the supply to the grid.
This can alleviate the imbalance of power and the uncertainty of RG's power production.
State of Charge of ESS means the amount of charge.
The value of this is determined by the supply and demand of electricity.
It also affects battery life and peak load response.
This can be adjusted using DR.
|
|
ICACT20220153 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
First, I will explain the background of the study.
Smart Grid is the next generation of power technology that combines IT with power grid.
It can do the innovation efficiency and reliability of energy processing.
The renewable generator is a technology that generates electricity using natural energy.
Its technologies include photovoltaic and wind turbines.
Because it is based on natural energy, it has uncertainty about power production.
Demand response is a technology that cooperates with power providers and consumers and can be used to stabilize power grids.
It's used to reduce peak demand or secure spare electricity.
Methods of this include Interruptible Load and Vehicle to Grid.
|
|
ICACT20220153 Slide.01
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Hello, I'm Yong Hee Park, who will be presenting a paper.
The title of the paper is Reinforcement Learning Base DR Method for ESS SoC Optimization and Users Satisfaction.
I will start the presentation.
|
