 ICACT20220065 Slide.01
[Big Slide]
|
Chrome  Click!! |
 |
Hello, I'm Yong Hee Park, who will be presenting a paper. The title of the paper is The Method of Learning Personal Preference with Reinforcement Learning I will start the presentation. |
| ICACT20220065 Slide.02
[Big Slide]
|
Chrome Click!! |
 |
First, a description of background research. Reinforcement learning is a method of learning by interacting with agents and the environment. It learns a policy that determines action in a specific state. Personal preference learning for many studies that were conducted. Learned personal preferences are primarily used in marketing and advertising. This is a big paradigm that can be divided into two. One is a content-based recommendation. It is a method of recommending items that the user liked in the past. The other is a collaborative recommendation. It identifies users similar to those preferred by the user. Some features of existing studies probabilistic models, Bayesian or neural networks, decision trees of machine learning methods shall be used. |
| ICACT20220065 Slide.03
[Big Slide]
|
Chrome Click!! |
 |
In Personal Preference Learning, our goal is to compare learned data with users' choices.
Existing personal preference learning does not fit this purpose because it is a method of supervised learning or unsupervised learning.
Therefore, we propose a personal preference learning method using reinforcement learning based on user selection.
To this end, we learn the choices for each state.
It also creates a policy for user patterns.
|
| ICACT20220065 Slide.04
[Big Slide]
|
Chrome Click!! |
 |
Previously, various learning methods for user preference have been proposed.
One study proposed a personal preference learning method using Support Vector Machine (SVM).
One study proposed a method of learning and recommending personal preferences using the Restricted Boltzmann machine.
One study proposed a method of analyzing the relationship with preference by extracting keywords from images.
Our goal is to learn the user's preference based on the user's decision, which is different from the above.
To this end, the input data and the learned actual data are compared and evaluated.
|
| ICACT20220065 Slide.05
[Big Slide]
|
Chrome Click!! |
 |
Reinforcement learning is a method in which agents and environments interact with each other to learn.
This proceeds with learning through the results of the action in a specific state.
It aims to find its optimal policy and calculates the reward.
It is characterized by being defined in the 5-tuple form of the Markov Decision Process.
5-tuple includes elements of state, action, transition probability, reward, and Discount Factor.
The figure below is an overall structural diagram of reward calculation and reinforcement learning for state and action.
|
| ICACT20220065 Slide.06
[Big Slide]
|
Chrome Click!! |
 |
The Q-value function is a value function for action in a specific state.
Therefore, it is expressed as a pair of state and action.
The expression of the value function includes immediate rewards and future rewards.
Q-table is a table that can represent q-value, and its structure is as follows.
|
| ICACT20220065 Slide.07
[Big Slide]
|
Chrome Click!! |
 |
Q-learning updates q-value and proceeds with learning.
This updates the existing q-value and the newly learned q-value by weighted sum.
The formula and process for Update are shown in the following figure.
|
| ICACT20220065 Slide.08
[Big Slide]
|
Chrome Click!! |
 |
The following is the proposed method.
State includes time information and decision information to learn about the user's selection.
In addition, external elements utilize weather data.
Action is the number of cases that the user can select.
The reward increases if the action selected by the agent and the actual user's choice is correct, and decreases if not.
The overall system structure is shown in the figure below.
|
| ICACT20220065 Slide.09
[Big Slide]
|
Chrome Click!! |
 |
The following is an algorithm for Learning Personal Preference with RL.
First, Environment, State, Action, Q-value, and Variable parameters are initialized.
The learning process first selects an action and then calculates the next state, result, and done.
Finally, update the Q-value.
|
| ICACT20220065 Slide.10
[Big Slide]
|
Chrome Click!! |
 |
The following is an algorithm for the Environment Step.
Next state is the next user's choice.
Reward increases if action and state are the same, and decreases if not.
At the end of the day, an episode ends.
|
| ICACT20220065 Slide.11
[Big Slide]
|
Chrome Click!! |
 |
Next is a simulation.
When simulating in this paper, data uses virtual data.
A method for creating virtual data is as follows.
The date was set to one year.
Decision data was assumed to be a choice for a food menu.
In this case, the menu was divided into eight categories from A to H.
X is the case when the menu is not selected.
Weather data were collected from the Meteorological Administration's ground time data inquiry service API in Korea.
|
| ICACT20220065 Slide.12
[Big Slide]
|
Chrome Click!! |
 |
Virtual data was generated stochastically.
Mealtime was classified into four categories: breakfast, lunch, dinner, and night food.
The range of mealtimes for each time was set as shown in the figure at the top.
The probability of mealtime is a probability table for what time the subject will choose within meal time.
In this case, the probability was set differently by classifying the same cases as when the temperature exceeded 20 degrees.
The Probability of Menu Decision is a table of what menu to choose at each mealtime.
Likewise, the probability distribution was set differently based on the temperature of 20 degrees.
|
| ICACT20220065 Slide.13
[Big Slide]
|
Chrome Click!! |
 |
Next is the simulation setting.
The specific scope and setting for the Observation state are shown in the table above the left.
Action is divided into indexes, which are managed as action vectors.
Parameters of Environment include setting the number of executions of the episode and whether to learn repeatedly one day.
Parameters of an agent include epsilon, learning rate, and gamma, which are parameters of RL.
It also includes rewards when the agent's action is correct and not.
|
| ICACT20220065 Slide.14
[Big Slide]
|
Chrome Click!! |
 |
The simulation results are as follows.
The figure above is a graph of the reward by episode.
Rewards do not converge because virtual data was generated probabilistically.
When performing a greedy search in the learned results, it appears as shown in the figure below.
If j is 0, the temperature is less than 20 degrees, and if j is 1, it is higher than 20 degrees.
From the learning results, we were able to confirm that when generating virtual data, the time and menu with high probability distribution were selected.
|
| ICACT20220065 Slide.15
[Big Slide]
|
Chrome Click!! |
 |
In conclusion, we confirmed that the learned policy is similar to the probabilistic generated virtual data.
However, compared to the actual data, some errors were included.
This is because the input data is based on probability.
We learned personal preferences using only basic processes.
As a way to compensate for this, it plans to utilize optimal learning methods such as improvement of accuracy in the future.
|
