ICACT20220024 Slide.17
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Thank you for your attention!
|
|
ICACT20220024 Slide.16
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We have shown an explainable flow-data classification model for hacker’s attacks and malwares. The flow data used for training the model were converted from packets by CICFlowMeter. This process greatly shrank the size of data which reduced the requirement for data storage. We then introduced CSTITool to feature extraction processes with an aim of improving the model performance. We compared the XGBoost models that were trained with features obtained by CICFlowMeter and CSTITool. The result showed that the additional descriptive flow statistics created by CSTITool can help improve the performance metrics. In order to understand how the model made predictions, we used SHAP to further investigate the relation between cyberattacks and network flow features. The experiences showed that our model achieved good accuracy in both hacker’s attack and malware classification. It provided interpretability in hacker’s attacks. However, there is still no sufficient theoretical explanation for the causal relationship between the features and the malwares, and further research is needed.
|
|
ICACT20220024 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We have mentioned that the CSTI-10-B model is expected to achieve better performance than the CSTI-10-A model due to the extra features it uses. We sum the mean absolute Shapley values per feature across the data of each malware, and then sort the features by decreasing importance. We found that among the top 20 important features of these ten malwares, on average at least 7 are additional features that we added, which means that the additional features have a considerable contribution in identifying malware.
The data from Emotet and Rig have better improvement in performance, so we use them to plot SHAP beeswarm plot.
Time-related features such as Interarrival Time and Flow Time have a stronger impact on Emotet. For example, if the value of idle max or idle std is higher, the case is easier to be classified as Emotet. Cases with a low value of Flow IAT Min also tend to be labelled as Emotet. Besides, features related to Flow Descriptors and Flags are more influential to Rig class. For example, cases with a large Fwd Act Data Pkts, large Fwd Act Data Pkts, or low FIN Flag Count have a higher chance to be classified as Rig.
The SHAP tool provides an approach to understand the impact of the network flow features on the malware classification. However, much uncertainty still exists about the causal relationship between the features and the malwares. For example, although the idle and IAT related features are influential to Emotet class, and Flow Descriptors and Flags are important to classify Rig class, a theoretical explanation for their relation is still lacking. This indicates a need for further research on how the important features impact on each malware.
|
|
ICACT20220024 Slide.14
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Next, we want to see how features contribute to the prediction for a specific observation. For multi-class classification, SHAP will return a separate matrix for each class for an input. It means that we expect five SAHP force plots for one prediction.
|
|
ICACT20220024 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
To further understand how each individual feature affects the classification, in other words, to understand the relation between the magnitude of a certain feature and the impact on the prediction, we use SHAP beeswarm plot.
Figure a and Figure b show that an attack with small Fwd IAT Min value has a higher chance of being classified as Brute Force or Endpoint Denial of Service. IAT is the average interval time between two packets. As Brute Force and Endpoint Denial of Service are both types of attacks that attempt to connect to the target many times in a short period of time, they usually have a small Fwd IAT value.
By means of port scans and vulnerability scans, a hacker may attempt to get a listing of services running on the targeted hosts. CIC-IDS2017 records Network Service Scanning behaviours which adversaries utilized Nmap to implement Port Scan on the remote hosts. One of the port scan techniques supported by Nmap is Xmas scan, which sets the FIN, PSH, and URG flags. This may be responsible for the high PSH Flag Count in Network Service Scanning class.
Contrary to Brute Force and Endpoint Denial of Service, the connection frequency of Remote Access Software is lower, that is, the IAT value should be smaller. As a result, the larger the value of Bwd IAT Min and Fwd IAT Min, the higher chance it will be classified as Remote Access Software.
The beeswarm plot combines feature importance with feature effects, giving us preliminary understanding of the relationship between the value of a feature and the impact on the prediction. To display explanations for individual predictions, we can use SHAP force plot.
|
|
ICACT20220024 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The dependence shows the interaction between the SHAP value for Destination Port, Destination Port, and Source Port. The cases with 8080 Destination Port have high SHAP values, validating our presumption. The bar chart also shows that Network Service Scanning can be detected by Backward Packet Per Second and Packet Length Mean and Brute Force can be effectively identified by Minimal Forward Segment Size.
|
|
ICACT20220024 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The experiments suggest that our models trained with network flow data are able to achieve good performance in both hacking and malware classification. However, in addition to the effectiveness, the users also expect transparency, interpretability and explainability of the model. Here we apply the open-source tool SHAP to achieve this goal.
We calculate SHAP value of all CIC-IDS2017 features that are used to construct hacker’s attacks classification model. The features with the top 10 SHAP values are depicted in a bar chart. In this plot the impact of a network flow feature on 5 classes, Benign, T1046, T1110, T1219, T1499, are stacked to create the feature importance plot.
According to the bar chart, Destination Port is the most influential feature. It is also the most important feature for classifying Endpoint Denial of Service, Remote Access Software and Brute Force. Even though Destination Port is the most influential feature for the model, it is insignificant for Network Service Scanning. The second most important feature is Source Port, which is also the second most important feature that influences predicting Remote Access Software. We presume that Destination Port is influential to Remote Access Software due to the fact that this hacking technique is often implemented by a Botnet tool called ARES, which by default is executed by using port 8080. To support this presumption, we have to look at SHAP dependence plot, a scatter plot that illustrates the effect a single feature has on the predictions.
|
|
ICACT20220024 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We applied feature selection to the CSTI-10 dataset and created two variations, CSTI-10-A and CSTI-10-B. For CSTI-10-A, we tested features obtained from CICFlowMeter and 68 were identified as important features. As for CSTI-10-B, a total of 377 features were obtained from CSTITool. A part of the increase in number of features (from 79 to 377) was contributed by One-Hot Encoding. After the feature selection process, only 142 features remain in CSTI-10-B.
We trained two malware classification models based on CSTI-10-A and CSTI-10-B respectively. The performance metrics is illustrated in this fifure. Both flow-data based models performed well in most classes with Precision and Recall higher than 90%. The CSTI-10-B model is expected to achieve better performance as it was built with more features. The extra features in CSTI-10-B help improve the performance metrics of most categories; for instance, Rig’s Precision and Recall are increased by 1.23% and 1.59% respectively, and Recall value of Emotet is increased by 1.15%. Although the Recall value of Nuclear is reduced by 2.7% to 97.3%, the Precision and Recall value of Macro in CSTI-10-B model are higher than CSTI-10-A by 0.1 to 0.2%.
We investigated into three categories with the least accuracy in performance, which are Emotet, TrickBot, and Rig. 71 out of 1743 Emotet were misclassified as TrickBot, while 102 out of 5039 TrickBot were labelled as Emotet. We found that Emotet is often used together with TrickBot, causing the model to learn TrickBot’s behaviours and is responsible for the misclassification. Having only 186 cases, Rig's Recall is also low. It is because data insufficiency will not only make the model unable to learn the characteristics of this type of malware, but also reduce the performance metrics with a small amount of misclassification. In this experiment, there were only 189 Rig cases, 25 of which were incorrectly classified, resulting in only 86.24% of Recall value.
|
|
ICACT20220024 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
This table shows the confusion matrix and performance metrics of the model trained with CIC-IDS2017 dataset. It can be found that the values of Precision and Recall are close to 1 regardless of the predicted category. For the category T1219, which has the least data among all categories, the Precision and Recall are 97.97% and 98.81% respectively. The confusion matrix indicates that the model for detecting hacker attacks which is trained with network flow data has good performance in identifying T1110, T1499, T1046, and T1219.
|
|
ICACT20220024 Slide.08
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We select Network Service Scanning, Endpoint Denial of Service, Brute and Remote Access ,a total of four techniques and benign network flow data for the follow-up experiments. The comparison and the counts of cases are shown in this table.
|
|
ICACT20220024 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We will present two experiments and their results. One is a hacker’s attack classification model trained by CIC-IDS2017 dataset; the other is a malware classification model trained by CSTI-10 dataset. CIC-IDS2017 is an open-source dataset that records hacker attacks. CSTI-10 is a combination of different datasets that record the network traffic behaviours of malwares.
|
|
ICACT20220024 Slide.06
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
AI can make good predictions but since the process between input and output is opaque, the logic behind those predictions is hard to understand. And Explainable AI can be a solution for such problem. Here we use SHAP to help use understanding why the model makes certain decisions. This tool utilizes SHAP values that are based on Shapley values.
In this experiment, we will present three visualization methods provided by the Python SHAP package. They are bar plot, beeswarm plot, and force plot.
Bar plot simply takes the mean absolute value of the SHAP values per feature across the data. It shows the SHAP feature importance in descending order.
Beeswarm plot is also known as summary plot. It gives an overview of which features are more important for a model by plotting the SHAP values of every feature for every instance. The position on the x-axis represents the SHAP value of an instance and the colour represents the value of the feature. For each feature, instances with the same SHAP value are placed along the y-axis, this gives us a sense of distribution of SHAP value. Force plot visualizes every feature as force that either increases or decreases the prediction by pushing it from the baseline for Shapley value to the model output.
|
|
ICACT20220024 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
For the experiment, we selected XGBoost model to implement the classification of malicious behaviours. We also designed an automatic optimization process to help the model determine the most suitable algorithm structure in response to different environments. This automatic optimization process has two steps. One is hyperparameter optimization and the other is important features selection.
For the purpose of hyperparameter optimization, the grid search algorithm has been performed. Grid search trains an XGBoost model with the Cartesian product of all hyperparameter sets, and evaluates their performance. The combination of hyperparameters with best accuracy is then chosen for building the XGBoost model.
As for important features selection, we used feature importance calculated by the XGBoost model.
|
|
ICACT20220024 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We used CICFlowMeter to obtain 82 statistical network traffic features. It is an open-source network traffic flow generator which generates Biflows from pcap files, and extracts features from these flows. We also introduced CSTITool to extract 53 more features. The features were then grouped into 9 types by flow data characteristics, as illustrated on this table. The subcolumn CICFlowMeter shows the number of features generated by CICFlowMeter in each type, while the subcolumn CSTITool shows the features that were generated by CICFlowMeter along with other techniques.
|
|
ICACT20220024 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We will talk about some of the analysis methods used in our study. First is the features extraction process from packets to network flow data along with several feature engineering techniques. Then we will introduce a machine learning algorithm, XGBoost. Finally we will talk about SHAP.
|
|
ICACT20220024 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
For as long as there have been hackers, cybersecurity scientists and engineers have been proposing solutions to defend against potential attacks. Most traditional approaches use rule-based detection methods. It monitors the content of network packets in real time, and then identify information such as suspicious strings in the payload. However, if hackers transmit malicious commands in encrypted packets, traditional methods cannot withstand such attacks. Another problem is that the number of network packets increases with the complexity of the network environment. They consume enormous resources when compared with the rules.
To solve the shortcomings of traditional methods, Cisco developed Netflow. It reduces consumption of storage spaceby aggregating packet information and discarding nonessential information, such as payload, then replacing original packet information with network traffic behaviour.
With the help of Netflow data, many have trained Artificial Intelligent with NetFlow data to build models of abnormal network behaviour for detecting anomalies. By learning network traffic behaviour, AI can not only effectively identify known malicious threats, but also detect their variations.
While AI can give us answers, it is also important to know why and how AI has made a specific decision. This is where the explainable AI comes to the rescue. This paper presents an explainable flow-data AI model to identify malicious behaviours and explain how these behaviours were classified.
|
|
ICACT20220024 Slide.01
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Hello everyone. My name is Boryau Hsupeng. I am an artificial intelligence engineer from Institute of Cybersecurity Technology Institute, Institute for Information Industry. Today I will talk about an explainable AI for malware detection by using network flow features.
|
