 |
We applied feature selection to the CSTI-10 dataset and created two variations, CSTI-10-A and CSTI-10-B. For CSTI-10-A, we tested features obtained from CICFlowMeter and 68 were identified as important features. As for CSTI-10-B, a total of 377 features were obtained from CSTITool. A part of the increase in number of features (from 79 to 377) was contributed by One-Hot Encoding. After the feature selection process, only 142 features remain in CSTI-10-B.
We trained two malware classification models based on CSTI-10-A and CSTI-10-B respectively. The performance metrics is illustrated in this fifure. Both flow-data based models performed well in most classes with Precision and Recall higher than 90%. The CSTI-10-B model is expected to achieve better performance as it was built with more features. The extra features in CSTI-10-B help improve the performance metrics of most categories; for instance, Rig¡¯s Precision and Recall are increased by 1.23% and 1.59% respectively, and Recall value of Emotet is increased by 1.15%. Although the Recall value of Nuclear is reduced by 2.7% to 97.3%, the Precision and Recall value of Macro in CSTI-10-B model are higher than CSTI-10-A by 0.1 to 0.2%.
We investigated into three categories with the least accuracy in performance, which are Emotet, TrickBot, and Rig. 71 out of 1743 Emotet were misclassified as TrickBot, while 102 out of 5039 TrickBot were labelled as Emotet. We found that Emotet is often used together with TrickBot, causing the model to learn TrickBot¡¯s behaviours and is responsible for the misclassification. Having only 186 cases, Rig's Recall is also low. It is because data insufficiency will not only make the model unable to learn the characteristics of this type of malware, but also reduce the performance metrics with a small amount of misclassification. In this experiment, there were only 189 Rig cases, 25 of which were incorrectly classified, resulting in only 86.24% of Recall value.
|
IEEE/ICACT20220024 Slide.10
[Big Slide]