ICACT20220038 Slide.30
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Thanks for your listening
|
|
ICACT20220038 Slide.29
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We propose a mechanism for labeling attack tactics of network intrusion detection system (NIDS) rules on the basis of text mining and machine learning, which can assist security operators in tracing hackers' behaviors and footprints. Our proposed mechanism is separated into two parts, including similarity-based tactic labeling and machine-based tactic labeling. In the similarity-based tactic labeling part, the accuracy reaches over 70% and up to 72.6%. It can assist experts in labeling tactics for numerous detection rules efficiently and accumulate the labeled dataset quickly. In the machine-based labeling part, we utilize multiple machine learning algorithms to train the dataset, which expand from the similarity-based labeling part. The experiment result shows that the accuracy of tactic labeling achieves over 90% and up to 96%. Therefore, our proposed mechanism can effectively correlate detection rules from various cyber security applications and contribute to APT attacks hunting.
|
|
ICACT20220038 Slide.28
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In the conclusion part, the contribution of this study will be included.
|
|
ICACT20220038 Slide.27
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Finally, we make the effectiveness comparison of tactic labeling between our approach and the rcATT. The experiment result described in the table shows that our approach is significantly better than the rcATT. Although the rcATT study has good TTP labeling results for long articles, their method is limited for handling data with little information. The comparison results show that our approach can still label tactics effectively even with little information.
|
|
ICACT20220038 Slide.26
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We apply four different methods, including "K-Nearest Neighbours (KNN)", "Support Vector Machine (SVM)", "Decision Tree", and "Random Forest". In the experiment, 5-fold cross-validation is used for evaluation. The experiment results show that the F1 score of tactic labeling can reach over 90% and up to 96%, which can significantly help security experts improve the effectiveness of tactic labeling. From the results, we can find that the KNN result is slightly worse because we treat keyword terms in the tactic group as features, and these features form a multi-dimension feature vector, resulting in the KNN not performing well in the data with higher dimensions.
|
|
ICACT20220038 Slide.25
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We choose "Pearson Correlation Coefficient", which obtain the best score in the experiment results of different similarity algorithms, as the primary method. We observe that the overall results of the malware-related category, such as "malware-cnc" and "malware-other", are better than the other categories. We then further analyze the reasons and discover that the categories with better results are because the texts in the "msg" field have a higher overlap with the texts in tactic articles. For example, the "download" term and "backdoor" term appeared in the malware-related category are the keyword in the "Command and Control" tactic and the "Persistence" tactic, respectively. However, the terms in the "server-webapp" category are mainly presented as the "OWASP Top 10" term for web-related attacks, with very few attack-related technique terms. Therefore, they have a lower overlap with the terms in the tactic articles, which is the main reason for the poor results.
|
|
ICACT20220038 Slide.24
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
We choose the TF-IDF algorithm as a feature weighting method for textual analysis and apply three different similarity calculation algorithms, including "Cosine Similarity", "Pearson Correlation Coefficient", and "Euclidean Distance". We can find that the experiment results of all three algorithms are similar. To further assess the effectiveness of our approach in different rule categories, we conduct separate experiments for each rule category.
Overall, the experiment results show that the F1 score of tactic labeling can reach over 70% and up to 73%, which can significantly help security experts improve the efficiency of tactic labeling. After this stage, a large number of tactic-labeled datasets are produced. Then, we use these accumulated datasets in "Machine Learning-Based."
|
|
ICACT20220038 Slide.23
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We evaluate the effectiveness of tactic labeling results by following four indicators: Accuracy, Precision, Recall, F1 score. Let's first focus on the various algebraic aspects of each formula. TP, FP, TN, FN represent true positive, false positive, true negative, and false negative, respectively. L is the number of labels. Then, we introduce four indicators. Accuracy is simply a ratio of correctly predicted observation to the total observations. Precision is the ratio of correctly predicted positive observations to the total predicted positive observations. Recall is The ratio of correctly predicted positive observations to all observations in the actual class. F1 score is the weighted average of Precision and Recall. The score takes both false positives and false negatives into account.
|
|
ICACT20220038 Slide.22
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The second dataset includes Suricata rules officially released by Snort. We select four types of rules, namely "malware_cnc", "malware_other", "server-webapp," and "top-100". The "malware_cnc" is used to detect command and control activities such as file dropping, data exfiltration, and credential theft. The "malware-other" is used to detect behaviors related to malicious programs such as remote code execution and keylogger runtime. The "server-webapp" is used to detect attacks against vulnerabilities of web applications. The “top-100” includes high-risk attack techniques, such as Cobalt Strike, Eternalblue and so on. In summary, the dataset contains four categories, 12,978 rules and 83,176 words. The statistics of the rule dataset is shown in the table.
|
|
ICACT20220038 Slide.21
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We collect two kinds of datasets; the first dataset is composed of "technique" article sets in all network-related tactics from MITRE ATT&CK framework, including 12 tactics, 137 definition articles, and 6711 words. The statistics of the articles collected for each tactic is summarized in the table. The reason for selecting only all network-related tactics is that our target of tactic labeling is NIDS rule.
|
|
ICACT20220038 Slide.20
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In this part, to evaluate the proposed mechanism, we make experiments for assessing the effectiveness of tactic labeling.
|
|
ICACT20220038 Slide.19
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
For tactic learning, we choose linear model classifiers and tree-based model classifiers and evaluate the labeling result from different Machine learning algorithms.
For tactic labeling with linear model classifiers, we choose Support Vector Machine (SVM) and K-Nearest Neighbours (KNN) as the evaluated targets. SVM classifies data through multi-dimensional hyperplane segmentation. The appropriate weights will be obtained in the training phase. KNN generally does not have an actual data training process. Instead, it calculates the distance between the test data and each training data in the prediction stage, finds the 8 adjacent data, and uses the category with the highest frequency as the classification result.
For tactic labeling with tree-based model classifiers, we use Decision Tree and Random Forest as the evaluated targets. A decision tree is a flowchart-like structure. Each internal node represents a "test" on an attribute, each branch represents the test result, and each leaf node represents a class label. The paths from the root to the leaf represent classification rules. Random forest is an ensemble learning method for classification work that operates by constructing many decision trees at training time. The output of the random forest is the class selected by most trees.
For tactic testing, the feature of testing dataset are extracts through the TF-IDF Vectorizer function, and then the tactic to which the Suricata rule belongs is predicted with the pre-trained model.
|
|
ICACT20220038 Slide.18
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
TF-IDF Vectorizer in Machine learning-based labeling includes the training phase and testing phase. In the training phase, training dataset is composed of two sources. One is MITRE textual data, the same as the one in similarity-based labeling. The other one is the labeled result from similarity-based labeling. In the testing phase, the terms in unlabeled Suricata rules will be converted into the feature vector for machine learning methods to identify tactics.
In this function, the ways for preprocessing is the same as the ways used in similarity-based labeling that is removing stopwords and the tag prefix in messages field, restoring the abbreviations and lemmatizing each vocabulary. By doing so, we reduce the influence of irrelevant noise for the prediction model.
After the preprocessing, we still use the original unigram and combine it with 2-gram to retain the semantic and contextual information.
|
|
ICACT20220038 Slide.17
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
With the help of "Similarity-based Labeling," the labeled dataset can be expanded efficiently. Next, we aim to learn the features of each tactic to improve the accuracy of labeling through machine learning algorithms. Machine learning can be effective for subsequent prediction only when the dataset is sufficient. Thus, we use the large labeling dataset extended by Similarity-based Labeling part to train models for tactic classification in Machine Learning-based Tactic Labeling part. The architecture of "Machine Learning-based Tactic Labeling" is shown in the figure. First, the labeled Suricata rule dataset is vectorized by the TF-IDF Vectorizer. Next, the attack tactic learning module trains the model with specified supervised learning algorithms. Finally, the rules that need to be labeled are vectorized into features and then predicted the tactic group with the model.
|
|
ICACT20220038 Slide.16
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
After TF-IDF Vectorizer calculates the respective feature vectors for MITRE articles of each tactic and the “msg” field in Suricata rules, we use different similarity algorithms (including Cosine Similarity, Pearson Correlation Coefficient, and Euclidean Distance) to label the tactic of the detection rule.
The cosine similarity is evaluated by calculating the angle between two vectors in space; when the angle between the two vectors is smaller, the cosine value tends to 1, reaching the maximum value of 1 at 0 degrees.
The Pearson correlation coefficient subtracts an average calculated from the vectors in all dimensions and then computes the cosine similarity. The result for the Pearson correlation coefficient has been proved to accord with the actual condition while keeping the original characteristics.
Euclidean distance is a common method in geometry to calculate the linear distance between two points.
|
|
ICACT20220038 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Tactic-based Text Grouping: Considering the information in the Suricata rules is limited, it may significantly affect the effectiveness of machine learning approaches. Therefore, we refer to the ATT&CK framework proposed by the MITRE organization for the tactics, techniques, and tools used by attackers. These definitions are now the mainstream reference for the industry, and we believe that they will be helpful for tactic labeling. We collect each technique definition (including attack scenarios) and categorize all terms contained therein into their respective tactic for subsequent information retrieval and classification.
TF-IDF Vectorizer: Before analyzing the text information, we need to pre-process the data to filter terms that may potentially affect the accuracy of the results. First, we remove any non-text characters, including punctuation marks and numbers. Besides, to prevent the English tense from affecting the result of TF-IDF, we lemmatize each vocabulary and convert it to its base form. Furthermore, we delete meaningless stop words such as 'he' (pronoun), 'a' (article). Moreover, we remove the tag prefix that often appears in the "msg" field of the Suricata rule. By doing so, we reduce the influence of irrelevant noise for the prediction model. Finally, many attacks and professional terms in information security usually consist of two or three words and are presented in abbreviations. Thus, we convert the abbreviations into complete terms at this stage. Then, we use the original unigram and combine it with 2-gram to retain the semantic and contextual information.
|
|
ICACT20220038 Slide.14
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
To accelerate the efficiency and accuracy of the labeling process, matching rules are developed by information security experts for tactic labeling. Since some specific keywords or information such as protocol, threat type, and malware family can be directly used to determine the tactic group of the Suricata rule, experts define keywords for each tactic group. For example, the "remote code execution" and "Zerologon" can be labeled as "lateral movement" tactic, and the terms related to OWASP Top 10 vulnerabilities can be labeled as "initial-access" tactic.
|
|
ICACT20220038 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In the beginning, we regard tactic labeling for Suricata rules as an information retrieval problem because of the limited dataset. We consider the description of the "msg" field in Suricata rules as the text to be queried and compare them with matching rules defined by experts in the "Keywords for Tactic Matching" function. If no matching rules can be met, the Suricata rule to be labeled will then be passed to the TF-IDF Vectorizer in the similarity-based labeling phase. Afterward, the vectorizer calculates the importance of terms in each tactic through TF-IDF algorithm. Finally, according to the similarity between the Suricata rule and each tactic group, the tactic group with the highest similarity is taken as the labeling result.
|
|
ICACT20220038 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
To label tactic, we propose an "Attack Tactic Labeling" mechanism. The mechanism mainly consists of two parts. The first part is similarity-based tactic labeling, which aims to efficiently assist experts with tactic labeling and accumulate the dataset quickly. The second part is machine learning-based tactic labeling, learning features of different tactics from the expanded dataset.
First step is "TTP Labeling by experts". Before similarity-based tactic labeling, we need the labeled dataset as the training dataset, but experts labeling makes time-consuming and labor-intensive. Therefore, the next stage "Similarity-based Tactic Labeling" labels the Tactic for numerous Suricata rules and it can expand the training dataset. The third step is "TTP Result Validation by experts". We have to ensure training dataset is reliable, so the labeled result needs to be validated in this step. The "Machine Learning-based Tactic Labeling" is the final step. We train the expanded dataset as a model by machine learning algorithms and label the subsequent detection rules by trained model. Finally, there is a feedback mechanism for continuously enhancing the accuracy of tactic labeling.
|
|
ICACT20220038 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Field information in detection rule is meaningful. Take "Suricata NIDS" as an example, we observed that threat description in message field was meaningful and those information could be related to the description of TTP in MITRE ATT&CK framework. Also, the type of attack in "classtype" field and additional information in metadata was helpful for tactic labeling because they provided the attack targets or CVE which give good reference to label the tactic of detection rules. According to the observation, we used this characteristic to label the detection rule tactic. However, we met a challenge because of few public TTP-labeled datasets for Suricata rules. The lack of labeled dataset will make tactic labeling work by Machine Learning ineffective, so we propose a mechanism for attack tactic labeling.
|
|
ICACT20220038 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In this part, we will introduce an "Attack Tactic Labeling" mechanism for further cyber threat hunting considering labeling. The mechanism consists of two parts. The first part is similarity-based tactic labeling and the other part is machine learning-based tactic labeling.
|
|
ICACT20220038 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The third study developed an automatic TTP labeling tool for cyber threat reports named rcATT. The TTP labeling tool aimed to help cyber security experts easily obtain threat actions from threat reports. They used three weighting scheme, such as TF, TF-IDF, Word2Vec, to determine the weight of each term, and Binary Relevance, Classifier Chain, Bernoulli Naive Bayes classifier, KNN for TTP labeling. Their method is limited when handling data with little information.
The fourth paper proposed a tool, called TTPDrill, which can automatically and accurately extract threat actions from unstructured text of CTI sources. They also develop automated context-aware analytics of cyber threat intelligence, which can assist experts in mapping threat actions to attack patterns (TTPs). They also used TF-IDF to determine the weight of each term, and BM25 for TTP labeling. However, TTPDrill is lack of user feedback mechanism, which makes continuous improvement more difficult.
|
|
ICACT20220038 Slide.08
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
The first paper, they adopted fuzzy clustering to correlate alerts to APT attack sequence sets. Then, they used probability transfer matrix to construct APT attack scenarios. In their experiments, they classify approximately 1000 valid alerts. And the APT attack scenarios constructed by the alerts can match the actual situation and be useful for APT attack detection. Although the experiment results seem to be good, the experiment has not been applied to a more realistic situation.
The second paper, they labeled the kill chain stage by deducing from the network direction of each IDS alert. Then, they used the finite state machine method to depict the APT scenario graphs. Approximately 446,458 IDS alerts were condensed into 700 APT scenario graphs through this method. The limitation of this paper is that their algorithm can only processes network-level information.
There are more researches also use kill chain for cyber attack detection, which shows that kill chain analysis is a common and reliable way to detect cyber attack. Nowadays, the rise in the number of attack techniques and corresponding detection rules will limit the detection coverage rate. Therefore, to rapidly expand the detection coverage, developing a mechanism for continuously labeling amounts of rules with TTP is urgent.
|
|
ICACT20220038 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Most studies have applied kill chain and ATT&CK framework to detect cyber attacks. In this part, we will provide some related works.
|
|
ICACT20220038 Slide.06
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
The primary contributions of this study include:
1. providing a mechanism to help security experts quickly and correctly label attack tactics of intrusion detection rules, making it easier to trace behaviors and footprints of attackers.
2. our approach keeps flexibility which can be further applied in various cyber security applications and different object labeling such as techniques and kill chain stages.
3. providing a solid base for further alert correlation.
|
|
ICACT20220038 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In short, TTP labeling of attack detection rules is essential for further alert correlation and cyber threat hunting.
However, with the rapid changes of attack techniques, there is no systematic and continuous labeling mechanism for the detection rules.
It would be difficult to identify new types of attacks, and there will be breakpoints in the hackers’ footprints.
Therefore, we aim to develop a TTP Labeling mechanism for cyber threat hunting and, importantly, to be systematic and continuous.
To achieve this goal, we will face a number of challenges, including
Only few public labeled rulesets, and
The information of the rules to be labeled is quite limit
This makes it difficult for machine learning algorithms to have sufficient information and datasets for learning.
|
|
ICACT20220038 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Nowadays, cyber attack tend to be multi-stage and multi-technique.
This makes traditional security defense mechanisms based on the "Indicator of Compromise" ineffective.
Besides, fail to consider attack kill chain may lead to a high false-positive rate for attack detection.
To trace hackers’ behavior and intention, it is crucial to provide additional information such as attack tactics, techniques, and procedures, so-called TTP.
|
|
ICACT20220038 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Now, start the introduction part.
|
|
ICACT20220038 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Here is today's outline, I'll first introduce the motivation, goal, challenge and contribution of our work. Next, several studies on APT hunting and TTP Labeling will be reported. After that, our proposed Tactic labeling approach will be explained in detail. And finally, the experiment result and conclusion will be presented.
|
|
ICACT20220038 Slide.01
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
Hi everybody, my name is Zong-Jyun Li. For ICACT 2022, I have cooperated with three partners from III. Mr. Sheng-Xiang Lin, Mr. Tzu-Yang Chen, and Mr. Dong-Jie Wu. Our presentation today is about a mechanism for attack tactic labeling, which aims to assist in cyber threat hunting.
|
