 ICACT20220035 Slide.01
[Big Slide]
[YouTube] |
Chrome  Click!! |
 |
Hello everyone, my name is Nishat I Mowla from RISE Research Institutes of Sweden. Our paper is part of the Cyber Resilience for Vehicles project at RISE in close collaboration with academia and industry (i.e., Volvo for this project). Our presentation today is titled Dynamic Voting based Explainable Intrusion Detection System for In-vehicle Network
|
| ICACT20220035 Slide.02
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
By 2025, 60% of all vehicles will be internet-connected. A modern vehicle also contains a large number of ECU or electronic control units that communicate over an in-vehicle network. However, this increased connectivity and complexity is also increasing the security risks for the vehicles. It has been shown that attackers could exploit vulnerabilities in the infotainment systems and the communication interfaces to access the internal network and inject messages to cause flooding, fuzzy attacks, and malfunction of the vehicle.
|
| ICACT20220035 Slide.03
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
To solve this problem of attacks on the in-vehicle network, intrusion detection systems have been proposed to detect active attacks. Many intrusion detection solutions that utilize machine learning have been designed in the research community over the past years. However, a large portion of academic research in this field has not been converted to practical applications. One important reason may be that the decisions made by these systems are hard to interpret and understand. Existing IDS solutions don’t answer the questions to why the IDS chose a certain decision, when it succeeds and when it fails, and how reliable the decision is. Therefore, we propose an automotive intrusion detection system that uses a dynamic voting technique to provide interpretability and explainability through feature and sub-model exploration. In particular, a set of random forest models are leveraged to find a robust solution to address the challenge.
|
| ICACT20220035 Slide.04
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In particular, we propose a reliable intrusion detection system solution for the Controller Area Network using the concept of explanation as exploration for addressing explainable AI goals. For doing so, we performed extensive experimental analysis on two publicly available CAN bus datasets to evaluate and validate the performance of the proposed solutions. We also evaluate the explainability of the proposed solution and further use it to show that reasonable performance can be achieved with a subset of the available features that will also allow reducing the average training time.
|
| ICACT20220035 Slide.05
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In general, there are two classes of intrusion detection systems. Rule-based solutions create a set of normal and attack data signatures and normally require expert knowledge to create the database, which must also be constantly updated as new attacks are discovered or as new devices or functionality is added to the vehicle. On the other hand, anomaly-based solutions mathematically model normal behavior and use deviations from it as anomalies. Anomaly-based solutions rely mainly on the availability of data, which possibly makes them more suitable for use in automotive solutions.
Automotive intrusion detection systems use the data related to the message payload, timing information and physical attributes to attack attacks.
In recent years, machine learning-based automotive IDS solutions have been proposed which have explored a plethora of ML algorithms such as neural networks, recurrent neural networks, reconstruction networks, one-class support vector machines and hidden markov models. Random Forest using an ensemble of decision trees has shown very promising results as they are less sensitive to some common machine learning problems such as overfitting and generally performs well in intrusion detection tasks. Performance can sometimes be improved by increasing forest size and complexity, although this comes with an increased computation cost. It may be possible to find the optimum parameters by using a hyperparameter optimization method, such as grid search.
While the explainablity of machine learning-based decisions has been explored in the fields of computer vision and natural language processing, not much work has been done towards explaining decisions made by ML-based intrusion detection systems. In the field of security, Wang et al. proposed a framework to explain why certain decisions are made and what features they use for intrusion detection. In some research work, the explainable AI concept is leveraged to enhance trust management by exploring the decision tree model for intrusion detection. Some recent work focuses on explainable AI to make intrusion detection decisions more interpretable while most of the previous works on model interpretation focuses on other fields like computer vision, natural language processing, and biology. The lack of research in this direction makes it difficult for cybersecurity experts to optimize their decisions according to the judgments of the model in practice.
|
| ICACT20220035 Slide.06
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
The proposed intrusion detection model has three main parts. The data pre-processing part uses a data aggregation technique based on a specific window size to extract a set of time and frequency-based features from the CAN bus data. To aid in the feature exploration, the feature sets are organized as different sizes of feature sets where a univariate feature selection mechanism is followed. This generates selected and ranked features with tailored intervals dividing the total number of features into four sections. The univariate feature selection expedites the selection of the most important features based on univariate statistical tests. As a result, the univariate feature selection enables strong statistical tests that can be used to select those features that are most important and have the strongest relationship with the output variable. The model exploration is done through setting up three control models and evaluating the final model’s decision based on the contribution of each of the three control models. The contributions from each control model are represented in the form of a vote to the final model. This supports the training of the binary classification problem with m number of models. And finally, the model testing is performed which outputs the result as either an attack instance or normal instance.
|
| ICACT20220035 Slide.07
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Algorithm 1 shows the detailed working on the dynamic voting mechanism with feature and model explorations. In the first parts, parameters and weights are initialized. Next, for each model and feature set pair, the performance is measured which is used to score each participating sub-model. The scores are then used to assign weights dynamically based on their current performance. Finally, the participating models participate in a soft voting with their achieved weights to predict the final outcome. More detail of this algorithm can be found in our paper.
|
| ICACT20220035 Slide.08
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
The survival analysis dataset includes CAN bus data from three different vehicles. Normal driving data and three different injection attacks for each vehicle are included in this dataset. The attack packets were injected for five seconds every 20 seconds for the three attack scenarios where the attacks do have a real effect on the vehicle they are injected to. The flooding, fuzzy, malfunction attack scenarios were included. A disadvantage of this dataset is that it has a fewer number of attacks to evaluate.
The Oak Ridge National Laboratory dataset contains advanced and real attacks. The dataset contains anonymized CAN data recorded during ambient driving and during several network attacks. A variety of physically verified CAN attacks such as fuzzing fabrication attack, many targeted fabrication attacks, and masquerade attacks are included in this dataset. The ORNL datasets attacks are made stealthy by manipulating only the necessary portions of the data field and sending a single manipulated message per ambient message of the same ID.
|
| ICACT20220035 Slide.09
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
A total of 25784 and 27997 instances from the survival analysis dataset and ORNL dataset were used for experimentation. A 60 to 40 and 50 to 50 of training to test data split was used for the survival analysis dataset and ORNL dataset, respectively. Tables I and II show the model training performance with different feature sets and 10-fold cross-validation. We evaluated the training performance in terms of their F1 score where we can see that our proposed model VGRF performs quite well compared to the other models and submodels used for the experimentation. In the survival analysis dataset, the proposed VGRF model outperforms all the other state-of-the-art models in one out of four tests. In the case of the ORNL dataset, the proposed VGRF model outperforms all the other datasets in three of the tests.
|
| ICACT20220035 Slide.10
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Tables III and IV show the model testing performance evaluated with accuracy, precision, recall, and F1-score. We select the feature set with the highest F1-score from the training to evaluate the model for testing. The SAD dataset outperforms all four tests while the highest scores in all the four metrics are achieved by the VRF model (i.e., model without one of the control models of grid search) in ORNL which is, however, close to the performance of the VGRF model.
|
| ICACT20220035 Slide.11
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
As shown in Table V, the most contributing features in descending order were ID (ID of the 10 frames), dt(time difference
between timestamps of 10 frames), dt_ID(the time difference between each individual CAN ID of 10 frames), win_most_freq_id (number of occurrences of the most frequent CAN ID per window), win_dur (time duration from first frame (t set to 0) to the last frame in the window), win_mean_dt (mean time difference between the CAN-frames in the sliding window). The same order of importance was also observed by the LIME feature values. Among the control models, the models are affected by the selected feature set which also affects their individual performances. For VGRF to give its best performance, the most contributing control model in both the SAD and ORNL dataset was RF max and its dynamic weights were the highest. RF min was the least contributing control model and its dynamic weight was the least which however helped VGRF achieve its highest performance.
|
| ICACT20220035 Slide.12
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
The average training time increases with the increasing number of features in both the datasets as expected. However, the training performance in terms of F1-score is not always increasing proportionately with an increasing number of features. In the case of our proposed model VGRF, with the SAD dataset, the highest F1-score is achieved at 113 features which have an average training time of 110s which is closer to the training time measured at 90 features (i.e., 95.03s). The ORNL dataset achieves the highest F1-score with 113 features at an average training time of 202:95s which is closer to the F1-score achieved with 90 features with an average training time of 142.51s. |
| ICACT20220035 Slide.13
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In conclusion, we have proposed a reliable and interpretable model for an in-vehicle network intrusion detection system. The proposed IDS model is suitable for detecting attacks in CAN bus with a unique frequency-based approach. We have proposed an interpretable two-class attack detection model that leverages a dynamic voting-based exploration of features and control models. Moreover, our proposed approach can generalize to different datasets evaluated in terms of standard performance metrics such as F1 score and time comparison. In the future, we will implement the proposed approach to other in-vehicle network datasets and explainability metrics.
|
| ICACT20220035 Slide.14
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
This is our reference page 1.
|
| ICACT20220035 Slide.15
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
This is our reference page 2.
|
| ICACT20220035 Slide.16
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
This is our reference page 3
|
| ICACT20220035 Slide.17
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Thank you for listening to our presentation.
|
