 ICACT20220050 Slide.01
[Big Slide]
[YouTube] |
Chrome  Click!! |
 |
Hello everyone,
I am Zakaria Sawadogo, PhD student in Gaston Berger University under the PASET scholarship
I am happy to present our work about
Android malware detection : Investigating the impact of imbalanced datasets on the performance of machine learning models.
|
| ICACT20220050 Slide.02
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
The outline of the presentation will be as follow
We start with the Context and the Problem.
Then some related works and our Contributions.
After that we present our approach to the investigation followed by the experimental process
next, we present the results
and lastly we conclude
|
| ICACT20220050 Slide.03
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Detection or classification of malicious applications android uses machine learning.
According to Cho et al. these techniques require a large amount of datasets and are balanced between classes , for model training, otherwise the small size of the dataset or the imbalanced datasets affects the effective model training in machine learning.
We are not aware of any work analyzing the impact of imbalanced datasets on machine learning models for detecting malware android.
|
| ICACT20220050 Slide.04
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Eitrich and Lang ,2006[2] have proposed an approach to efficiently optimize machine learning parameters of support vectors for unbalanced data sets.
Pozzolo et al. 2015[3] uses Bayes minimum risk theory to find the correct classification threshold and show how to adjust it after downsampling. From imbalanced data sets, the objective functions used to learn classifiers generally tend to favor larger and less important classes.
Amalia et al,2019[4] perform an impact analysis on the performance of unbalanced datasets
but focusing on the effects of data resampling.
Johnson and Khoshgoftaar 2019 [5]did a survey that highlights various gaps in deep learning from imbalanced data.
We are not aware of any work analyzing the impact of imbalanced datasets on machine learning models for detecting malicious android applications. That justifies our contribution.
|
| ICACT20220050 Slide.05
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
We have investigated the impact that imbalanced datasets can have on the performance of models produced by classification algorithms.
we make the following contributions:
The sensitivity status of eleven classification algorithms for the detection of android malware Applications with an imbalanced dataset.
The analysis of the most suitable evaluation metrics for imbalanced dataset
|
| ICACT20220050 Slide.06
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
1. We have selected papers from the literature that address the detection or classification of malicious android applications.
2. For each paper, we looked at :
the size of the dataset used
whether the dataset is unbalanced or not
the metrics used to evaluate the model.
3. We constructed two datasets (balanced and unbalanced) to assess the impact on the models
|
| ICACT20220050 Slide.07
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Figure 1, shows us that more than 77% of the papers use unbalanced data sets.
Figure 2, shows us that the size of the data is relatively small.
|
| ICACT20220050 Slide.08
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In figure 3, we have the proportions of use of metrics. Accuracy is the most used metric with 68%. |
| ICACT20220050 Slide.09
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
We used an imbalanced and balanced multi-class dataset.
We selected eleven classification algorithms for our experiment.
|
| ICACT20220050 Slide.10
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
We have considered two types of metrics for the evaluation of the models produced by the different algorithms:
The standard metrics: Accuracy, Precision, Recall and F 1-Score.
Balanced metrics: Matthews correlation coefficient (MCC), Balanced accuracy (BA), Geometric Mean (GM) and Fowlkes-Mallows (FM)
|
| ICACT20220050 Slide.11
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Looking at Figure 4 and Figure 5, we can see that the performance of the metrics decreases with the use of balanced data.
This decrease in performance is more important for the accuracy metric.
|
| ICACT20220050 Slide.12
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Figure 6 and 7 show us the performance obtained with the balanced metrics. We see a decrease in performance with the use of balanced data, but this is less than with the standard metrics in Figures 4 and 5. |
| ICACT20220050 Slide.13
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
We have also shown that the metrics Commonly used in the literature are not always suitable for imbalanced data.
In addition, the accuracy values with balanced datasets are approximate to those obtained with balanced accuracy for imbalanced datasets.
We show that inappropriate use of evaluation metrics and imbalanced data sets results in flattering performance
|
| ICACT20220050 Slide.14
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
In this paper, we evaluated eleven classification algorithms based on the performance obtained by the commonly used metrics and by the balancing metric.
This study allows us to assert, as shown by the results, that some algorithms are less suitable for imbalanced datasets. We have also shown that the metrics commonly used in the literature are not always suitable. Metrics such as balanced accuracy and Geometric mean are more suitable for imbalanced dataset
|
| ICACT20220050 Slide.15
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
We thank PASET-RSIF for their financial support of our work |
| ICACT20220050 Slide.16
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
We present an excerpt from our literature review |
| ICACT20220050 Slide.17
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
Thank you for your attention |
