ICACT20240217 Slide.26
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
|
ICACT20240217 Slide.25
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
|
ICACT20240217 Slide.24
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
|
ICACT20240217 Slide.23
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
|
ICACT20240217 Slide.22
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
|
ICACT20240217 Slide.21
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
.
|
|
ICACT20240217 Slide.20
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
.
|
|
ICACT20240217 Slide.19
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
This paper introduces a Z-score-based outlier detection to neuron coverage-based TCP. The proposed method exceeds the SOTA NC-based TCP both in effectiveness and efficiency, across various experimental settings Selecting diverse samples is also important in data collection. Thus, we our future work is to extend the NC-based TCP approaches to consider sample diversity and error-inducing sample detection performance together.
|
|
ICACT20240217 Slide.18
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Now, we'll conclude our work and provide future works.
|
|
ICACT20240217 Slide.17
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Above figure shows the samples based on different priority, ordered by the proposed ZSNC. Samples with high priority tend to exhibit hard conditions such as indistinct differentiation from the background, low lighting conditions, or misalignment of objects. On the other hand, samples present easier conditions such as clear distinction between the background and the object, distinctive characteristics of prediction class, such as animal faces.
|
|
ICACT20240217 Slide.16
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Table 4 shows a comparison of total inference time of NC-based TCP techniques for seven different experiments. Consistent to the results in Table 3, NBC enables the fastest inference and ours take 2nd place. FD+ takes 2.46 times longer than the proposed method. Considering that the proposed method outperforms the other NC-based TCP methods in error-inducing sample detection performance, making it the most proper option among the NC-based TCP methods.
|
|
ICACT20240217 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
.Table 3 shows the time for pattern extraction for each neuron from a training set, in seconds. NLC does not require train set pattern extraction. The proposed method 7.19 times faster compared to the latest algorithm, FD+.
|
|
ICACT20240217 Slide.14
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
A higher ATPF value implies a more effective identification of error-inducing samples within the same labeling budget.
The proposed ZSNC takes the first place for five among seven experiments.
The comparison between NBC and ZSNC shows that employing class-specific threshold for each neuron can lead the meaningful performance enhancement based on the more detailed patterns from the training set. A comparison of FD+ with ZSNC reveals the superiority of single staged TCP and z-score-based intervals over the sampling of exact percentile samples
|
|
ICACT20240217 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We evaluate the Effectiveness and efficiency of proposed method.
For effectiveness, we used the Average Test Percentage of Fault (ATPF). ATPF is the average of the ratios of the number of erroneous instances detected within the n-th labeling budget.
For an efficiency, we measure the time required for the pattern extraction from and the total inference time.
|
|
ICACT20240217 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We compared three different NC-basaed method with the proposed ZSNC.
|
|
ICACT20240217 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
we evaluate our method with three datasets for image classification benchmark. Also, we employ real-world corruptions to CIFAR10 to evaluate the TCPs in real-world scenarios.
Different sizes of models are adopted.
Table 1 represents details about our evaluation setups. The experiment IDs F to H represents CIFAR10 with real-world corruptions, while the trained model is same with the experiment IDs B to D.
|
|
ICACT20240217 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
.
|
|
ICACT20240217 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
during inference, we observes the ratio of neurons that do not fall within an interval for a given class.
|
|
ICACT20240217 Slide.08
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The proposed method is called Z-score based NC, ZSNC. It observes the ratio of neurons that do not fall within an interval for a given class. The class-specific bounds for each neuron consists of mean and standard deviation of class-specific features in train set, which can be calculated iteratively and scaled by epsilon that is selected from z-score table. To fit the range with FD+, we select 1.96 as a scaling factor. The proposed method does not need to save all output values from training set.
|
|
ICACT20240217 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We have questioned about the pattern extraction of FD+.
Indeed, we observed that for each classes, the neuron output for false prediction deviates far from that of correct predictions.
Also, instead of exact percentile value extraction, we adopt z-score approximation as the value range for each classes as many anomaly detection researches did. one defines the anomalous sample as the one does not fall within the z-score based interval,
|
|
ICACT20240217 Slide.06
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
However, FD+ requires a higher cost than other NC-based methods. Both active status classification and percentile value extraction requires to save outputs of all neurons for all training instances. This slows down the pattern extraction process of FD+.
|
|
ICACT20240217 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Here are some examples of NC and its variants. brackets above each neuron in NBC represents min-max range of each neuron output values
|
|
ICACT20240217 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
After then, variants of NC are comes out.
Neural Boundary Coverage extract the minimum and maximum values of the total neuron outputs upon the training set. Then, when the test sample is given, they observe the number of neurons that does not lies between min-max range.
Familiarity Degree is a state-of-the-art.
It extracts class-specific patterns for each neuron from a training set. For each class, the patterns contains whether neuron 𝑛 is “active” or “inactive” for class c. If a neuron is “active”, then the 5% percentile value of the neuron for the train set is extracted, and 95% percentile value is extracted for inactive neurons.
The classification of "active" neuron is that if the 95% of the collected neuron output for the given class are greater than zero, otherwise "inactive".
|
|
ICACT20240217 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Inspired by the traditional software testing,
Neuron Coverage was proposed in 2018.
Given a test input, it calculates the ratio of the number of neurons whose output values exceed threshold to the total number of neurons.
The authors of origin NC assumed that a sample with a higher NC value is more difficult compared to the lower one.
The assumption stems from the fact that a higher NC-valued sample requires the greater number of neuron explorations.
|
|
ICACT20240217 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Now, I introduce some related works and proposed method
|
|
ICACT20240217 Slide.01
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
With the exponential growth of deep learning, TCP for DNN is now has an important role.
The goal of TCP is to identify and prioritize error-inducing instances in an unlabeled test set in limited annotation budget.
Based on it, the bug report of trained model can be made and collected data based on the report can improve the model by additional training.
|
|
ICACT20240217 Slide.00
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Hi everybody, my name is Hyekyoung Hwang.
Our presentation today is about to improve the effectiveness of existing Test case prioritization technique based on z-score approximation
|
![[Big slide for presentation]](/upload/2024/0217/Presentation/26/27.jpg){kind=link}
![[Big slide for presentation]](/upload/2024/0217/Presentation/25/27.jpg){kind=link}
![[Big slide for presentation]](/upload/2024/0217/Presentation/24/27.jpg){kind=link}
![[Big slide for presentation]](/upload/2024/0217/Presentation/23/27.jpg){kind=link}
![[Big slide for presentation]](/upload/2024/0217/Presentation/22/27.jpg){kind=link}