ICACT20230030 Slide.29
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Thanks for your attention! Now! Floor is open for QnA. Welcome any comment or question.
|
|
ICACT20230030 Slide.28
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
.
|
|
ICACT20230030 Slide.27
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Here are our references.
|
|
ICACT20230030 Slide.26
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
This paper summarises and describes the advantages of the linear leakage method, in addition to its significant disadvantages. First, the linear leakage dynamical model bionicity is between IF and LIF models, and it can be regarded as a simplified form of LIF, but cannot meet the original intention of SNN to effectively simulate human brain neurons. Secondly, although linear leakage is robust, the error convergence rate and final accuracy are slightly inferior to those of LIF model, which are the problems of linear leakage that need to be solved urgently. Therefore, the subsequent research direction of us is to design experiments to find the balance between robustness and accuracy based on the current work, and to actively explore the influence of the inherent parameters of LIF neurons on the robustness of the model.
|
|
ICACT20230030 Slide.25
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The low power consumption and high robustness of SNNs make them a potential replacement for ANNs in noisy input scenarios. In this paper, we propose a method for using linear leakage in neuronal models of directly trained SNNs based on in-depth research in related experiments and articles, and develops a series of experiments around this method. Our answers to the questions raised in the introductory section are given here as follows.
1) By changing the leakage of LIF neurons, the linear leakage has better robustness compared to the no-leakage (IF) and conventional exponential leakage (LIF) models.
2) We demonstrate that the spiking rate decreases and the sparsity of the network increases when the leakage increases through deriving the process of integrating and firing spikes of LIF neurons based on linear leakage. In addition, we observe that the effect of the linear leakage parameter on the pulse firing rate somewhat affects the classification accuracy of the network.
3) It is experimentally demonstrated that linear leakage enhances the robustness of SNN absolute accuracy by 25% compared to conventional exponential leakage under specific parameters. Also, this robustness is generalizable across different datasets and network models.
|
|
ICACT20230030 Slide.24
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
CONCLUSIONS
|
|
ICACT20230030 Slide.23
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We compare the performance of the ResNet-19 model accuracy under the influence of Gaussian and impulse noise for the no-leakage (IF), exponential leakage (LIF) and linear leakage methods (see Figure 12 and Figure 13). We find that linear leakage maintains the best robustness in the case of complex datasets and deep networks compared to other leakage methods. The difference in absolute accuracy between linear and exponential leakage reaches a maximum of 17% at mark ③ when the Gaussian noise amplitude parameter is 20.
|
|
ICACT20230030 Slide.22
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We compare the performance of the ResNet-19 model accuracy under the influence of Gaussian and impulse noise for the no-leakage (IF), exponential leakage (LIF) and linear leakage methods (see Figure 12 and Figure 13). We find that linear leakage maintains the best robustness in the case of complex datasets and deep networks compared to other leakage methods. The difference in absolute accuracy between linear and exponential leakage reaches a maximum of 17% at mark ③ when the Gaussian noise amplitude parameter is 20.
|
|
ICACT20230030 Slide.21
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We compare the performance of the CifarNet model accuracy under the influence of Gaussian and impulse noise for the no-leakage (IF), exponential leakage (LIF) and linear leakage methods (see Figure 10 and Figure 11). At mark ②, the absolute accuracy difference between linear and exponential leakage reaches a maximum of 20% when the Gaussian noise amplitude parameter is 18.
|
|
ICACT20230030 Slide.20
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Figure 9 shows the comparison of the feature maps extracted by the network under different leakage methods. From the figure, it can be seen that the feature maps extracted after linear leakage are less noisy compared with those of the other two leakage methods.
|
|
ICACT20230030 Slide.19
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We compare the performance of the MNISTNet model accuracy under the influence of Gaussian and impulse noise for the no-leakage (IF), exponential leakage (LIF) and linear leakage methods (as shown in Figure 7 and Figure 8). The experimental results show that the linear leakage is the best in terms of robustness under different noise influences, and in particular, the linear leakage shows extremely high robustness under small noise conditions. The absolute accuracy difference between linear leakage and exponential leakage reaches a maximum of 25% at mark ① when the impulse noise amplitude is 0.2. We visualize the feature map of the MNIST dataset extracted by the network at marker ①.
|
|
ICACT20230030 Slide.18
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
To ensure that the linear leakage approach is robust, we test on the MNIST dataset first. The STBP framework is used in this experiment, and the first convolutional layer is used as the coding layer of the network. The structure of MNISTNet network model we use is represented as (28*28-15C5-2P-40C5-2P-300FC-10FC). Where the input is a 28*28 grayscale map, C5 represents a convolutional layer with a convolutional kernel size of 5*5, 2P represents a mean pooling layer with a kernel of 2*2, and FC represents a fully connected layer. A spiking layer is added to spike the float values after each layer.
We conduct the experiments on the Cifar10 dataset to demonstrate the generalization of linear leakage robustness.The structure of CifarNet model can be shown as (32*32*3-128C3-256C3-2P-512C3-2P-1024C3-512C3-32768FC-100FC-10P). Where the input is a 32*32 color map, C3 represents a convolutional layer with a convolutional kernel size of 3*3, 10P represents a one-dimensional average pooling layer with a kernel of 10*10 and a padding of 10 and FC represents a fully connected layer. Each convolutional layer is followed by the tdBN layer [2]. After each network layer, a spiking layer is added to spike the float values.
The structure of ResNet-19 model can be shown as [32*32*3-128C3-(128C3-128C3)*3-(256C3-256C3)*3-(512C3-512C3)*2-1P -256FC-10FC]. Where the input is a 32*32 color map, C3 represents a convolutional layer with a convolutional kernel size of 3*3, the brackets represent a residual block, 1P represents a two-dimensional average pooling layer with a kernel of 1*1 and a padding of 1, and FC represents a fully-connected layer. An spiking layer is added after the last fully connected layer.
In this experiment, we uniformly used the parameters: Timestep = 2, 𝑈_𝑡ℎ =0.2, 𝑈_𝑚𝑒𝑚 =0.9. Table 1 shows the specific leaky factors of different leaky methods for different datasets. In order to prevent the phenomenon of overfitting of different leaky methods trained in the model, in this paper, the network model weights of each leakage method are trained to the same accuracy and then the experimental data are saved and tested with a noisy dataset. Among them, the accuracy of MNISTNet is 99%, CifarNet is 80%, and ReaNet-19 is 85%.
|
|
ICACT20230030 Slide.17
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In this paper, we use a clean dataset for training first and then add noise to test the robustness of the model. The reason for not adding noise to the train dataset are as follows: on the one hand, the robustness of SNN models trained with distinct noise and leaky factors is different, and it is difficult to distinguish between the robustness brought by the linear leakage method and the robustness brought by training with a noise-enhanced dataset. On the other hand, robustness generated under the same noise training does not generalise, and we need to obtain robustness of SNN models with generalisation capability, i.e. models that are robust under different noises. The similar method was used by [5,13]. Figure 6 shows the experimental procedure.
|
|
ICACT20230030 Slide.16
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
There are two common approaches to train SNN:
Pre-training ANN Model and Converting to SNN: this approach[10-12] usually requires a relatively larger time step, which reduces information loss while also introducing higher latency and energy consumption, while this method ignores the SNN time domain information, which is contrary to the original purpose of SNN.
Direct Training of SNN: this approach can be divided into unsupervised training and supervised training. The former uses the Spike Time Dependent Plasticity (STDP) rule [7], this method is often limited to shallow structures and cannot be applied to complex datasets [3]. The latter uses the gradient substitution method to solve the problem of unsteady SNN gradients.Wu [1] et al. proposed the STBP method and applied it to deep SNNs [2,9], smaller time steps are used while taking into account temporal and spatial information, thus achieving higher accuracy rates in large data sets. In this experiment, we use the STBP framework as baseline.
|
|
ICACT20230030 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
EXPERIMENTS
|
|
ICACT20230030 Slide.14
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Figure 4 is a function plot of the 𝑅_1, 𝑅_2 and spiking rate 1/T that we have drawn. Under different 𝑅_1, the spiking rate 1/𝑇 shows a decreasing trend when 𝑅_2 varying from 0.05 to 0.25, which means, the spiking rate decreases and the sparsity of the network increases as the leakage increases. In addition, we find that the variation of the linear leakage parameter affects the classification accuracy of the network to some extent. Figure 5 shows the variation of accuracy with different network models when the size of the linear leaky factor is varied.
|
|
ICACT20230030 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
This paper explores the intrinsic relationship between linear leakage and pulse emissivity through the derivation of equations. As shown in Figure 2 and Figure 3, assume that a spike signal 𝑈_𝑚𝑒𝑚 is received at each time step in a single neuron, the membrane voltage 𝑈_([𝑡]) integrates each spike signal and linear decays by the leaky factor 𝜏_𝑚 at each time step. Linear leakage implies that the membrane voltage 𝑈_([𝑡]) decays a fixed value at each time step. In order to understand the leakage method more intuitively, Figure 3 shows the change trend of the LIF neuron membrane voltage signal under different leakage modes when no other signal is received. In terms of bionicity, linear leakage is somewhere between exponential leakage and no leakage.
At the time step 𝑇, the integrated membrane voltage 𝑈_([𝑡]) is equal to the threshold voltage 𝑈_𝑡ℎ. This paper uses the variable 𝑅_1 represents the rate of the threshold voltage 𝑈_𝑡ℎ and the spike signal 𝑈_𝑚𝑒𝑚, and the variable 𝑅_2 represents the rate of the leaky factor 𝜏_𝑚 and spike signal 𝑈_𝑚𝑒𝑚, then we can obtain the relationship between the spiking rate (1/𝑇), 𝑅_1 and 𝑅_2, which is shown in Equation (5).
|
|
ICACT20230030 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We present a linear leakage method that is more similar to the one used on neuromorphic chips. Starting from the second time step, the linear leakage quantitatively reduces the membrane voltage at each time step. We have two reasons for proposing linear leakage method. First, the leakage term uses subtraction compared to multiplication requiring less computation, which is in line with the low-power characteristics of SNNs. Second, the use of subtraction for the leakage term is more effective in offsetting noise than multiplication, which can bring better robustness to the SNN. Overall, we propose linear leakage in order to optimize the conventional exponential leakage to obtain lower power consumption and higher robustness. Algorithm 1 explains the LIF neuron state update step base on linear leakage.
|
|
ICACT20230030 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The event-based approach to information propagation of SNNs allows for low power consumption, and as a result, SNNs are receiving industry attention and neuromorphic architecture hardware such as IBM TrueNorth and Intel Loihi are being developed to accommodate itself. This paper proposes a linear leakage method inspired by the method used by Y. Kuang [18] and others in the development of neuromorphic chips. During the operation of the neuromorphic chip PKUNC64C, the voltage leakage within each time step is represented by subtracting a fixed value to accommodate the low power consumption of SNN. Equation (4) represents the way the chip updates the voltage at the hardware level.
Where 𝑈_𝑗 [𝑡] denotes the membrane voltage of the time step 𝑡 in the layer 𝑗, 𝑥_𝑖 is the spike signal from the presynaptic of the layer 𝑖, 𝑤_𝑖𝑗 is the connection weight of layer 𝑖 and layer 𝑗, 𝑠_𝑖𝑗 is the synaptic connectivity between layer 𝑖 and layer 𝑗, and 𝜆_𝑗 is the leakage constant of the layer 𝑗. This method reduces the chip computing power and improves the computational robustness.
|
|
ICACT20230030 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Linear Leakage
This section details the inspiration, advantages and theoretical support of linear leakage, and also demonstrates the feasibility of linear leakage.
|
|
ICACT20230030 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Equation (1) is the first-order differential equation of the LIF model, where 𝜏_𝑚=𝑅_𝑚 𝐶_𝑚 is the membrane time constant, 𝑈 is the membrane voltage, 𝐼 is the sum of the firing currents of each presynaptic neuron, 𝑈_𝑟𝑒𝑠𝑒𝑡 is the reset voltage, and 𝑈_𝑟𝑒𝑠𝑡 is the resting voltage. Figure 1(a) and Figure 1(b) shows the process of neuronal spike sequence input and firing.
We refer to the conventional leakage process as exponential leakage because the membrane voltage leakage term exists in an exponential form. The state update process of the conventional LIF model can be expressed as Equations (2) and (3), where 𝑈[𝑡] denotes the membrane voltage at the current moment, 𝑈[𝑡−1] represents the membrane voltage at the previous moment, 𝑒^((−1)/𝜏_𝑚 ) is the membrane voltage leakage term, 𝑂[𝑡] represents the pulse emission function and W is the weight. At the membrane potential 𝑈 does not reach the threshold voltage 𝑈_𝑡ℎ the spike signal accumulates in the time domain and decays according to the membrane time constant 𝜏_𝑚 with time step; once the membrane potential 𝑈 reaches the threshold voltage 𝑈_𝑡ℎ, the neuron immediately fires a spike signal and resets to 𝑈_𝑟𝑒𝑠𝑒𝑡.
|
|
ICACT20230030 Slide.08
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Spiking neural networks, as the third generation of neural networks, have lower power binary inputs and more bionic neuron models. Among many models, the LIF model visually describes the relationship between the membrane potential 𝑈 and the input current 𝐼, and its equivalent circuit is shown in Figure 1(c). In the equivalent circuit, the resistance 𝑅_𝑚 and capacitance 𝐶_𝑚 are connected in parallel. When the presynaptic neuron generates a spike, a current 𝐼 is generated on the prominence connected to the postsynaptic neuron that partly charging the capacitor 𝐶_𝑚 and partly leaking on the resistor 𝑅_𝑚.
|
|
ICACT20230030 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
SNN FOUNDATION
|
|
ICACT20230030 Slide.06
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
For the above problem, this paper conducts a series of SNN robustness experiments around linear leakage. To the best of our knowledge, this paper is the first to propose the use of linear leakage on LIF neurons with directly trained SNNs (explained in Section 4.1).The key contributions can be summarized as follows:
We propose a method that uses linear leakage on LIF neurons of direct training of SNN.
We analyze the process of integrating and firing spikes based on linear leakage of LIF neurons, and demonstrates the inverse relationship between the leakage parameters τ_m and spiking rate (1/T) in linear leakage.
In this paper, a linear leakage method is applied on three different network models based on MNIST and Cifar10 datasets. After experimental comparison and analysis, we found that the linear leakage method in this paper has stronger robustness compared to the no-leakage and exponential leakage approaches.
|
|
ICACT20230030 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
To address the existing problem, we want to find a way of leakage to make the network perform well in robustness based on direct training of SNNs. The experiments and analysis in this paper revolve around the following three key questions, which are summarised in turn in the conclusion section.
Can the original robustness of the SNN be improved if the leakage method is changed (linear leakage)?
Assuming that question 1) is valid, what role does linear leakage play in the LIF neuron model in the SNN?
Assuming that question 1) is valid, how much robustness is improved by linear leakage? Is this robustness generalisable across different datasets and networks?
|
|
ICACT20230030 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
As research progresses, attention has been paid to the effects of the LIF neuron model and the leaky factor on SNN robustness. Among them, the LIF model is the most common SNN neuron model and the leaky factor is the signature parameter of LIF neurons. 2020, S. Sharmin [4] et al. demonstrated the potential of LIF neurons to be noise-resistant, and they pointed out that the leakage parameter is an additional knob to control the noise resistance of the model based on the effect of the inherent structural parameters of LIF neurons on the accuracy of the model. In 2021, S. S. Chowdhury [5] et al. investigated the role played by leakage in SNNs and concluded from frequency domain analysis that leakage can filter high frequency components thus improving model robustness. However, this field still needs a lot of theoretical and experimental support to explore highly robust SNNs and optimize neuronal models.
|
|
ICACT20230030 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In 2015, J. Yang [17] et al. demonstrated that the classification capability of SNNs is robust to sinusoidal and Gaussian noise of the input signal. In 2019, it has been demonstrated that SNNs are more resilient than ANNs under black-box attacks [19]. In 2021, R. El-Allami et al. [6] investigated the effect of different structural parameters (e.g. threshold voltage, time step) on the robustness of SNN models under white-box attacks based on previous experiments, and the experimental results showed that SNNs improved robustness by 85% over ANNs for specific parameters. The SNN is thus shown to have the potential to replace ANNs in noisy inputs scenarios.
|
|
ICACT20230030 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In recent years, the mature ANN has been used in many important areas such as finance, security and healthcare, however, the large amount of computational resources consumed makes it computationally expensive for ANN to solve simple problems. In addition, misclassification brought by invisible noise perturbation during ANN training will lead to irreparable consequences. A large number of robust ANN-based experiments [14-16] have emerged to reduce the effects of noise, but they are all accompanied by cumbersome experimental steps, so there is an urgent need for a simple and effective highly robust scheme to emerge as an alternative to ANN in noise input.
Meanwhile, a new generation of neural network, SNN, has been explored to simulate the function of human brain to achieve complex cognitive tasks under low power consumption, with the advantageous features of simulating biological neuron mechanism and propagating spatio-temporal information by spike as signal.
|
|
ICACT20230030 Slide.01
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
INTRODUCTION
|
|
ICACT20230030 Slide.00
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Hi everybody, my name is Jinming Che. Our team members are mainly from the School of Software and Microelectronics, Peking University. I am very glad to have this opportunity to share the work of our team. The main content of this presentation is about the innovative leakage method of LIF neurons in pulsed neural networks, which is what we call linear leakage.
|
