ICACT20210159 Slide.17
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
A hybrid recognition method based on convolution neural network and hmm-gmm proposed in this paper can greatly improve the accuracy of recognition in actual situations, even in the case of poor environment. In this paper, when recognizing gestures, we do not directly use pictures to recognize gestures, but recognize human joints in pictures to calculate the direction of gestures, and only recognize isolated words in their speech recognition. It is difficult to recognize many times and can be trained by neural network to obtain more accurate model.
|
|
ICACT20210159 Slide.16
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The hybrid recognition method based on convolution neural network and HMM GMM proposed in this paper can greatly improve the accuracy of recognition in the actual situation, even in the worse environment. In this paper, we do not use the picture to recognize the gesture directly, but recognize the human joint points in the picture. We can calculate the direction of the gesture, and only recognize isolated words in the aspect of speech recognition. It is difficult to recognize many times, and we can use neural network to train it to get a more accurate model.
|
|
ICACT20210159 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
For example, to find all the left upper arm, there is already part information for the left shoulder and elbow, and paf information for the left upper arm. All the part in the left shoulder are matched with all the parts of the left elbow, and the integral of paf information between the two parts is calculated as the score of the connected part. Finally, all scores are sorted to determine whether the part are connected.
|
|
ICACT20210159 Slide.14
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
By Through the first step, we obtain the heatmap of the joint. In this step, the specific position of the joint is extracted from the Heatmap.
|
|
ICACT20210159 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The first branch is used to predict confidence map st, and the second one is used to predict PAFS Lt
|
|
ICACT20210159 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
This way, if there are two different people with the same limb overlap, in the overlap part of the average, can take into account the occlusion limb and the information of the occluded limb. the final obtained groundtruth paf were used as paf training samples.
Using the above training samples to train the neural network, the final output dimension of the network is , including part heatmap and paf.
|
|
ICACT20210159 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The training data set is coco, the data set contains the coordinates and type information of human keypoint. To generate part and paf training samples based on keypoint information.
In this way, if two different people have the same joint close to each other, their maximum value will not be affected (because the non maximum inhibition (NMS) algorithm is used to obtain the peak value as the joint position). In this way, we finally get the groundtruth confidence map as the training sample of the part. It contains 18 part and 1 background.
|
|
ICACT20210159 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Hmm-gmm matching method, this method has high accuracy in identifying isolated words, good model construction and fast processing speed, but its noise processing is not satisfactory, so we use iFLYTEK technology for reference in the next work, and use the method of artificial intelligence to build its model to increase its noise processing effect.
|
|
ICACT20210159 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In gesture recognition, we mainly use the open source code openpose provided by CMU to transform it. Due to the lack of data set, we can only use the model provided by openpose. Because we use the model, it is difficult to transform the model. When we use openpose, its environment is difficult to deal with, and its processing speed is also lacking, so we use the model After lightweight processing, the original video processing efficiency is changed from 1 ~ 3fps to 5 ~ 9fps. At the same time, we also change the original video stream processing to image stream processing to increase its efficiency and accuracy, but its speed is still different from the expected. Therefore, we should further lightweight its model to reduce its processing time
|
|
ICACT20210159 Slide.08
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Using cloud to do fuzzy data analysis, the load pressure at the edge end is alleviated. The edge end evaluates the visual acuity of the detected person in real time, realizes the independent decision of the edge end, and ensures the real-time data processing. The timeliness and reliability of the system are improved by cloud edge collaborative calculation.
|
|
ICACT20210159 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In order to reduce the network delay and bandwidth of video data in the transmission process, improve the speed of vision detection, enhance the real-time of vision detection, and improve the accuracy of vision detection, this project intends to study the framework of gesture recognition vision detection under cloud cooperative computing system.
|
|
ICACT20210159 Slide.06
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
About the system, it is mainly based on the cloud and the edge of the collaborative implementation, at the same time, based on gesture and speech mixed recognition, complete a complete set of autonomous vision detection equipment
|
|
ICACT20210159 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The system includes several modules: edge acquisition device, gesture and speech recognition, cloud edge collaborative data processing, vision icon conversion and voice broadcast.
Combined with visual analysis and machine learning algorithm, by changing the size and direction of the letters displayed on the touch screen, it interacts with the actions of teenagers' vision detection, records the interactive elements through gesture and speech recognition, and sends them to the cloud. After that, Huawei cloud machine learning algorithm is used to explain and analyze the meaning of the actions, so as to complete the task of teenagers' vision detection.
The system includes several modules: edge acquisition device, gesture and speech recognition, cloud edge collaborative data processing, vision icon conversion and voice broadcast.
|
|
ICACT20210159 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
HiLens hardware preset development framework, encapsulate multimedia library, operator library, model management library, provide audio and video management, external interface management, data automatic acquisition and other capabilities; pre-adaptive ROS system, more convenient development of automatic control class applications; compatible with TensorFlow、Caffe and other mainstream deep learning framework; support Python and C++ mainstream languages.
|
|
ICACT20210159 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The data processing process is as follows: first of all, in a school vision test, take the class as the unit, collect data information, automatically upload to the cloud, form the data tracking of personal vision. On this basis, the vision test report of the class and a primary school was formed. Then collect the data to analyze the vision test results of a certain area.
|
|
ICACT20210159 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
About the system, it is mainly based on the cloud and the edge of the collaborative implementation, at the same time, based on gesture and speech mixed recognition, complete a complete set of autonomous vision detection equipment
|
|
ICACT20210159 Slide.01
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Hello everyone, my name is Hu Jingjing. For icact 2020, I have cooperated with several partners of Northeast University in Shenyang, China. Professor Guo Jun and other Northeastern University students. Our speech today is about recommending an autonomous vision detection system based on cloud collaboration, which we call vision detection system based on gesture and speech hybrid recognition.
|
