 ICACT20230152 Slide.00
[Big Slide]
|
Chrome  Click!! |
 |
AIShell NER + CommonVoice[zh]
Dataset: AI-SHELL 2, 1k-hour,
MTL FT + Weak supervision Experiment
Normal FT
NER FT
MTL FT
MTL FT + WS
|
| ICACT20230152 Slide.01
[Big Slide]
|
Chrome Click!! |
 |
ASR & NER problems
Pipeline-ASR-NER, noise amplification, ASR quality affects NER quality
End-2-end ASR-NER, data is sparse, and the quality is not good enough. The pre-trained language model can help improve the anti-noise capability of the NER, but the ASR and NER quality is still low.
Whisper + ASR + NER
Whisper , Robust Speech Recognition
Whisper + MTL
Whisper + MTL + Weak Supervised Data Augment
Whisper Experiments
Whisper MTL Finetuning with Weak Supvervision
Whisper ASR Finetuning
Whisper NER Finetuning
Whisper Category NER analysis
Fine-tuning epoch selection |
| ICACT20230152 Slide.02
[Big Slide]
|
Chrome Click!! |
 |
Pipeline-NER-ASR :
the ASR error is amplified, and the NER quality deteriorates
EA-ASR :
End-to-end models, scarce data, and low accuracy
EA-ASR + PT-LM :
The language model is used to enhance the NER, but the accuracy of the ASR/NER is low.
|
| ICACT20230152 Slide.03
[Big Slide]
|
Chrome Click!! |
 |
whisper
Multitask Training
Sequence-to-sequence learning
Multitask Training Format
|
| ICACT20230152 Slide.05
[Big Slide]
|
Chrome Click!! |
 |
Whisper + ASR + NER
Whisper , Robust Speech Recognition
Whisper + MTL
Whisper + MTL + Weak Supervised Data Augment
|
| ICACT20230152 Slide.06
[Big Slide]
|
Chrome Click!! |
 |
Whisper + MTL finetuning
Normal Finetuning
Prompt Finetuning
MTL Finetuning
Weak Supervision Finetuning
|
| ICACT20230152 Slide.07
[Big Slide]
|
Chrome Click!! |
 |
Normal FT
Wav – Text
Fy = Seq2Seq(Y_n | mel, y_1, …, y_n-1)
Prompt FT
Wav – Prompt - Text
Fy = Seq2Seq(Y_n | mel, y_1, …, y_n-1)
MTL FT
FT1 + FT2
MTL FT + Weak Supervision
AiSHELL NER + CommonVoice
|
| ICACT20230152 Slide.08
[Big Slide]
|
Chrome Click!! |
 |
Whisper Experiments
Whisper MTL Finetuning with Weak Supvervision
Whisper ASR Finetuning
Whisper NER Finetuning
Whisper Category NER analysis
Fine-tuning epoch selection |
| ICACT20230152 Slide.10
[Big Slide]
|
Chrome Click!! |
 |
Evaluation Metrics
CER for transcript
F1 for NER
|
| ICACT20230152 Slide.11
[Big Slide]
|
Chrome Click!! |
 |
CER &NER F1
Pipeline-ASR-NER
EA-ASR
EA-ASR + PL-LM
Whisper_ASR_NER is better
Whisper_ASR_NER+WS is much better
|
| ICACT20230152 Slide.12
[Big Slide]
|
Chrome Click!! |
 |
Categorized NER
PER and ORG is SOTA
Avg is better
PER improved most ( 56.78% => 78.83%)
|
| ICACT20230152 Slide.13
[Big Slide]
|
Chrome Click!! |
 |
Categorized NER
PER and ORG is SOTA
Avg is better
PER improved most ( 56.78% => 78.83%)
|
| ICACT20230152 Slide.14
[Big Slide]
|
Chrome Click!! |
 |
Fine-tuning Epoch selection
norm NLP fine-tuning task epoch ~ 3
Wav fine-tuning task with epoch ~7
|
| ICACT20230152 Slide.15
[Big Slide]
|
Chrome Click!! |
 |
Fine-tuning Epoch selection
norm NLP fine-tuning task epoch ~ 3
Wav fine-tuning task with epoch ~7
|
| ICACT20230152 Slide.16
[Big Slide]
|
Chrome Click!! |
 |
ASR and Speech NER can be leveraged with Whisper fine-tuning
Multi-task prompt based fine-tuning with Whisper help every task
Weak supervised data can promote model performance
Future works will major in fine-tuning with more languages and more speech tasks
|
