 |
While methods like GEMBA achieve state-of-the-art performance for system-level quality estimation (QE), segment-level and step-by-step QE remains an open challenge. We investigate whether large language models can be adapted for fine-grained QE. We propose a method with 3 single-step QE models using perplexity, token, and sentence features and 2 chain-of-thought (COT) multi-step models combining perplexity + token and perplexity + token + sentence. Experiments on the WMT18 dataset and a segment-level dataset show single-step QE performance with 5-class classification outperforming 3-class, and COT models achieving competitive results, outperforming single-step approaches. The COT explanations trace model reasoning, providing more interpretability than similarity-based explainability methods like BertScore and TeacherSim. Our work demonstrates promise in adapting large language models for fine-grained and interpretable QE. |
IEEE/ICACT20240321 Slide.01
[Big Slide]