 |
When prompting large language models to assess machine translation output quality in terms of perplexity/fluency, utilizing a 5-point scale (highly fluent to highly disfluent) produces significantly better correlation results compared to coarser 3-point rating (fluent/neutral/disfluent).
Analyzing the Chinese-to-English dataset annotations, we found:
For 3-category rating, 76% of translations are clustered into the neutral quality bucket.
For 5-category rating, only 4% received a neutral score, with most distributed across high, mid and low fluency levels.
The extreme class imbalance in 3-category ranking highlights why it achieves far poorer Kendall's Tau correlation versus human judgments. Much finer-grained distinctions are necessary. Our experiments definitively demonstrate the superiority of 5-category quality annotation for achieving reliable machine translation evaluation using prompted language models. |
IEEE/ICACT20240321 Slide.13
[Big Slide]