 |
Machine translation quality estimation (QE) aims to automatically evaluate the quality of machine-generated translations without access to reference human translations. Formally, QE can be represented as a function f(src, mt) that takes as input the source text (src) and machine translation output (mt), and outputs a quality score.
There are two primary categories of QE:
System-level QE ranks collections of multiple (src, mt) pairs to compare systems. This is formalized as Ranking(f(src_list, mt_list)).
Segment-level QE ranks the quality of a single (src, mt) pair, represented as Ranking(f(src, mt)).
For both variants, the primary evaluation metric is ranking correlation versus human quality judgements, measured by metrics like Kendall's Tau. Higher correlation indicates more human-like evaluation ability.
While system-level QE performance is nearing human levels with recent advances, segment-level QE remains an open challenge. Future work should explore new approaches to achieve both high segment-level performance and explainability. |
IEEE/ICACT20240321 Slide.06
[Big Slide]