 |
Recent work by Tom Kocmi (2023) demonstrated that large language models (LLMs) fine-tuned as evaluators achieve state-of-the-art performance on system-level quality estimation (QE). However, performance on segment-level QE remains an open challenge.
Where fine-tuning requires full model training, prompt tuning offers an easier alternative, already showing promise for some translation tasks (Vilar et al., 2022).
Formally, LLM-based QE can be represented as f(prompt, src_list, mt_list), taking as input a prompt template string, source sentence list, and machine translation output list. The model output is an estimated quality score.
The only difference between LLM QE and traditional QE lies in the additional prompt engineering to frame the quality assessment task for the LLM.
While system-level performance matches fine-tuned models, it remains to be seen if prompt tuning LLMs for segment-level QE can reach similar effectiveness. Exploring knowledge prompting strategies for fine-grained QE offers a promising direction aligning large language model capabilities with quality estimation needs. |
IEEE/ICACT20240321 Slide.04
[Big Slide]