Now 4431 visitors
Today:0 Yesterday:0
Total: 143407
215S 51P 5R
2026-10-21, Week 43 |
| Member Login |
| TACT Journal Homepage |
| Call for Paper |
| Committee |
| Find My Paper |
| Author Homepage |
| Statistics & History |
| Welcome Message |
| Paper Procedure |
| FAQ |
| Paper Archives |
| Outstanding Papers |
| Author Homepage |
| - Paper Procedure |
| - Journal Procedure |
| - Presentation Tips |
|
| Program & Proceedings |
| Program with Papers |
| Plenary Session |
| Tutorial Session |
|
| Presentation Platform |
| Hotel & Travel Info |
| About Korea |
| Accommodation |
| Transportation |
| VISA |
| Other Infomation |
|
| Photo Gallery |
| Scheduler Login |
| Seminar |
| Archives Login |
|
| Sponsors |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
main_email:yanghao30@huawei.com
| Save the slide and description |
|
*** You can edit any slide by selecting the Slide number below***
|
ICACT20240321 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
LLM QE is comparable for Segment-level
COT can be used in LLM QE, for step-by-step, PPL, Token, Sentence; also Sentence can be overcome by token + ppl
Explainable, KPE are much better than BertScore and TeacherSim
|
|
ICACT20240321 Slide.14
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
A key advantage of our token-level alignment KPE metric is improved explainability versus past approaches through visualizing token similarities. We found both BertScore and TeacherSim alignment heatmaps exhibit concerning patterns indicating potential exploitation.
Specifically, BertScore's alignments overwhelmingly cluster to the maximum similarity score, reducing meaningfulness. And TeacherSim often assigns the highest attention exclusively to the last token, failing to capture broader context.
In contrast, our KPE token alignment scores are much more evenly distributed, with clear variation highlighting both closer and more distant token mappings. This provides more nuanced explanations that better match human translator reasoning.
Overall, while attention-based similarity methods like BertScore and TeacherSim achieve reasonable performance, their explainability is limited by susceptibility to attention optimization rather than holistic understanding. Our experiments demonstrate KPE produces superior faithfulness in explanation thanks to prompting strategy effectiveness and resistance to exploitation.
|
|
ICACT20240321 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
When prompting large language models to assess machine translation output quality in terms of perplexity/fluency, utilizing a 5-point scale (highly fluent to highly disfluent) produces significantly better correlation results compared to coarser 3-point rating (fluent/neutral/disfluent).
Analyzing the Chinese-to-English dataset annotations, we found:
For 3-category rating, 76% of translations are clustered into the neutral quality bucket.
For 5-category rating, only 4% received a neutral score, with most distributed across high, mid and low fluency levels.
The extreme class imbalance in 3-category ranking highlights why it achieves far poorer Kendall's Tau correlation versus human judgments. Much finer-grained distinctions are necessary. Our experiments definitively demonstrate the superiority of 5-category quality annotation for achieving reliable machine translation evaluation using prompted language models.
|
|
ICACT20240321 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Our single-step knowledge-prompted evaluation (KPE) metrics using perplexity, token, and sentence features outperform past pre-neural quality estimation methods, demonstrating the superior evaluation capabilities of large language models.
Critically, the chain-of-thought (CoT) metrics combining step-wise reasoning trace model judgments more effectively. Both CoT1 and CoT2 significantly outperform the single-step KPE metrics across all test languages.
Specifically, CoT1 achieves state-of-the-art segment-level correlation versus human ratings on 5 out of 7 benchmark language pairs. On average, CoT1 also sets a new highest Kendall's Tau correlation level, surpassing past learning-based and feature-engineered quality estimation systems.
In summary, chaining step-wise KPE reasoning to assess translation adequacy, fluency and semantics sets a new performance level for fine-grained machine translation evaluation using prompted language models, while increasing model interpretability. Our work highlights the promise of knowledge prompting strategies to unlock greater understanding of model decisions.
|
|
ICACT20240321 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We utilize the WMT18 news translation dataset as our evaluation benchmark, consisting of 3,000 source sentences translated into 14 language pairs like DE-EN, FI-EN and ZH-EN. The dataset contains translations from all participating WMT18 teams, with DE-EN comprising the most sentence pairs (77,811) and CS-EN the least (5,110).
Importantly, the translated sentences were manually evaluated by professional multilingual human experts through pairwise comparisons. These human quality judgements serve as the gold labels to assess machine translation quality estimation performance.
In total across 14 language directions, the rich diversity of translations from dozens of WMT18 systems paired with meticulous expert quality ratings creates a robust, real-world dataset to thoroughly benchmark our proposed quality estimation methods against past leading approaches. The scale and quality of the data facilitates in-depth analyses into the strengths, weaknesses and differences of various strategies.
|
|
ICACT20240321 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Step 1: Assess the fluency of the machine translation "{mt}" using the following 5-point scale:
5 - Highly fluent
4 - Moderately fluent
3 - Neutral fluency
2 - Somewhat disfluent
1 - Highly disfluent
Fluency rating (1-5):
Step 2: Assess the similarity between the source "{src}" and translation "{mt}" by evaluating token-level alignment. Use the following 5-point scale:
5 - All tokens aligned
4 - Most tokens aligned
3 - Partial token alignment
2 - Few tokens aligned
1 - No token alignment
Token alignment rating (1-5):
Step 3: Evaluate sentence-level semantic similarity between "{src}" and "{mt}" on the following 5-point scale:
5 - Identical meanings
4 - Largely similar meanings
3 - Partial similarity in meaning
2 - Little similarity in meaning
1 - Completely different meanings
Semantic similarity rating (1-5):
Step 4: Combine the step 1-3 ratings to assign an overall quality estimation (1-5):
Overall quality rating (1-5):
|
|
ICACT20240321 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Prompt 1:
On a scale of 1 to 5, with 1 being highly disfluent and 5 being highly fluent, how would you rate the fluency of the following machine translated sentences? Disfluency means awkwardness in phrasing and grammar. Higher fluency means overall smoothness.
Source: [source_sentence]
Machine translation: [translated_sentence]
Fluency rating (1-5):
Prompt 2:
Please assess the fluency of the below machine translation by selecting one of the following categories:
5 - Highly fluent, reads like perfectly smooth natural language
4 - Moderately fluent overall despite some minor awkwardness
3 - Neutral fluency with some clearly disfluent phrases but still interpretable
2 - Somewhat disfluent, multiple phrases have significant awkwardness or grammar issues
1 - Highly disfluent, major grammatical/fluency issues impeding understanding
Source: [source_sentence]
Translation: [translated_sentence]
Fluency category:
Prompt 3:
On a 1 to 5 fluency scale, how would you rate the machine's translation from natural language phrasing into awkward, disfluent wording? 1 means major fluency errors making it unintelligible. 5 means perfectly smooth and natural wording.
Source: [source_sentence]
Machine translation: [translated_sentence]
Fluency score (1-5):
Prompt 4:
Please classify the fluency of the below machine translation into one of 5 levels:
Level 5: Flawless fluency just like a human translation
Level 4: Largely fluent with only minor awkward phrasings
Level 3: Mix of fluent and disfluent phrases, but meaning discernible
Level 2: Significant awkwardness, but main idea interpretable
Level 1: Completely disfluent and incomprehensible
Source: [source_sentence]
Translation: [translated_sentence]
Fluency level (1-5):
Prompt 5:
On a 5 point scale, please rate the machine translation from perfectly natural fluency (5) to completely unnatural and awkward (1):
Source: [source_sentence]
Translation: [translated_sentence]
Fluency rating (1-5):
|
|
ICACT20240321 Slide.08
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We introduce knowledge-prompted evaluation (KPE) metrics for machine translation quality estimation, including both single-step and chain-of-thought (CoT) variants.
Single-step KPE metrics evaluate:
Perplexity: fun_ppl(prompt, src, mt)
Token alignment: fun_token(prompt, src, mt)
Sentence alignment: fun_sent(prompt, src, mt)
The two CoT metrics trace model reasoning over multiple steps:
Cot1: fun_cot1(prompt, fun_ppl, fun_token)
Cot2: fun_cot2(prompt, fun_ppl, fun_sent)
We generate candidate prompts using LLM recommendations then perform human prompt engineering (PE) to select optimal prompts.
Finally, we classify mt quality based on the KPE scores. We compare rating on both 3-class and 5-class scales to evaluate granularity.
In summary, we propose a systematic framework to evaluate combinations of knowledge prompting strategies and CoT explainability for fine-grained machine translation quality estimation using large pretrained language models.
|
|
ICACT20240321 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Recently, prompt-tuning LLMs has shown promise as an alternative to full model fine-tuning for machine translation QE. Formally, LLM-based QE is represented as f(prompt, src, mt), adding a prompt template string to frame the task.
While prompted LLMs achieve state-of-the-art performance for system-level QE, segment-level QE remains an open challenge.
Effective prompt engineering is key to success. We propose having LLMs recommend candidate prompts for the QE task, with humans reviewing and selecting the best prompt.
We further explore chain-of-thought (CoT) step-by-step prompting to trace model reasoning for segment-level scores. CoT explanations also aim to increase model interpretability compared to current approaches.
In summary, while prompt-tuned LLM evaluators set a new state-of-the-art for system-level MT quality estimation, segment-level performance remains to be improved. Advances in knowledge prompting and CoT explainability offer promising directions toward human-competitive fine-grained quality assessment using large language models.
|
|
ICACT20240321 Slide.06
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Machine translation quality estimation (QE) aims to automatically evaluate the quality of machine-generated translations without access to reference human translations. Formally, QE can be represented as a function f(src, mt) that takes as input the source text (src) and machine translation output (mt), and outputs a quality score.
There are two primary categories of QE:
System-level QE ranks collections of multiple (src, mt) pairs to compare systems. This is formalized as Ranking(f(src_list, mt_list)).
Segment-level QE ranks the quality of a single (src, mt) pair, represented as Ranking(f(src, mt)).
For both variants, the primary evaluation metric is ranking correlation versus human quality judgements, measured by metrics like Kendall's Tau. Higher correlation indicates more human-like evaluation ability.
While system-level QE performance is nearing human levels with recent advances, segment-level QE remains an open challenge. Future work should explore new approaches to achieve both high segment-level performance and explainability.
|
|
ICACT20240321 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
key points and questions on using large language models (LLMs) for fine-grained, interpretable quality estimation:
Can chain-of-thought (CoT) prompting make LLMs competitive with state-of-the-art models for segment-level machine translation evaluation?
What is the optimal CoT prompt design tracing model reasoning step-by-step?
Formally, LLM quality estimation (QE) is f(perplexity, token, sentence) scores.
We compare:
3 single-step models using each feature individually
2 CoT models combining perplexity + token and perplexity + token + sentence
How to design effective single-step vs CoT prompts?
We hypothesize CoT explanations can increase model interpretability. However, common approaches like XLM-R and TeacherSim overly optimize attention to the last token, reducing meaningfulness.
Key questions are:
Can CoT prompting improve on single-step performance?
How to design prompts guiding reasoning and explanation?
Can CoT interpretability outperform attention-based methods?
We aim to thoroughly evaluate knowledge prompting strategies for segment-level quality estimation using LLMs.
|
|
ICACT20240321 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Recent work by Tom Kocmi (2023) demonstrated that large language models (LLMs) fine-tuned as evaluators achieve state-of-the-art performance on system-level quality estimation (QE). However, performance on segment-level QE remains an open challenge.
Where fine-tuning requires full model training, prompt tuning offers an easier alternative, already showing promise for some translation tasks (Vilar et al., 2022).
Formally, LLM-based QE can be represented as f(prompt, src_list, mt_list), taking as input a prompt template string, source sentence list, and machine translation output list. The model output is an estimated quality score.
The only difference between LLM QE and traditional QE lies in the additional prompt engineering to frame the quality assessment task for the LLM.
While system-level performance matches fine-tuned models, it remains to be seen if prompt tuning LLMs for segment-level QE can reach similar effectiveness. Exploring knowledge prompting strategies for fine-grained QE offers a promising direction aligning large language model capabilities with quality estimation needs.
|
|
ICACT20240321 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Where traditional deep learning models learn a function f(x) mapping inputs to outputs, LLMs are prompted to exploit their knowledge capabilities, formalized as f(prompt, x). Prompting strategies for LLMs fall into several categories:
Zero-shot prompting - No demonstration data, only a prompt string
One-shot prompting - A single demonstrative input-output example
Few-shot prompting - A small set of examples, like 5-10
Role prompting - Framing the LLM's output role via the prompt
Chain-of-thought prompting - Multi-step prompting that traces model reasoning
Prompting unlocks the knowledge already encoded in LLMs' parameters without further training. Strategies range from providing no data to few-shot demonstration, and prompting the model role. Chain-of-thought prompting traces step-by-step inference. Together these methods allow LLMs to perform diverse reasoning-intensive tasks.
|
|
ICACT20240321 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Recent research has shown increasing promise in adapting large language models (LLMs) for translation and translation quality estimation tasks. Brown et al. (2020) first demonstrated LLMs' effectiveness for multilingual question answering. Vilar et al. (2022) then showed LLMs can reach high translation quality for some language pairs via prompt-tuning without full fine-tuning. Jiao et al. (2023) found that ChatGPT paired with GPT-4 engines approaches human translation quality on high-resource language pairs. Hendy et al. (2023) and Wang et al. (2023) further showed LLMs achieving state-of-the-art results for both sentence and document-level translation tasks. Finally, Tom Kocmi (2023) demonstrated LLMs serve as highly effective system-level evaluator models for quality estimation, outperforming past approaches. Together these works highlight adaptable abilities of LLMs for translation and evaluation, opening promising avenues for continued investigation of knowledge-intensive, interpretable language processing.
|
|
ICACT20240321 Slide.01
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
While methods like GEMBA achieve state-of-the-art performance for system-level quality estimation (QE), segment-level and step-by-step QE remains an open challenge. We investigate whether large language models can be adapted for fine-grained QE. We propose a method with 3 single-step QE models using perplexity, token, and sentence features and 2 chain-of-thought (COT) multi-step models combining perplexity + token and perplexity + token + sentence. Experiments on the WMT18 dataset and a segment-level dataset show single-step QE performance with 5-class classification outperforming 3-class, and COT models achieving competitive results, outperforming single-step approaches. The COT explanations trace model reasoning, providing more interpretability than similarity-based explainability methods like BertScore and TeacherSim. Our work demonstrates promise in adapting large language models for fine-grained and interpretable QE.
|
|
ICACT20240321 Slide.00
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We propose a novel machine translation evaluation approach that incorporates world knowledge into assessment and explanation. By prompting an evaluator model with relevant facts and contexts, more meaningful, interpretable scores can be produced along with natural language justifications grounded in knowledge and reasoning.
|
| |
