 |
A key advantage of our token-level alignment KPE metric is improved explainability versus past approaches through visualizing token similarities. We found both BertScore and TeacherSim alignment heatmaps exhibit concerning patterns indicating potential exploitation.
Specifically, BertScore's alignments overwhelmingly cluster to the maximum similarity score, reducing meaningfulness. And TeacherSim often assigns the highest attention exclusively to the last token, failing to capture broader context.
In contrast, our KPE token alignment scores are much more evenly distributed, with clear variation highlighting both closer and more distant token mappings. This provides more nuanced explanations that better match human translator reasoning.
Overall, while attention-based similarity methods like BertScore and TeacherSim achieve reasonable performance, their explainability is limited by susceptibility to attention optimization rather than holistic understanding. Our experiments demonstrate KPE produces superior faithfulness in explanation thanks to prompting strategy effectiveness and resistance to exploitation.
|
IEEE/ICACT20240321 Slide.14
[Big Slide]