Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation

Diptesh Kanojia, Marina Fomicheva, Tharindu Ranasinghe, Frédéric Blain, Constantin Orasan, Lucia Specia

    Research output: Chapter in Book/Published conference outputConference publication


    Current Machine Translation (MT) systems achieve very good results on a growing variety of language pairs and datasets. However, they are known to produce fluent translation outputs that can contain important meaning errors, thus undermining their reliability in practice. Quality Estimation (QE) is the task of automatically assessing the performance of MT systems at test time. Thus, in order to be useful, QE systems should be able to detect such errors. However, this ability is yet to be tested in the current evaluation practices, where QE systems are assessed only in terms of their correlation with human judgements. In this work, we bridge this gap by proposing a general methodology for adversarial testing of QE for MT. First, we show that despite a high correlation with human judgements achieved by the recent SOTA, certain types of meaning errors are still problematic for QE to detect. Second, we show that on average, the ability of a given model to discriminate between meaning-preserving and meaning-altering perturbations is predictive of its overall performance, thus potentially allowing for comparing QE systems without relying on manual quality annotation.

    Original languageEnglish
    Title of host publicationWMT 2021 - 6th Conference on Machine Translation, Proceedings
    PublisherAssociation for Computational Linguistics (ACL)
    Number of pages14
    ISBN (Electronic)9781954085947
    Publication statusPublished - 22 Sept 2021
    Event6th Conference on Machine Translation, WMT 2021 - Virtual, Online, Dominican Republic
    Duration: 10 Nov 202111 Nov 2021

    Publication series

    NameWMT 2021 - 6th Conference on Machine Translation, Proceedings


    Conference6th Conference on Machine Translation, WMT 2021
    Country/TerritoryDominican Republic
    CityVirtual, Online

    Bibliographical note

    Copyright 2021 the authors, Creative Commons Attribution 4.0 International (CC BY 4.0)


    Dive into the research topics of 'Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation'. Together they form a unique fingerprint.

    Cite this