AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages

Jiayi Wang; David Ifeoluwa Adelani; Sweta Agrawal; Marek Masiak; Ricardo Rei; Eleftheria Briakou; Marine Carpuat; Xuanli He; Sofia Bourhim; Andiswa Bukula; Muhidin Mohamed; Temitayo Olatoye; Tosin Adewumi; Hamam Mokayed; Christine Mwase; Wangui Kimotho; Foutse Yuehgoh; Anuoluwapo Aremu; Jessica Ojo; Shamsuddeen Hassan Muhammad; Salomey Osei; Abdul-Hakeem Omotayo; Chiamaka Chukwuneke; Perez Ogayo; Oumaima Hourrane; Salma El Anigri; Lolwethu Ndolela; Thabiso Mangwana; Shafie Abdi Mohamed; Ayinde Hassan; Oluwabusayo Olufunke Awoyomi; Lama Alkhaled; Sana Al-Azzawi; Naome A. Etori; Millicent Ochieng; Clemencia Siro; Samuel Njoroge; Eric Muchiri; Wangari Kimotho; Lyse Naomi Wamba Momo; Daud Abolade; Simbiat Ajao; Iyanuoluwa Shode; Ricky Macharm; Ruqayya Nasir Iro; Saheed S. Abdullahi; Stephen E. Moore; Bernard Opoku; Zainab Akinjobi; Abeeb Afolabi; Nnaemeka Obiefuna; Onyekachi Raphael Ogbu; Sam Brian; Verrah Akinyi Otiende; Chinedu Emmanuel Mbonu; Sakayo Toadoum Sari; Yao Lu; Pontus Stenetorp

doi:10.48550/arXiv.2311.09828

AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages

Jiayi Wang, David Ifeoluwa Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou, Marine Carpuat, Xuanli He, Sofia Bourhim, Andiswa Bukula, Muhidin Mohamed, Temitayo Olatoye, Tosin Adewumi, Hamam Mokayed, Christine Mwase, Wangui Kimotho, Foutse Yuehgoh, Anuoluwapo Aremu, Jessica Ojo, Shamsuddeen Hassan MuhammadSalomey Osei, Abdul-Hakeem Omotayo, Chiamaka Chukwuneke, Perez Ogayo, Oumaima Hourrane, Salma El Anigri, Lolwethu Ndolela, Thabiso Mangwana, Shafie Abdi Mohamed, Ayinde Hassan, Oluwabusayo Olufunke Awoyomi, Lama Alkhaled, Sana Al-Azzawi, Naome A. Etori, Millicent Ochieng, Clemencia Siro, Samuel Njoroge, Eric Muchiri, Wangari Kimotho, Lyse Naomi Wamba Momo, Daud Abolade, Simbiat Ajao, Iyanuoluwa Shode, Ricky Macharm, Ruqayya Nasir Iro, Saheed S. Abdullahi, Stephen E. Moore, Bernard Opoku, Zainab Akinjobi, Abeeb Afolabi, Nnaemeka Obiefuna, Onyekachi Raphael Ogbu, Sam Brian, Verrah Akinyi Otiende, Chinedu Emmanuel Mbonu, Sakayo Toadoum Sari, Yao Lu, Pontus Stenetorp

Operations & Information Management

Research output: Preprint or Working paper › Preprint › peer-review

Abstract

Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation with human judgments. Learned metrics such as COMET have higher correlation; however, the lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with simplified MQM guidelines for error detection and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AfriCOMET: COMET evaluation metrics for African languages by leveraging DA data from well-resourced languages and an African-centric multilingual encoder (AfroXLM-R) to create the state-of-the-art MT evaluation metrics for African languages with respect to Spearman-rank correlation with human judgments (0.441).

Original language	English
DOIs	https://doi.org/10.48550/arXiv.2311.09828
Publication status	Published - 16 Nov 2023

Bibliographical note

Paper submitted to NAACL 2024. This is an open access publication distributed under the terms of the Creative Commons Attribution License CC BY [https://creativecommons.org/licenses/by/4.0/], which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Access to Document

10.48550/arXiv.2311.09828Licence: CC BY 4.0

2311.09828v3
This is an open access publication distributed under the terms of the Creative Commons Attribution License CC BY [https://creativecommons.org/licenses/by/4.0/], which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Submitted manuscript, 2.79 MBLicence: CC BY 4.0

Cite this

Wang, J., Adelani, D. I., Agrawal, S., Masiak, M., Rei, R., Briakou, E., Carpuat, M., He, X., Bourhim, S., Bukula, A., Mohamed, M., Olatoye, T., Adewumi, T., Mokayed, H., Mwase, C., Kimotho, W., Yuehgoh, F., Aremu, A., Ojo, J., ... Stenetorp, P. (2023). AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages. https://doi.org/10.48550/arXiv.2311.09828

@techreport{49523cd7b96a491f90f4a05fbcb5d182,

title = "AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages",

abstract = " Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation with human judgments. Learned metrics such as COMET have higher correlation; however, the lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with simplified MQM guidelines for error detection and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AfriCOMET: COMET evaluation metrics for African languages by leveraging DA data from well-resourced languages and an African-centric multilingual encoder (AfroXLM-R) to create the state-of-the-art MT evaluation metrics for African languages with respect to Spearman-rank correlation with human judgments (0.441). ",

author = "Jiayi Wang and Adelani, {David Ifeoluwa} and Sweta Agrawal and Marek Masiak and Ricardo Rei and Eleftheria Briakou and Marine Carpuat and Xuanli He and Sofia Bourhim and Andiswa Bukula and Muhidin Mohamed and Temitayo Olatoye and Tosin Adewumi and Hamam Mokayed and Christine Mwase and Wangui Kimotho and Foutse Yuehgoh and Anuoluwapo Aremu and Jessica Ojo and Muhammad, {Shamsuddeen Hassan} and Salomey Osei and Abdul-Hakeem Omotayo and Chiamaka Chukwuneke and Perez Ogayo and Oumaima Hourrane and Anigri, {Salma El} and Lolwethu Ndolela and Thabiso Mangwana and Mohamed, {Shafie Abdi} and Ayinde Hassan and Awoyomi, {Oluwabusayo Olufunke} and Lama Alkhaled and Sana Al-Azzawi and Etori, {Naome A.} and Millicent Ochieng and Clemencia Siro and Samuel Njoroge and Eric Muchiri and Wangari Kimotho and Momo, {Lyse Naomi Wamba} and Daud Abolade and Simbiat Ajao and Iyanuoluwa Shode and Ricky Macharm and Iro, {Ruqayya Nasir} and Abdullahi, {Saheed S.} and Moore, {Stephen E.} and Bernard Opoku and Zainab Akinjobi and Abeeb Afolabi and Nnaemeka Obiefuna and Ogbu, {Onyekachi Raphael} and Sam Brian and Otiende, {Verrah Akinyi} and Mbonu, {Chinedu Emmanuel} and Sari, {Sakayo Toadoum} and Yao Lu and Pontus Stenetorp",

note = "Paper submitted to NAACL 2024. This is an open access publication distributed under the terms of the Creative Commons Attribution License CC BY [https://creativecommons.org/licenses/by/4.0/], which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.",

year = "2023",

month = nov,

day = "16",

doi = "10.48550/arXiv.2311.09828",

language = "English",

type = "WorkingPaper",

}

Wang, J, Adelani, DI, Agrawal, S, Masiak, M, Rei, R, Briakou, E, Carpuat, M, He, X, Bourhim, S, Bukula, A, Mohamed, M, Olatoye, T, Adewumi, T, Mokayed, H, Mwase, C, Kimotho, W, Yuehgoh, F, Aremu, A, Ojo, J, Muhammad, SH, Osei, S, Omotayo, A-H, Chukwuneke, C, Ogayo, P, Hourrane, O, Anigri, SE, Ndolela, L, Mangwana, T, Mohamed, SA, Hassan, A, Awoyomi, OO, Alkhaled, L, Al-Azzawi, S, Etori, NA, Ochieng, M, Siro, C, Njoroge, S, Muchiri, E, Kimotho, W, Momo, LNW, Abolade, D, Ajao, S, Shode, I, Macharm, R, Iro, RN, Abdullahi, SS, Moore, SE, Opoku, B, Akinjobi, Z, Afolabi, A, Obiefuna, N, Ogbu, OR, Brian, S, Otiende, VA, Mbonu, CE, Sari, ST, Lu, Y & Stenetorp, P 2023 'AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages'. https://doi.org/10.48550/arXiv.2311.09828

TY - UNPB

T1 - AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages

AU - Wang, Jiayi

AU - Adelani, David Ifeoluwa

AU - Agrawal, Sweta

AU - Masiak, Marek

AU - Rei, Ricardo

AU - Briakou, Eleftheria

AU - Carpuat, Marine

AU - He, Xuanli

AU - Bourhim, Sofia

AU - Bukula, Andiswa

AU - Mohamed, Muhidin

AU - Olatoye, Temitayo

AU - Adewumi, Tosin

AU - Mokayed, Hamam

AU - Mwase, Christine

AU - Kimotho, Wangui

AU - Yuehgoh, Foutse

AU - Aremu, Anuoluwapo

AU - Ojo, Jessica

AU - Muhammad, Shamsuddeen Hassan

AU - Osei, Salomey

AU - Omotayo, Abdul-Hakeem

AU - Chukwuneke, Chiamaka

AU - Ogayo, Perez

AU - Hourrane, Oumaima

AU - Anigri, Salma El

AU - Ndolela, Lolwethu

AU - Mangwana, Thabiso

AU - Mohamed, Shafie Abdi

AU - Hassan, Ayinde

AU - Awoyomi, Oluwabusayo Olufunke

AU - Alkhaled, Lama

AU - Al-Azzawi, Sana

AU - Etori, Naome A.

AU - Ochieng, Millicent

AU - Siro, Clemencia

AU - Njoroge, Samuel

AU - Muchiri, Eric

AU - Kimotho, Wangari

AU - Momo, Lyse Naomi Wamba

AU - Abolade, Daud

AU - Ajao, Simbiat

AU - Shode, Iyanuoluwa

AU - Macharm, Ricky

AU - Iro, Ruqayya Nasir

AU - Abdullahi, Saheed S.

AU - Moore, Stephen E.

AU - Opoku, Bernard

AU - Akinjobi, Zainab

AU - Afolabi, Abeeb

AU - Obiefuna, Nnaemeka

AU - Ogbu, Onyekachi Raphael

AU - Brian, Sam

AU - Otiende, Verrah Akinyi

AU - Mbonu, Chinedu Emmanuel

AU - Sari, Sakayo Toadoum

AU - Lu, Yao

AU - Stenetorp, Pontus

N1 - Paper submitted to NAACL 2024. This is an open access publication distributed under the terms of the Creative Commons Attribution License CC BY [https://creativecommons.org/licenses/by/4.0/], which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PY - 2023/11/16

Y1 - 2023/11/16

N2 - Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation with human judgments. Learned metrics such as COMET have higher correlation; however, the lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with simplified MQM guidelines for error detection and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AfriCOMET: COMET evaluation metrics for African languages by leveraging DA data from well-resourced languages and an African-centric multilingual encoder (AfroXLM-R) to create the state-of-the-art MT evaluation metrics for African languages with respect to Spearman-rank correlation with human judgments (0.441).

AB - Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation with human judgments. Learned metrics such as COMET have higher correlation; however, the lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with simplified MQM guidelines for error detection and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AfriCOMET: COMET evaluation metrics for African languages by leveraging DA data from well-resourced languages and an African-centric multilingual encoder (AfroXLM-R) to create the state-of-the-art MT evaluation metrics for African languages with respect to Spearman-rank correlation with human judgments (0.441).

UR - https://arxiv.org/abs/2311.09828

U2 - 10.48550/arXiv.2311.09828

DO - 10.48550/arXiv.2311.09828

M3 - Preprint

BT - AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages

ER -

AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages

Abstract

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this