Building a Corpus of Errors and Quality in Machine Translation: Experiments on Error Impact

Ângela Costa; Rui Correia; Luísa Coheur

Building a Corpus of Errors and Quality in Machine Translation: Experiments on Error Impact

Ângela Costa, Rui Correia, Luísa Coheur

Centro de Linguística da UNL (CLUNL)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper we describe a corpus of automatic translations annotated with both error type and quality. The 300 sentences that we have selected were generated by Google Translate, Systran and two in-house Machine Translation systems that use Moses technology. The errors present on the translations were annotated with an error taxonomy that divides errors in five main linguistic categories (Orthography, Lexis, Grammar, Semantics and Discourse), reflecting the language level where the error is located. After the error annotation process, we accessed the translation quality of each sentence using a four point comprehension scale from 1 to 5. Both tasks of error and quality annotation were performed by two different annotators, achieving good levels of inter-annotator agreement. The creation of this corpus allowed us to use it as training data for a translation quality classifier. We concluded on error severity by observing the outputs of two machine learning classifiers: a decision tree and a regression model.

Original language	English
Title of host publication	Proceedings of the Tenth International Conference on Language Resources and Evaluation
Editors	Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, Stelios Piperidis
Place of Publication	Portoroz
Publisher	ELRA
Pages	288-292
Number of pages	5
ISBN (Print)	978-2-9517408-9-1
Publication status	Published - 2016
Event	LREC 2016 - 10th International Conference on Language Resources and Evaluation - Grand Hotel Bernardin Conference Center, Portoroz, Slovenia Duration: 23 May 2016 → 28 May 2016 http://www.lrec-conf.org/proceedings/lrec2016/index.html

Conference

Conference	LREC 2016 - 10th International Conference on Language Resources and Evaluation
Country	Slovenia
City	Portoroz
Period	23/05/16 → 28/05/16
Internet address	http://www.lrec-conf.org/proceedings/lrec2016/index.html

Keywords

Machine Translation
Translation Errors
Translation Quality

Access to Document

Building a Corpus of Errors and Quality in Machine Translation- Experiments on Error ImpactFinal published version, 169 KBLicence: CC BY-NC

http://www.lrec-conf.org/proceedings/lrec2016/pdf/283_Paper.pdfLicence: CC BY-NC

Fingerprint Dive into the research topics of 'Building a Corpus of Errors and Quality in Machine Translation: Experiments on Error Impact'. Together they form a unique fingerprint.

Cite this

Costa, Â., Correia, R., & Coheur, L. (2016). Building a Corpus of Errors and Quality in Machine Translation: Experiments on Error Impact. In N. Calzolari, K. Choukri, T. Declerck, S. Goggi, M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the Tenth International Conference on Language Resources and Evaluation (pp. 288-292). ELRA. http://www.lrec-conf.org/proceedings/lrec2016/pdf/283_Paper.pdf

Costa, Ângela ; Correia, Rui ; Coheur, Luísa. / Building a Corpus of Errors and Quality in Machine Translation : Experiments on Error Impact. Proceedings of the Tenth International Conference on Language Resources and Evaluation. editor / Nicoletta Calzolari ; Khalid Choukri ; Thierry Declerck ; Sara Goggi ; Marko Grobelnik ; Bente Maegaard ; Joseph Mariani ; Hélène Mazo ; Asunción Moreno ; Jan Odijk ; Stelios Piperidis. Portoroz : ELRA, 2016. pp. 288-292

@inproceedings{571dd28c75bd40c8971f3710e502f568,

title = "Building a Corpus of Errors and Quality in Machine Translation: Experiments on Error Impact",

abstract = "In this paper we describe a corpus of automatic translations annotated with both error type and quality. The 300 sentences that we have selected were generated by Google Translate, Systran and two in-house Machine Translation systems that use Moses technology. The errors present on the translations were annotated with an error taxonomy that divides errors in five main linguistic categories (Orthography, Lexis, Grammar, Semantics and Discourse), reflecting the language level where the error is located. After the error annotation process, we accessed the translation quality of each sentence using a four point comprehension scale from 1 to 5. Both tasks of error and quality annotation were performed by two different annotators, achieving good levels of inter-annotator agreement. The creation of this corpus allowed us to use it as training data for a translation quality classifier. We concluded on error severity by observing the outputs of two machine learning classifiers: a decision tree and a regression model. ",

keywords = "Machine Translation, Translation Errors, Translation Quality",

author = "{\^A}ngela Costa and Rui Correia and Lu{\'i}sa Coheur",

note = "info:eu-repo/grantAgreement/FCT/3599-PPCDT/133002/PT# info:eu-repo/grantAgreement/FCT/SFRH/SFRH%2FBD%2F85737%2F2012/PT# PEst-OE/EEI/LA0021/2013 SFRH/BD/85737/2012; LREC 2016 - 10th International Conference on Language Resources and Evaluation ; Conference date: 23-05-2016 Through 28-05-2016",

year = "2016",

language = "English",

isbn = "978-2-9517408-9-1",

pages = "288--292",

editor = "Nicoletta Calzolari and Khalid Choukri and Thierry Declerck and Sara Goggi and Marko Grobelnik and Bente Maegaard and Joseph Mariani and H{\'e}l{\`e}ne Mazo and Asunci{\'o}n Moreno and Jan Odijk and Stelios Piperidis",

booktitle = "Proceedings of the Tenth International Conference on Language Resources and Evaluation",

publisher = "ELRA",

url = "http://www.lrec-conf.org/proceedings/lrec2016/index.html",

}

Costa, Â, Correia, R & Coheur, L 2016, Building a Corpus of Errors and Quality in Machine Translation: Experiments on Error Impact. in N Calzolari, K Choukri, T Declerck, S Goggi, M Grobelnik, B Maegaard, J Mariani, H Mazo, A Moreno, J Odijk & S Piperidis (eds), Proceedings of the Tenth International Conference on Language Resources and Evaluation. ELRA, Portoroz, pp. 288-292, LREC 2016 - 10th International Conference on Language Resources and Evaluation, Portoroz, Slovenia, 23/05/16. <http://www.lrec-conf.org/proceedings/lrec2016/pdf/283_Paper.pdf>

Building a Corpus of Errors and Quality in Machine Translation : Experiments on Error Impact. / Costa, Ângela; Correia, Rui; Coheur, Luísa.

Proceedings of the Tenth International Conference on Language Resources and Evaluation. ed. / Nicoletta Calzolari; Khalid Choukri; Thierry Declerck; Sara Goggi; Marko Grobelnik; Bente Maegaard; Joseph Mariani; Hélène Mazo; Asunción Moreno; Jan Odijk; Stelios Piperidis. Portoroz : ELRA, 2016. p. 288-292.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Building a Corpus of Errors and Quality in Machine Translation

T2 - LREC 2016 - 10th International Conference on Language Resources and Evaluation

AU - Costa, Ângela

AU - Correia, Rui

AU - Coheur, Luísa

N1 - info:eu-repo/grantAgreement/FCT/3599-PPCDT/133002/PT# info:eu-repo/grantAgreement/FCT/SFRH/SFRH%2FBD%2F85737%2F2012/PT# PEst-OE/EEI/LA0021/2013 SFRH/BD/85737/2012

PY - 2016

Y1 - 2016

N2 - In this paper we describe a corpus of automatic translations annotated with both error type and quality. The 300 sentences that we have selected were generated by Google Translate, Systran and two in-house Machine Translation systems that use Moses technology. The errors present on the translations were annotated with an error taxonomy that divides errors in five main linguistic categories (Orthography, Lexis, Grammar, Semantics and Discourse), reflecting the language level where the error is located. After the error annotation process, we accessed the translation quality of each sentence using a four point comprehension scale from 1 to 5. Both tasks of error and quality annotation were performed by two different annotators, achieving good levels of inter-annotator agreement. The creation of this corpus allowed us to use it as training data for a translation quality classifier. We concluded on error severity by observing the outputs of two machine learning classifiers: a decision tree and a regression model.

AB - In this paper we describe a corpus of automatic translations annotated with both error type and quality. The 300 sentences that we have selected were generated by Google Translate, Systran and two in-house Machine Translation systems that use Moses technology. The errors present on the translations were annotated with an error taxonomy that divides errors in five main linguistic categories (Orthography, Lexis, Grammar, Semantics and Discourse), reflecting the language level where the error is located. After the error annotation process, we accessed the translation quality of each sentence using a four point comprehension scale from 1 to 5. Both tasks of error and quality annotation were performed by two different annotators, achieving good levels of inter-annotator agreement. The creation of this corpus allowed us to use it as training data for a translation quality classifier. We concluded on error severity by observing the outputs of two machine learning classifiers: a decision tree and a regression model.

KW - Machine Translation

KW - Translation Errors

KW - Translation Quality

M3 - Conference contribution

SN - 978-2-9517408-9-1

SP - 288

EP - 292

BT - Proceedings of the Tenth International Conference on Language Resources and Evaluation

A2 - Calzolari, Nicoletta

A2 - Choukri, Khalid

A2 - Declerck, Thierry

A2 - Goggi, Sara

A2 - Grobelnik, Marko

A2 - Maegaard, Bente

A2 - Mariani, Joseph

A2 - Mazo, Hélène

A2 - Moreno, Asunción

A2 - Odijk, Jan

A2 - Piperidis, Stelios

PB - ELRA

CY - Portoroz

Y2 - 23 May 2016 through 28 May 2016

ER -

Costa Â, Correia R, Coheur L. Building a Corpus of Errors and Quality in Machine Translation: Experiments on Error Impact. In Calzolari N, Choukri K, Declerck T, Goggi S, Grobelnik M, Maegaard B, Mariani J, Mazo H, Moreno A, Odijk J, Piperidis S, editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation. Portoroz: ELRA. 2016. p. 288-292