Evaluation of English-Welsh machine translation
An article about machine translation was translated into Welsh by Google Translate (https://translate.google.com). In January 2010, professional translators evaluated the translation for fluency and for accuracy.
The results for fluency are in Figure 1. The results for accuracy are in Figure 2. Full details of the evaluation are in mt-evaluation-en-cy.xls.
Figure 1. The fluency of Welsh machine translation of international English
Figure 2. The accuracy of Welsh machine translation of international English
The following translators evaluated the machine translations:
- T1, Tegwen Williams
- T2, Lowri Griffith
- T3, Geraint Lovgreen
- T4, Carolyn Iorwerth
- T5, Aled Jones
The project management was by Cymen (www.cymen.cymru/).
Discussion of the results
Comments from the translators show that the primary problems with the machine translation are as follows:
- The translation does not have a verb.
- The word order is not correct.
Sentence 38 is an example of what not to write. The translation is low quality.
Some of the English text contains terms and examples that we do not want to translate. For example, sentence 41 is, "For example, instead of 'carry out', use 'complete', 'do', or 'perform'." Because of the method that we used, we could not prevent Google Translate from translating 'carry out', 'complete', 'do', or 'perform'.
Sometimes, translators gave different evaluations. For example, for accuracy, sentence 23 has the following evaluations:
- Correct meaning: 1
- Partly correct meaning: 2
- Incorrect meaning: 1
- Nonsense: 2
In other evaluations, a similar problem occurred. To prevent this problem, we gave additional guidelines for the Welsh evaluation.
The additional guidelines were not sufficient to prevent large differences for some evaluations. Unfortunately, the project did not include a review stage, in which translators can discuss their different evaluations. In future projects, we will include a review stage.
For translations of English to Spanish and English to Norwegian, Google Translate gives satisfactory translations. Therefore, the low quality of the Welsh machine translation is for one or both of the following reasons:
- Machine translation is suitable for only a subset of languages.
- Google Translate does not have sufficient data to give satisfactory machine translation from English to Welsh.
An expert at the Language Technologies Unit (www.bangor.ac.uk/canolfanbedwyr/index.php.en), Bangor University, says that Google Translate gives better translations from Welsh to English than it does from English to Welsh. He thinks that some of the reasons for the difference are as follows:
- Google has a very good model of English. Probably, the model for Welsh is simple in comparison to the model for English. Google Translate uses a statistical method to create a model, and most of the bilingual text comes from Welsh Assembly's records of proceedings.
- Welsh has a more complex word structure (morphology) than English (https://en.wikipedia.org/wiki/Colloquial_Welsh_morphology). Therefore, translating from Welsh to English is easier than translating from English to Welsh.
Some comments from the translators
"Although I can generally understand the article, many of the sentences might not be easily understood by people who are not linguists. For example I instantly recognise the mistranslation of 'have/has' and can substitute the correct words, but this would probably be totally misleading for the general reader. The same is true of the lack of verbs, wrong word order etc." T4.
"There is a general problem with the translation of 'machine translation'. It is sometimes translated accurately, at other times translated as 'translation machine'. However, this does not really cloud the meaning in the context." T4.
"I wouldn't publish any phrases in this document apart from the sentences marked with 'Correct meaning'. However, if this tool was used so that a reader can understand the text on a website for example, it would be sufficient." T5.
RSS feed