Multi-modal and Language Technology Based Post-Editing Support for Machine Translation

Find Out More

General Information

In order to ensure professional human quality level translation results, in many cases, the output of Machine Translation (MT) systems has to be manually post-edited by human experts. The postediting process is carried out within a post-editing (PE) environment, a user-interface, which supports the capture and correction of mistakes, as well as the selection, manipulation, adaptation and recombination of good segments. PE is a complex and challenging task involving considerable cognitive load. To date, PE environments mostly rely on traditional graphical user interfaces (GUIs), involving a computer screen as display and keyboard and mouse as input devices. In this research project we propose the design, development, implementation and extensive road-testing and evaluation of a novel multi-modal post-editing support for machine translation for translation professionals, which extends traditional input techniques of a PE system, such as keyboard and mouse, with novel free-hand and screen gestures, as well as speech and gaze input modalities (and their combinations). The objectives of the research are to increase the usability and the user experience of post-editing Machine Translation and to reduce the overall cognitive load of the translation task, supporting (i) the core post-editing tasks as well as (ii) controlling the PE system and environment. The multimodal PE environments will be integrated with quality prediction (QE) to automatically guide search for useful segments and mistakes, as well as automatic PE via incremental adaptation of MT to PEs to avoid repeat mistakes, in order to achieve the above mentioned objectives. The environments will be road-tested with human translation professionals and trainees and (where possible) within the partner projects in the Paketantrag (Riezler, Frazer, Ney and Waibel) and (where possible) the post-edited data captured will feed into dynamic and incremental MT retraining and update approaches pursued in the partner projects.



Santanu Pal

Santanu Pal

Postdoctoral Researcher

Nico Herbig

Nico Herbig

Doctoral Researcher

Josef van Genabith

Josef van Genabith

Professor for Multilingual Technologies

Antonio Krüger

Antonio Krüger

Professor for Human Computer Interaction


The prototype (including several improvements since this video) is available on GitHub.


Multi-Modal Post-Editing of Machine Translation

Herbig, N. (2022). PhD Thesis 

Investigating the Helpfulness of Word-Level Quality Estimation for Post-Editing Machine Translation Output

Shenoy, R., Herbig, N., Krüger, A., van Genabith, J. (2021). EMNLP 

Mid-Air Hand Gestures for Post-Editing of Machine Translation

Jamara, R., Herbig, N., Krüger, A., van Genabith, J. (2021). ACL 

Multi-Modal Estimation of Cognitive Load in Post-Editing of Machine Translation

Herbig, N., Pal, S., Krüger, A., van Genabith, J. (2021). Language Science Press 

The Transference Architecture for Automatic Post-Editing

Pal, S., Xu, H., Herbig, N., Naskar, S., Krüger, A., van Genabith, J. (2020). COLING 

Improving the Multi-Modal Post-Editing (MMPE) CAT Environment based on Professional Translators' Feedback

Herbig, N., Pal, S., Düwel, T., Shenoy, R., Krüger, A., van Genabith, J. (2020). AMTA 2020 Workshop on Post-Editing in Modern-Day Translation 

MMPE: A Multi-Modal Interface for Post-Editing Machine Translation

Herbig, N., Düwel, T., Pal, S., Meladaki, K., Monshizadeh, M., Krüger, A., van Genabith, J. (2020). ACL 

MMPE: A Multi-Modal Interface using Handwriting, Touch Reordering, and Speech Commands for Post-Editing Machine Translation

Herbig, N., Pal, S., Düwel, T., Meladaki, K., Monshizadeh, M., Hnatovskiy, V., Krüger, A., van Genabith, J. (2020). ACL 

Improving CAT Tools in the Translation Workflow: New Approaches and Evaluation

Vela, M., Pal, S., Zampieri, M., Naskar, S. K., van Genabith, J. (2019). Machine Translation Summit 

USAAR-DFKI – The Transference Architecture for English–German Automatic Post-Editing

Pal, S., Xu, H., Herbig, N., Krüger, A., van Genabith, J. (2019). WMT 

UDS-DFKI Submission to the WMT 2019 Czech-Polish Similar Language Translation Shared Task

Pal, S., Zampieri, M., van Genabith, J. (2019). WMT 

Findings of the 2019 Conference on Machine Translation (WMT19)

Barrault, L., Bojar, O., Costa-jussà, M., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., Monz, C., Müller, M., Pal, S., Post, M., Zampieri, M. (2019). WMT 

Multi-Modal Approaches for Post-Editing Machine Translation

Herbig, N., Pal, S., van Genabith, J., Krüger, A. (2019). CHI 

Multi-modal Indicators for Estimating Perceived Cognitive Load in Post-editing of Machine Translation

Herbig, N., Pal, S., Vela, M., Krüger, A., van Genabith, J. (2019). Machine Translation 

Integrating Artificial and Human Intelligence for Efficient Translation

Herbig, N., Pal, S., van Genabith, J., Krüger, A. (2019). arXiv preprint arXiv:1903.02978. 

A Transformer-Based Multi-Source Automatic Post-Editing System

Pal, S., Herbig, N., Krüger, A., van Genabith, J. (2018). WMT 

Neural Automatic Post-Editing Using Prior Alignment and Reranking

Pal, S., Naskar, S. K., Vela, M., Liu, Q., & van Genabith, J. (2017). EACL

CATaLog Online: A Web-based CAT Tool for Distributed Translation with Data Capture for APE and Translation Process Research

Pal, S., Naskar, S. K., Zampieri, M., Nayak, T., & van Genabith, J. (2016). COLING (Demos)

Multi-Engine and Multi-Alignment Based Automatic Post-Editing and its Impact on Translation Productivity

Pal, S., Naskar, S. K., & van Genabith, J. (2016). COLING

A neural network based approach to automatic post-editing

Pal, S., Naskar, S. K., Vela, M., & van Genabith, J. (2016). ACL

CATaLog Online: Porting a Post-editing Tool to the Web

Pal, S., Zampieri, M., Naskar, S. K., Nayak, T., Vela, M., & van Genabith, J. (2016). LREC

CATaLog: New approaches to tm and post editing interfaces

Nayek, T., Naskar, S. K., Pal, S., Zampieri, M., Vela, M., & van Genabith, J. (2015). RANLP


The Evolving Translator-Computer Interface

In eMpTy Pages Blog by Kirti Vashee

(More) Advanced Human-Computer Interaction for Translators, a Conversation with Samuel Läubli and Nico Herbig

In the Tool Box Journal (Issue 20-12-319) by Jost Zetzsche

Guest in "Post-Editing Sprechstunde"

By Sara Grizzo

New Technologies to Support Post-Editing of Machine Translations – MMPE Research Project Completed

Press Release by DFKI on DFKI Website and Informationsdienst Wissenschaft (idw)

(More) Advanced Human-Computer Interaction for Translators

In ATA Chronicle 50, 1

MMPE: A Multi-Modal Interface for Post-Editing Machine Translation

Presentation at GALA Connected 2021

MMPE: A Multi-Modal Interface for Post-Editing Machine Translation

Presentation at at the International Annual Meeting on Computer-Assisted Translation and Terminology (JIAMCATT) 2021

Multimodales Post-Editing-Interface MMPE: Entdecke die Möglichkeiten

Article in MDÜ Journal 1/2021

MMPE: Multi-Modal Interface Support for Post-Editing

Keynote at TRanslation and Interpreting Technology ONline (TRITON) 2021

New Technologies to Support Post-Editing of Machine Translations

Article in DFKI News 01/2021, page 36

MMPE: A Multi-Modal Interface for Post-Editing Machine Translation

Keynote at Recent Advances in Natural Language Processing (RANLP) 2021

MMPE: A Multi-Modal Interface for Post-Editing Machine Translation

Presentation at Innovation in Translation Summit 2021

Legal Information

Responsible service provider

Responsible for the content of the domain from the point of view of § 5 TMG:

Deutsches Forschungszentrum für Künstliche Intelligenz GmbH, DFKI
(German Research Center for Artificial Intelligence)

Prof. Dr. Antonio Krüger
Helmut Ditzer
Trippstadter Str. 122
67663 Kaiserslautern
Phone: +49 631 20575 0
Fax: +49 631 20575 5030

Register Court: Amtsgericht Kaiserslautern
Register Number: HRB 2313
ID-Number: DE 148 646 973

The person responsible for the editorial content of the domain of the German Research Center for Artificial Intelligence GmbH within the meaning of § 18 para. 2 MStV is:

Frederic Kerber, Researcher
Stuhlsatzenhausweg 3 (Campus D3 2)
D-66123 Saarbruecken
Phone: +49 (0)681 / 857 75-0
Telefax: +49 (0)681 / 857 75-5341

Liability for content

As a service provider, Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI) is responsible under general law for its own content published on this website in accordance with Section 7 para. 1 of the German Telemedia Act (TMG).

DFKI makes every effort to keep the information on our website accurate and current, nevertheless, errors and uncertainties cannot be entirely ruled out. For this reason, DFKI undertakes no liability for ensuring that the provided information is current, accurate or complete, and is not responsible for its quality. DFKI is not liable for material or immaterial damages caused directly or indirectly by the use or non-use of the offered information, or by the use of erroneous and incomplete information, unless willful or grossly negligent fault can be demonstrated. This also applies with respect to software or data provided for download.

DFKI reserves the right to modify, expand or delete parts of the website or the entire website without separate announcement, or to cease publication temporarily or definitively.

Liability for links

Pursuant to Section 7 para. 1 of TMG (German Tele-Media Act), the law limits our responsibility as a service provider to our own content on this website. According to Sections 8 – 10 TMG, we are not obliged to permanently monitor any transmitted or stored third-party information, nor to investigate circumstances that might indicate illegal activity. This does not affect our obligation to remove or block information according to general law. However, we can only assume liability for such data from the point in time at which a concrete legal infringement has been identified. Upon notification of any such legal infringement, we will immediately delete the infringing content.

Cross-references (“links”) to the content providers are to be distinguished from our own content. Our offer includes links to external third-party websites. Providers or operators of linked external pages are always responsible for their respective content. We cannot assume any liability for the content of the linked pages. This third-party content was checked by DFKI when the links were first set up to determine whether any legal infringements existed. At the time of the check, no legal infringements were apparent. However, it cannot be ruled out that the content is subsequently changed by the respective providers. A permanent control of the content of the linked pages is not reasonable without evidence of a legal infringement. Should you believe that the linked external pages infringe applicable law or otherwise contain inappropriate content, please notify us directly at:

In case DFKI should notice or receive any indication that an external offer to which it has linked might cause civil or criminal liability, DFKI will immediately delete this link.


The layout of the homepage, the graphics used and other content on the DFKI website are protected by copyright. The reproduction, processing, distribution and any type of use outside the boundaries of copyright law require the prior written approval of the DFKI. Insofar as any content on this page was not created by DFKI, the copyrights of third parties will be respected. If you believe that you discovered a copyright infringement, please report this to us accordingly. Upon becoming aware of any legal infringements, DFKI will remove or disable access to such infringing content immediately.

Data Protection Notice

The German Research Center for Artificial Intelligence (Deutsches Forschungszentrum für Künstliche Intelligenz -- DFKI) and its staff are committed to goal- and risk-oriented information privacy and the fundamental right to the protection of personal data. In this data protection policy we inform you about the processing of your personal data when visiting and using our web site.


Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI)
Phone: +49 (0)631 / 205 75-0, E-Mail:, Legal information

Data protection officer

Phone: +49 (0)631 / 205 75-0, E-Mail:

Purpose of processing

  • Provision of the information offering regarding the project
  • Establishment of contact and correspondence with visitors and users

Anonymous and protected usage

Visit and usage of our web site are anonymous. At our web site personal data are only collected to the technically necessary extent. The processed data will not be transmitted to any third parties or otherwise disclosed, except on the basis of concrete lawful obligations. Within our information offering we do not embed information or service offerings of third party providers.

While using our web site the data transmission in the internet is being protected by a generally accepted secure encryption procedure and hence cannot easily be eavesdropped or tampered.

Access data

On every access to our web site some usage, transmission and connection data will be collected, temporarily stored in a log file and regularly deleted after 90 days.

On every access/retrieval the following data are stored:

  • IP address
  • transmitted user agent information (in particular type/version of web browser, operating system etc.)
  • transmitted referrer information (URL of the referring page)
  • date and time of the access/retrieval
  • transmitted access method/function
  • transmitted input values (search terms etc.)
  • retrieved page resp. file
  • transmitted amount of data
  • status of processing the access/retrieval

The processing of the access data is lawful because it is necessary for the purposes of the legitimate interests pursued by DFKI. The legitimate interests pursued by DFKI are the adaptation and optimisation of the information offering and the investigation, detection and prosecution of illegal activities in connection with the usage of our web site.

The stored data records can be statistically evaluated in order to adapt and optimize our web site to the needs of our visitors. Any techniques that offer the possibility to retrace the access characteristics of users (tracking) will not be applied. The creation of user profiles and automated decision-making based on it is precluded.

The stored data records are not attributable to specific persons. They are generally not being combined with other data sources. However, the stored data can be analysed and combined with other data sources, if we become aware of concrete indications of any illegal usage.


We use so-called cookies on our web site. Cookies are small files that are being stored by your web browser. The cookies used on our web site do not harm your computer and do not contain any malicious software. They offer a user-friendly and effective usage of our web site. We do not use cookies for marketing purposes.

We transmit so-called session cookies to your web browser. They are valid only for the duration of your visit on our web site and they do not have any meaning outside of our web site. The session cookies are needed in order to identify your session with a unique number during your visit and to transmit our contents in your preferred language. At the end of your visit the session cookies will be automatically deleted upon termination of your web browser.

We also transmit permanent cookies to your web browser with a validity period of at most 365 days. We are exclusively using these cookies in order to respect your settings for the type of presentation (normal, inverted) and for the font size. Furthermore, it will be recorded whether you've taken notice of the information about the usage of cookies in your web browser.

You can adjust your web browser such that you will be informed on setting cookies and allow cookies on an individual basis resp. exclude the acceptance of cookies for specific cases or generally. You also can adjust the automatic deletion of cookies upon termination of your web browser. Upon deactivation of cookies the functionality of our web site can be limited. In any case, our information offering is available to its full extent.

Social Media

We do not embed social media plug-ins on our web site. When you are visiting our web site no data are transmitted to social media services. Profiling by any third parties hence is precluded. You though have the option to change over to our information offerings on Facebook, Twitter and YouTube. For the usage of these services we refer to the data protection policies of the respective service providers. We are processing your personal data within the social networks insofar as you post contributions, send messages or otherwise communicate with us.


You have the option to contact us by e-mail. We will use your e-mail address and other personal contact data for the correspondence with you. Due to lawful obligation every e-mail correspondence will be archived. Subject to our legitimate interests your e-mail address and other personal contact data can be stored in our contact data base. In this case you will receive a corresponding information on the processing of your contact data.

Access and Intervention

Besides the information in this data protection policy you have the right of access to your personal data. To ensure fair data processing, you have the following rights:

  • The right to rectification and completion of your personal data
  • The right to erasure of your personal data
  • The right to restriction of the processing of your personal data
  • The right to object to the processing of your personal data on grounds related to your particular situation

To exercise these rights, please contact our data protection officer.

Right to lodge a complaint

You have the right to lodge a complaint with a supervisory authority if you consider that the processing of your personal data infringes statutory data protection regulations.