Multi-modal and Language Technology Based Post-Editing Support for Machine Translation

Find Out More

General Information

In order to ensure professional human quality level translation results, in many cases, the output of Machine Translation (MT) systems has to be manually post-edited by human experts. The postediting process is carried out within a post-editing (PE) environment, a user-interface, which supports the capture and correction of mistakes, as well as the selection, manipulation, adaptation and recombination of good segments. PE is a complex and challenging task involving considerable cognitive load. To date, PE environments mostly rely on traditional graphical user interfaces (GUIs), involving a computer screen as display and keyboard and mouse as input devices. In this research project we propose the design, development, implementation and extensive road-testing and evaluation of a novel multi-modal post-editing support for machine translation for translation professionals, which extends traditional input techniques of a PE system, such as keyboard and mouse, with novel free-hand and screen gestures, as well as speech and gaze input modalities (and their combinations). The objectives of the research are to increase the usability and the user experience of post-editing Machine Translation and to reduce the overall cognitive load of the translation task, supporting (i) the core post-editing tasks as well as (ii) controlling the PE system and environment. The multimodal PE environments will be integrated with quality prediction (QE) to automatically guide search for useful segments and mistakes, as well as automatic PE via incremental adaptation of MT to PEs to avoid repeat mistakes, in order to achieve the above mentioned objectives. The environments will be road-tested with human translation professionals and trainees and (where possible) within the partner projects in the Paketantrag (Riezler, Frazer, Ney and Waibel) and (where possible) the post-edited data captured will feed into dynamic and incremental MT retraining and update approaches pursued in the partner projects.



Santanu Pal

Santanu Pal

Postdoctoral Researcher

Nico Herbig

Nico Herbig

Doctoral Researcher

Josef van Genabith

Josef van Genabith

Professor for Multilingual Technologies

Antonio Krüger

Antonio Krüger

Professor for Human Computer Interaction



MMPE: A Multi-Modal Interface for Post-Editing Machine Translation

Herbig, N., Düwel, T., Pal, S., Meladaki, K., Monshizadeh, M., Krüger, A., van Genabith, J. (2020). ACL 

MMPE: A Multi-Modal Interface using Handwriting, Touch Reordering, and Speech Commands for Post-Editing Machine Translation

Herbig, N., Pal, S., Düwel, T., Meladaki, K., Monshizadeh, M., Hnatovskiy, V., Krüger, A., van Genabith, J. (2020). ACL 

Improving CAT Tools in the Translation Workflow: New Approaches and Evaluation

Vela, M., Pal, S., Zampieri, M., Naskar, S. K., van Genabith, J. (2019). Machine Translation Summit 

The Transference Architecture for Automatic Post-Editing

Pal, S., Xu, H., Herbig, N., Naskar, S. K., Krüger, A., van Genabith, J. (2019). arXiv preprint arXiv:1908.06151. 

USAAR-DFKI – The Transference Architecture for English–German Automatic Post-Editing

Pal, S., Xu, H., Herbig, N., Krüger, A., van Genabith, J. (2019). WMT 

UDS-DFKI Submission to the WMT2019 Similar Language Translation Shared Task

Pal, S., Zampieri, M., van Genabith, J. (2019). WMT 

UDS-DFKI Submission to the WMT2019 Similar Language Translation Shared Task

Barrault, L., Bojar, O., Costa-jussà, M., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., Monz, C., Müller, M., Pal, S., Post, M., Zampieri, M. (2019). WMT 

Multi-Modal Approaches for Post-Editing Machine Translation

Herbig, N., Pal, S., van Genabith, J., Krüger, A. (2019). CHI 

Multi-modal Indicators for Estimating Perceived Cognitive Load in Post-editing of Machine Translation

Herbig, N., Pal, S., Vela, M., Krüger, A., van Genabith, J. (2019). Machine Translation 

Integrating Artificial and Human Intelligence for Efficient Translation

Herbig, N., Pal, S., van Genabith, J., Krüger, A. (2019). arXiv preprint arXiv:1903.02978. 

A Transformer-Based Multi-Source Automatic Post-Editing System

Pal, S., Herbig, N., Krüger, A., van Genabith, J. (2018). WMT 

Neural Automatic Post-Editing Using Prior Alignment and Reranking

Pal, S., Naskar, S. K., Vela, M., Liu, Q., & van Genabith, J. (2017). EACL

CATaLog Online: A Web-based CAT Tool for Distributed Translation with Data Capture for APE and Translation Process Research

Pal, S., Naskar, S. K., Zampieri, M., Nayak, T., & van Genabith, J. (2016). COLING (Demos)

Multi-Engine and Multi-Alignment Based Automatic Post-Editing and its Impact on Translation Productivity

Pal, S., Naskar, S. K., & van Genabith, J. (2016). COLING

A neural network based approach to automatic post-editing

Pal, S., Naskar, S. K., Vela, M., & van Genabith, J. (2016). ACL

CATaLog Online: Porting a Post-editing Tool to the Web

Pal, S., Zampieri, M., Naskar, S. K., Nayak, T., Vela, M., & van Genabith, J. (2016). LREC

CATaLog: New approaches to tm and post editing interfaces

Nayek, T., Naskar, S. K., Pal, S., Zampieri, M., Vela, M., & van Genabith, J. (2015). RANLP

Legal Information

Deutsches Forschungszentrum für Künstliche Intelligenz GmbH, DFKI
(German Research Center for Artificial Intelligence)

Postal Address
Postfach 20 80
D-67608 Kaiserslautern


Kaiserslautern Site
Trippstadter Straße 122
D-67663 Kaiserslautern
Phone: +49 (0)631 / 205 75-0
Telefax: +49 (0)631 / 205 75-5030

Saarbrücken Site
Stuhlsatzenhausweg 3 (Campus D3 2)
D-66123 Saarbruecken
Phone: +49 (0)681 / 857 75-0
Telefax: +49 (0)681 / 857 75-5341

Bremen Site
Robert-Hooke-Straße 1
D-28359 Bremen
Phone: +49 (0)421 / 178 45-0
Telefax: +49 (0)421 / 178 45-4150

DFKI Project Office Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 (0)30 / 238 95-0

Management Board
Prof. Dr. Antonio Krüger

Head of Supervisory Board
Dr.-Ing. Gabriël Clemens

Register Court: Amtsgericht Kaiserslautern Register Number: HRB 2313

ID-Number: DE 148 646 973

Data Protection Notice

The German Research Center for Artificial Intelligence (Deutsches Forschungszentrum für Künstliche Intelligenz -- DFKI) and its staff are committed to goal- and risk-oriented information privacy and the fundamental right to the protection of personal data. In this data protection policy we inform you about the processing of your personal data when visiting and using our web site.


Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI)
Phone: +49 (0)631 / 205 75-0, E-Mail:, Legal information

Data protection officer

Phone: +49 (0)631 / 205 75-0, E-Mail:

Purpose of processing

  • Provision of the information offering regarding the project
  • Establishment of contact and correspondence with visitors and users

Anonymous and protected usage

Visit and usage of our web site are anonymous. At our web site personal data are only collected to the technically necessary extent. The processed data will not be transmitted to any third parties or otherwise disclosed, except on the basis of concrete lawful obligations. Within our information offering we do not embed information or service offerings of third party providers.

While using our web site the data transmission in the internet is being protected by a generally accepted secure encryption procedure and hence cannot easily be eavesdropped or tampered.

Access data

On every access to our web site some usage, transmission and connection data will be collected, temporarily stored in a log file and regularly deleted after 90 days.

On every access/retrieval the following data are stored:

  • IP address
  • transmitted user agent information (in particular type/version of web browser, operating system etc.)
  • transmitted referrer information (URL of the referring page)
  • date and time of the access/retrieval
  • transmitted access method/function
  • transmitted input values (search terms etc.)
  • retrieved page resp. file
  • transmitted amount of data
  • status of processing the access/retrieval

The processing of the access data is lawful because it is necessary for the purposes of the legitimate interests pursued by DFKI. The legitimate interests pursued by DFKI are the adaptation and optimisation of the information offering and the investigation, detection and prosecution of illegal activities in connection with the usage of our web site.

The stored data records can be statistically evaluated in order to adapt and optimize our web site to the needs of our visitors. Any techniques that offer the possibility to retrace the access characteristics of users (tracking) will not be applied. The creation of user profiles and automated decision-making based on it is precluded.

The stored data records are not attributable to specific persons. They are generally not being combined with other data sources. However, the stored data can be analysed and combined with other data sources, if we become aware of concrete indications of any illegal usage.


We use so-called cookies on our web site. Cookies are small files that are being stored by your web browser. The cookies used on our web site do not harm your computer and do not contain any malicious software. They offer a user-friendly and effective usage of our web site. We do not use cookies for marketing purposes.

We transmit so-called session cookies to your web browser. They are valid only for the duration of your visit on our web site and they do not have any meaning outside of our web site. The session cookies are needed in order to identify your session with a unique number during your visit and to transmit our contents in your preferred language. At the end of your visit the session cookies will be automatically deleted upon termination of your web browser.

We also transmit permanent cookies to your web browser with a validity period of at most 365 days. We are exclusively using these cookies in order to respect your settings for the type of presentation (normal, inverted) and for the font size. Furthermore, it will be recorded whether you've taken notice of the information about the usage of cookies in your web browser.

You can adjust your web browser such that you will be informed on setting cookies and allow cookies on an individual basis resp. exclude the acceptance of cookies for specific cases or generally. You also can adjust the automatic deletion of cookies upon termination of your web browser. Upon deactivation of cookies the functionality of our web site can be limited. In any case, our information offering is available to its full extent.

Social Media

We do not embed social media plug-ins on our web site. When you are visiting our web site no data are transmitted to social media services. Profiling by any third parties hence is precluded. You though have the option to change over to our information offerings on Facebook, Twitter and YouTube. For the usage of these services we refer to the data protection policies of the respective service providers. We are processing your personal data within the social networks insofar as you post contributions, send messages or otherwise communicate with us.


You have the option to contact us by e-mail. We will use your e-mail address and other personal contact data for the correspondence with you. Due to lawful obligation every e-mail correspondence will be archived. Subject to our legitimate interests your e-mail address and other personal contact data can be stored in our contact data base. In this case you will receive a corresponding information on the processing of your contact data.

Access and Intervention

Besides the information in this data protection policy you have the right of access to your personal data. To ensure fair data processing, you have the following rights:

  • The right to rectification and completion of your personal data
  • The right to erasure of your personal data
  • The right to restriction of the processing of your personal data
  • The right to object to the processing of your personal data on grounds related to your particular situation

To exercise these rights, please contact our data protection officer.

Right to lodge a complaint

You have the right to lodge a complaint with a supervisory authority if you consider that the processing of your personal data infringes statutory data protection regulations.