Embodied dialogue System Research Engineer

Multimodal Dialogue that achieves both Task and Social Goals in Human-Computer dialogue

The objective of this project is to build embodied conversational agents (also known as ECAs, or virtual humans, chatbots, or multimodal dialogue systems) that have the ability to engage their users using language and nonverbal behavior, both social and task talk, where the social talk serves to improve task performance. In order to achieve this objective, we model human-human conversation using deep learning methods, integrate the models into ECAs, and then evaluate their performance.

The research engineer chosen for this project will work in a multi-disciplinary team to develop a state of the art Embodied Conversational Agent system (including using existing code) that can engage in natural conversations with people. It will be demonstrated on a large screen at Inria (and also in other venues), and also used for human-computer interaction experiments.


  1. The engineer chosen for this project should have a broad range of skills including at least several of the following: 
  2. Strong programming skills, particularly in python and java, in order to manage and build upon our existing conversational agents code base. Ideal candidates should have prior experience in building multimodal real-time machine learning pipelines. 
  3. Implementing modules for speech recognition, intention recognition, dialogue management, and natural language generation, adapting some off-the-shelf products, and developing others.
  4. Interfacing the dialogue system modules with competence modules such as intelligent tutors or recommendation systems.
  5. Working with architectures that include nonverbal behavior recognition as well as speech recognition, and with architectures that include an animated agent implemented in Unity for nonverbal behavior generation. 

For more information on the project, potential candidates should look at the SARA (Socially-Aware Robot Assistant) website at <http://articulab.hcii.cs.cmu.edu/projects/sara/> and read some of the publications associated with the project, here <http://articulab.hcii.cs.cmu.edu/publications/>