Enabling Human-Robot Joint Actions


Lab member Chien-Ming Huang gave his talk, “Enabling Human-Robot Joint Actions,” at the Google office in Madison. Drawing on his recently published research on multimodal behaviors (including speech, gaze, and gestures), Chien-Ming highlighted their importance in enabling successful human-robot interaction.

According to his speech, robots are a growing presence in human environments and must coordinate their actions with those of their users. Multimodal behaviors, as shown in Chien-Ming’s study, are a key factor in this join-action behavior. His research implemented a novel approach to modeling human behavior based on mimicking observed humanlike patterns, which is proving to be more useful for successful engagement with humans.

C.-M. Huang and B. Mutlu. Learning-Based Modeling of Multimodal Behaviors for Humanlike Robots. Proceedings of the 2014 ACM/IEEE Conference on Human-Robot Interaction (HRI 2014). March 2014. Bielefeld, Germany.

Source: HCI

Relationships: Robots and Humans. ROILA

ROILA, Robot Interaction Language, is a spoken language for robots. It is constructed to make it easy for humans to learn, but also easy for the robots to understand. ROILA is optimized for the robots’ automatic speech recognition and understanding.

The number of robots in our society is increasing rapidly. The number of service robots that interact with everyday people already outnumbers industrial robots. The easiest way to communicate with these service robots, such asRoomba or Nao, would be natural speech. But current speech recognition technology has not reached a level yet at which it would be easy to use. Often robots misunderstand words or are not able to make sense of them. Some researchers argue that speech recognition will never reach the level of humans.

Palm Inc. faced a similar problem with hand writing recognition for their handheld computers. They invented Graffiti, an artificial alphabet, that was easy to learn and easy for the computer to recognize.  ROILA takes a similar approach by offering an artificial language that is easy to learn for humans and easy to understand for robots. An artificial language as defined by the Oxford Encyclopedia is a language deliberately invented or constructed, especially as a means of communication in computing or information technology.

We reviewed the most successful artificial and natural languages across the dimensions of morphology and phonology (see overview in the form of a large table) and composed a language that is extremely easy to learn. The simple grammar has no irregularities and the words are composed of phonemes that are shared amongst the majority of natural languages. The set of major phonemes was generated from the overview of natural languages. Moreover, we composed a genetic algorithm that generated ROILA’s words in a way that they are easy to pronounce. The same algorithm makes sure that the words in the dictionary sound as different from each other as possible.  This helps the speech recognizer to accurately understand the human speaker.

Most previously developed artificial languages have not been able to attract many human speakers, with the exception of Esperanto. However, with the rise of robots a new community on our planet is formed and there is no reason why robots should not have their own language. Soon there will be millions or robots to which you can talk to in the ROILA language. In summary, we aim to design a “Robotic Interaction Language” that addresses the problems associated with speech interaction using natural languages. Our language is constructed on the basis of two important goals, firstly it should be learnable by the user and secondly, the language should be optimized for efficient recognition by a robot.

ROILA is free to use for everybody and we offer all the technical tools and manuals to make your robot understand and speak ROILA. At the same time we offer courses for humans to learn the ROILA language.

Automated Installer and Java Library March 17, 2013. Development versions of the Automated Installer and Java Library are currently available on GitHub. Please report any issues to Josh at 17019428@student.uws.edu.au. The automated installer and java library are designed to help make it easier to work with ROILA. They are still a work in progress, so there are some features that won’t work fully (especially in the library.) GitHub will be updated with improved copies in the coming months. It will be changed shortly to update a bug with the downloading of the pre-compiled library. There are more planned features for this to come, so keep an eye out on GitHub.

We are currently developing courses in ROILA. They are available in our ROILA Academy. We are also giving a ROILA introductory course to Dutch High School students at the Huygens College Eindhoven. The short course will consist of three lessons followed by a ROILA final exam. The ROILA course will be part of their Science curriculum. The homework curriculum for this course is uploaded in theROILA academy and also on an external website. The vocabulary for this course is uploaded here. You can also find a similar dictionary in the ROILA academy. We will post videos and power point PDFs of each lesson given at the school. We have removed parts of the video that were only relevant to the students (such as administration of the course, etc).

Lesson 1

Lesson 1 Powerpoint PDF

Homework Lessons requirement: Topic 1, 2, 3, 4

Lesson 1 – November 15, 2010

Lesson 1 – November 19, 2010

Lesson 2

Lesson 2 Powerpoint PDF

Homework Lessons requirement: Topic 5, 6, 7

Lesson 2 – November 22, 2010



We have published several articles about ROILA: