Purdue University

Investigating Embodied Intelligence via Assembly Imitation and Learning to Play Board Games

My research group is engaged in a concerted effort to ground the semanticsof natural language in computer vision and robotic manipulation. We have designed anovel custom robotic platform to support this effort and built three copies of thisplatform to allow investigation of linguistic communication between robotic and humanagents. We do this in the context of two tasks. In the first, one robot builds anassembly out of Lincoln Logs while a second robot observes that activity andcommunicates those observations, in natural language, to a third robot who mustreplicate that assembly. In the second, two robots play a board game, while a thirdrobot---that does not know the game rules---observes the play and must infer thegame rules. These tasks are specifically designed to support investigation intointegrating vision, robotics, natural language, learning, and planning with commonsemantic representations and stochastic inference mechanisms. This allows filling inmissing information from multiple modalities. When the vision system cannot fullydetermine the Lincoln Log assembly structure due to occlusion, it can ask question innatural language or integrate information from multiple views with different cameraposes or taken at different assembly stages. Likewise, when there are insufficienttraining examples to learn game rules, the learner can ask questions that can beanswered either linguistically or by robotic demonstration. I will discuss the commonstochastic inference mechanism built on top of a novel probabilistic programminglanguage augmented with automatic differentiation to support maximum-likelihoodestimation of rich complex models and how this software architecture together withour hardware platform and rich integrated tasks reflect our vision for investigatingembodied intelligence.
Joint work with Andrei Barbu, Seongwoon Ko, Siddharth Narayanaswamy, and Brian Thomas.

Slides <link