CouplingRobotPerceptionand.PDFVIP

DRAFT: DO NOT QUOTE OR CITE 1 Coupling Robot Perception and Online Simulation for Grounding Conversational Semantics Deb Roy, Kai-Yuh Hsiao, and Nikolaos Mavridis Abstract— How can we build robots that engage in ?uid spoken touch can be grounded in a visually-guided motor program conversations with people, moving beyond canned responses and that enables the robot to move towards and touch ob- towards actual understanding? Many dif?cult questions arise jects. This is an example of a procedural association regarding the nature of word meanings, and how those meanings are grounded in the world of the robot. We introduce an that depends on perception to guide action. architecture that provides the basis for grounding word meanings heavy speci?es a property of objects which involves af- in terms of robot perception, action, and memory. The robot’s fordances [9]. A light object affords manipulation perceptual system drives an online simulator that maintains a whereas a suf?ciently heavy one does not. To rep- virtual version of the physical environment in synchronization resent affordances, both procedural and perceptual with the robot’s noisy and changing perceptual input. The simu- lator serves as a “mental model” that enables object permanence representations must be combined. and virtual shifts of perspective. This architecture provides a blue speci?es a visual property, an example of perceptual rich set of data structures and procedures that serve as a basis grounding. set for grounding lexical semantics, a step towards situated, thing must be grounded in terms of both perception and conversational robots. affordances (one can see an object, and expect to Index Terms— Robots, Natural language interfaces, Knowledge reach out and touch it). representation, Active vision, Simulation.