Cross-modal Coordination of Expressive Strength between Voice and Gesture for Personified M.pdf

Cross-modal Coordination of Expressive Strength between Voice and Gesture for Personified Media Tomoko Yonezawa ? ATR-IRC Labs. / Nagoya University 2-2-2 Hikaridai, Seika, Soraku, Kyoto 619-0288, Japan yone@atr.jp Noriko Suzuki ? ATR-MIS Labs. 2-2-2 Hikaridai, Seika, Soraku, Kyoto 619-0288, Japan noriko@atr.jp Shinji Abe ? ATR-IRC Labs. 2-2-2 Hikaridai, Seika, Soraku, Kyoto 619-0288, Japan sabe@atr.jp Kenji Mase Nagoya University / ATR-IRC Labs. Furo-cho, Nagoya 464-8601, Japan mase@itc. nagoya-u.ac.jp Kiyoshi Kogure ?? ATR-MIS Labs. 2-2-2 Hikaridai, Seika, Soraku, Kyoto 619-0288, Japan kogure@atr.jp ABSTRACT The aim of this paper is to clarify the relationship between the expressive strengths of gestures and voice for embod- ied and personified interfaces. We conduct perceptual tests using a puppet interface, while controlling singing-voice ex- pressions, to empirically determine the naturalness and strength of various combinations of gesture and voice. The results show that (1) the strength of cross-modal percep- tion is affected more by gestural expression than by the ex- pressions of a singing voice, and (2) the appropriateness of cross-modal perception is affected by expressive combina- tions between singing voice and gestures in personified ex- pressions. As a promising solution, we propose balancing a singing voice and gestural expressions by expanding and correcting the width and shape of the curve of expressive strength in the singing voice. Categories and Subject Descriptors H.5.1 [Information Interfaces and Presentation]: Mul- timedia Information Systems General Terms Perceptual Experimentation and Design with Cross-modality Keywords Cross-modality, Vocal-gestural Expression, Perceptual Ex- periment, Personified Puppet-interface 1. INTRODUCTION Personified media such as humanoids, pet robots and an- droids are being developed with embodied interfaces for nat- ural communication with users. Among the expressions used ?presently also with ATR-MIS ?presently