Bimodal emotion recognition by man and machine.pdf

Bimodal Emotion Recognition by Man and MachineThomas S. Huang, Lawrence S. Chen Hai TaoBeckman Institute CSL, University of Illinois405 N. Mathews, Urbana IL 61801, U.S.A.fhuang,lchen,taog@ifp.uiuc.eduTsutomu Miyasato Ryohei NakatsuATR Media Integration and Communication Laboratories2-2, Hikari-dai, Seika-cho, Soraku-gun, Kyoto, 619-02 JAPANfmiyasato,nakatsug@mic.atr.co.jpAbstractIn this paper, we report preliminary results of rec-ognizing emotions using both speech and video by ma-chine. We applied automatic feature analysis and ex-traction algorithms to the same data used by De Silva etal. [5] in their human subjective studies. We comparemachine recognition results to human performances inthree tests:(1) video only, (2) audio only, (3) combinedaudio and video. In machine-assisted analyses, wefound these two modalities to be complementary. Emo-tion categories that have similar features in one modal-ity have very dierent features in the other modality.By using both, we show it is possible to achieve higherrecognition rates than either modality alone.1 IntroductionRecognizing human facial expression and emotionby computer is an interesting yet challenging problem.Especially in achieving natural human-computer inter-action (HCI), it is important for the computer to recog-nize emotion and facial expression of the user in orderto respond properly. In her recent book Aective Com-puting, Picard [15] listed several applications where thecomputers would recognize emotions of human users oreven have emotions of their own.There has been a number of studies on machinerecognition of human emotions from facial expressionsor emotional speech. However, most of these studiesuse only a single modality, either purely speech [1, 3, 4,11, 13, 17, 18] or solely video [6, 8, 14, 12, 16, 19, 21], asthe data. Relatively little work has been done in com-bining the two modalities for the machine to recognize human emotions [10, 19]. In this paper, we investigatethe integration of both t