《sensors》.pdfVIP

IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING J-STSP 1 Target detection and tracking with heterogeneous sensors Huiyu Zhou, Murtaza Taj, Andrea Cavallaro Abstract—We present a multimodal detection and tracking algorithm for sensors composed of a camera mounted between two microphones. Target localization is performed on color-based a b c change detection in the video modality and on Time Difference of Arrival TDOA estimation between the two microphones in the audio modality. The TDOA is computed by multi-band General- ized Cross Correlation GCC analysis. The estimated directions of arrival are then post-processed using a Riccati Kalman ?lter. The visual and audio estimates are ?nally integrated, at the likelihood level, into a particle ?lter PF that uses a zero-order d e f g motion model, and a Weighted Probabilistic Data Association WPDA scheme. We demonstrate that the Kalman ?ltering KF Fig. 1. Examples of sensor con?gurations for audiovisual object detection and improves the accuracy of the audio source localization and that tracking ?lled circles indicate microphones; empty circles indicate cameras : the WPDA helps to enhance the tracking performance of sensor a single microphone-camera pair; b-c Stereo Audio and Cycloptic Vision STAC sensors; d-e circular microphone array with single camera; f fusion in reverberant scenarios. The combination of multi-band triangular microphone array with single camera; g linear microphone array GCC, KF and WPDA within the particle ?ltering framework with single camera. improves the performance of the algorithm in noisy scenarios. We also show how the proposed audiovisual tracker summarizes the observed scene by generating metadata that can be transmitted to other network nodes instead of transmitting the raw images that the speakers face the microphone [6]; single or stereo and can be