1. Introduction Linking visual and textual data on video.pdf

1. Introduction The Informedia Digital Video Library Project at Carnegie Mellon University [1] combines speech, image and natural language understanding to automatically transcribe, segment and index video for intelligent search and image retrieval. Since 1995, thousands hours of video (over two terabytes of data) have been collected, with automatically generated metadata and indices for retrieving videos from the library. The distinguishing feature of the project is the integration of speech, language and image understanding technologies for efficient creation and exploration of the library. In video, the visual and textual information occur together. Integration of visual features (such as color, texture, shape, and motion features extracted from the visual data) with the semantics of the content (which comes from the text) is a challenging goal. Although, visual features and text can be insufficient alone, they are complementary for each other [2]. Integrating these two types of data improves the performance for several applications [2,3,4,5,6]. Some methods are proposed to link images with text using the annotated image collections [5,6,7,8]. In this study, similar approaches will be applied to video to link the visual features with text. In Section 2, our previous study that links image regions to words is described. Section 3 discusses the problems in video data that require finding the association of frames and text. The experimental results are presented in Section 4. Section 5 summarizes the results and discusses possible improvements to the model as a means of future directions. 2. Translating images to words There are a wide variety of annotated image collections (e.g. Corel data set, most of the museum collections, the news on the web, etc.). In these collections, although the annotation words are associated with the image, the correspondence between the words and the image regions are unknown. The problem of find