《Learning from narrated instruction videos》.pdf

Learning from narrated instruction videos Jean-Baptiste Alayrac Piotr Bojanowski Nishant Agrawal Josef Sivic ENS Inria IIIT Hyderabad Inria Ivan Laptev Simon Lacoste-Julien 5 Inria Inria 1 0 Abstract 2 l u We address the problem of automatically learning the main steps to complete a J certain task, such as changing a car tire, from a set of narrated instruction videos. 2 The contributions of this paper are three-fold. First, we develop a joint model for video and natural language narration that takes advantage of the complementary ] nature of the two signals. Second, we collect an annotated dataset of 57 Internet in- V struction videos containing more than 350,000 frames for two tasks (changing car C tire and CardioPulmonary Resuscitation). Third, we experimentally demonstrate . that the proposed model automatically discovers, in an unsupervised manner, the s main steps to achieve each task and locate them within the input videos. The re- c [ sults further show that the proposed model outperforms single-modality baselines, demonstrating the benefits of joint modeling video and text. 2 v 5 1 Introduction 1 2 Millions of people watch narrated instruction videos1 to learn n