Lecture Notes on “Data Warehousing by examples”.pdfVIP

Lecture Notes on “Data Warehousing by examples” Daniel Lemire and Owen Kaser February 22, 2006 1 Introduction Data Warehousing is an application-driven field. The Data Warehousing techniques can be applied to almost every type of data possible, as long as there is a lot of it. In this lecture, we review four Data Warehousing projects and some of the insights gained by the authors. All these papers are freely available over the Web. Only one of those projects is an industrial project [1], but all are concerned with concrete problems faced by Data Warehousing experts. The topics cover include: updates from legacy data sources, qualifying the quality of a data warehouse, scalability, and performance. We discuss the following projects: • The WareHouse Information Project at Stanford Project (1995) [3]: the project reported on its use of monitors to check for updates in data sources. • DWQ Project (1997): a European research project with some focus on data ware- housing quality [4]. • Data Warehouse of NewsGroup (1999): an research project on the application of Data Warehousing techniques on Internet Newsgroups [2]. • Data Warehouse Population Platform (2003): recent data warehousing project by Telecom Italia with strong performance concerns [1]. 2 WareHouse Information Project at Stanford (WHIPS) The WHIPS project [3] includes monitors responsible for checking the data source for any updates (see Fig. 1). While this may appear, conceptually, as a trivial task, legacy applications often do not fire any warning when their data changes and so, the monitor might have to do a data dump and check for updates by diffing out large files. Assuming the data is in key,value format, they distinguish between 3 types of changes: update, deletion, or insertion. T