High-Dimensional OLAP 数据挖掘课件.ppt

High-Dimensional OLAP:A Minimal Cubing Approach purpose How to cubing in High-Dimensinal data warehouses efficiently This paper propose a novel method that computes a thin layer of the data cube together with associated value-list indices Introduction Data cube has been playing an essential role in the implementation of fast OLAP operation There have been many efficient cube computation algorithms proposed Multiway array aggregation BUC H-cubing Star-cubing Introduction (cont.) Traditional data warehouse may have 10 dimensions , but more that 109 tuples But for bioinformatics, text processing , data are high in dimensionality, over 100,1000 dimensions but only medium in size, eg around 106 tuples. Existing method is too costly in computation time and storage space to high dimensional OLAP Introduction (cont.) new method called shell fragment Vertically partitions a high dimensional dataset into a set of disjoint low dimensional datasets For each fragment , compute it local data cube offline When query, assemble these fragment online Analysis Curse of Dimensionality A high dimensional data cube requires massive memory and disk space Current algorithms are unable to materialize the full cube under such conditions Iceberg Cube Computing only the cuboid cells whose count or other aggregates satisfying the condition: HAVING COUNT(*) = minsup Motivation Only a small portion of cube cells may be “above the water’’ in a sparse cube Only calculate “interesting” data— data above certain threshold Problem of Iceberg Cube First, if a high-dimensional cell has the support already passing the ceberg threshold, it cannot be pruned by the iceberg condition and will still generate a huge number of cells. a base cuboid cell: “(a1; a2; : : : ; a60):5 (i.e., with count 5) will still generate 260 iceberg cube cells. Problem of Iceberg Cube (cont.) Second, it isdifficult to set up an appropriate iceberg threshold. A too low threshold will still generate a huge cube