20100531从稀疏数据处理角度看多租户数据存储陈维良.PPT

* * * * * * * * * * * * * * 从稀疏数据处理角度看 多租户数据存储 陈维良 2010-05-31 主要参考文献 1 Storage and Querying of E-Commerce Data VLDB 2001 2 Extending RDBMSs To Support Sparse Datasets Using An Interpreted Attribute Storage Format VLDB 2006 3 The Case for a Wide-Table Approach to Manage Sparse Relational Data Sets SIGMOD 2007 4 A Comparison of Flexible Schemas for Software as a Service SIGMOD 2009 5 The Design of the F Multitenant Internet Application Development Platform SIGMOD 2009 主要内容 一 稀疏数据处理背景解决方案 二 多租户数据存储与传统稀疏数据处理异同 三 基于稀疏表的多租户数据存储解决方案 四 分桶稀疏表存储方案（自己的想法） 稀疏数据处理背景传统解决方案 1 新生的电子商务应用需要数据模式的持续演变并且有稀疏的特点 2 传统的水平存储模式不能很好的满足这些需求 3 VLDB01 提出使用垂直存储（vertical table），在其上建立逻辑水平视图（logical horizontal view），并利用这些视图进行查询转换 4 VLDB06 提出扩展RDBMS去使用一种解释属性存储（Interpreted attribute storage）方法来处理空值（nulls） 5 SIGMOD07 从模式设计角度出发，认为单表解决方案是较合适的选择（single-table approach is a necessary component），因为单表解决方案使用户从繁重的模式设计工作中解脱。 6 SIGMOD09 提出了F的多租户存储架构 稀疏数据处理背景传统解决方案 一：电子商务数据特点 In trying to store all our electronic parts in one table using this scheme, we ran into the following problems: Large Number of Columns The current database systems do not permit a large numbers of columns in a table. This limit is 1012 columns in DB2 (also in Oracle), whereas we had nearly 5000 attributes across different categories. Sparsity Even if DB2 were to allow the desired number of columns, we would have had nulls in most of the fields. In addition to creating storage overhead1, nulls increase the size of the index and they sort high in the DB2 B+ tree index. Schema Evolution We would need frequent altering of the table to accommodate new parts and categories. The schema evolution is expensive in the current database systems. Performance A query incurs a large performance penalty if the data records are very wide but only a few columns are used in the query. 惩罚，不利 稀疏数据处理背景传统解决方案 二：垂直存储解决方案（Vertical Representation） However, once the data is stored in the vertical format, new problems arise