[HiC2011]A Large-Scale Highly-Efficient Index (梁斌)英文版本.pdf

THUIRDB :A Large-Scale, Highly- Efficient Index, Fast-Access Key-Value Store 10billion records,1bit index per record,1million/sec throughput in 1 machine 梁斌 清华大学计算机科学与技术系 /thuirdb/ mgigabyte@ Structure of Report • Introduction • Requirement and motivation • Background review • Problems • Problems for binary-search • Problems for B+-tree • Problems for hash • Problems for building database • Rational • Separating key and value by location • Sorting then linear write • Data lay-out • Building the index bottom-up • Highly-efficiency compressing Requirement and motivation • A special requirement for key-value store ： （1）Large scale, billions of record （2 ）Random query, hundreds of random query for a task （3 ）Static dataset, once built never changed, just read no update （4 ）Low cost, sometimes should be built in a machine • Pratical application situation ： （1）log-based analysis systems （2 ）language-model based machine translation Backgroud Review • [Tobin 1986] issue an new data structure:T- tree . T-tree inherits from both B-tree and AVL-tree and is used by main-memory databases, such as Datablitz, eXtremeDB, MySQL Cluster. • [Philip 2001]pk-Ttree and pkB-tree add index compression. • [1998 Rao Jun] Rao created CSS-tree, and developed to CSB+ tree and CSS+tree Background Review Google introduced LMS-tree and multi-level skip list based key-value store called LevelDB in 2011. From Alon Efrat’s ppt of skip list