StoreEverythingOnlineInADatabase.pptVIP

/~gray/talks/Science_Data_Centers.ppt Store EverythingOnlineIn A Database Jim Gray Microsoft Research Gray@M /~gray/talks Outline Store Everything Online (Disk not Tape) In a Database How Much is Everything? Soon everything can be  recorded and indexed Most bytes will never be seen by humans. Data summarization, trend detection anomaly detection are key technologies See Mike Lesk: How much information is there: /mlesk/ksg97/ksg.html See Lyman Varian: How much information /research/projects/how-much-info/ Storage capacity beating Moore’s law 3 k$/TB today (raw disk) 1k$/TB by end of 2002 Outline Store Everything Online (Disk not Tape) In a Database Online Data Can build 1PB of NAS disk for 5M$ today Can SCAN (read or write) entire PB in 3 hours. Operate it as a data pump: continuous sequential scan Can deliver 1PB for 1M$ over Internet Access charge is 300$/Mbps bulk rate Need to Geoplex data (store it in two places). Need to filter/process data near the source, To minimize network costs. The “Absurd” Disk 2.5 hr scan time (poor sequential access) 1 access per second / 5 GB (VERY cold data) It’s a tape! Disk vs Tape Disk 80 GB 35 MBps 5 ms seek time 3 ms rotate latency 3$/GB for drive 2$/GB for ctlrs/cabinet 15 TB/rack 1 hour scan Tape 40 GB 10 MBps 10 sec pick time 30-120 second seek time 2$/GB for media8$/GB for drive+library 10 TB/rack 1 week scan Building a Petabyte Disk Store Cadillac ~ 500k$/TB = 500M$/PB plus FC switches plus… 800M$/PB TPC-C SANs (Brand PC 18GB/…) 60 M$/PB Brand PC local SCSI 20M$/PB Do it yourself ATA 5M$/PB Cheap Storage and/or Balanced System Low cost storage (2 x 3k$ servers) 5K$ TB2x ( 800 Mhz, 256Mb + 8x80GB disks + 100MbE)raid5 costs 6K$/TB Balanced server (5k$/.64 TB) 2x800Mhz (2k$) 512 MB 8 x 80 GB drives (2K$) Gbps Ethernet + switch (300$/port) 9k$/TB 18K$/mirrored TB Next step in the Evolution Disks become supercomputers Controller will have 1bip