opentsdb-hbasecon很骄傲的哗啦的广联达.pdfVIP

Lessons Learned from OpenTSDB Or why OpenTSDB is the way it is and how it changed iteratively to correct some of the mistakes made Benoît “tsuna” Sigoure tsuna@ Key concepts • Data Points (time, value) • Metrics proc.loadavg.1m • Tags host=web42 pool=static • Metric + Tags = Time Series 6 12 • Order of magnitude: >10 time series, >10 data points put proc.loadavg.1m 1234567890 0.42 host=web42 pool=static OpenTSDB @ StumbleUpon • Main production monitoring system for ~2 years • Storing hundreds of billions of data points • Adding over 1 billion data points per day • 13000 data points/s → 130 QPS on HBase • If you had a 5 node cluster, this load would hardly make it sweat Do’s • Wider rows to seek faster before: ~4KB/row, after: ~20KB • Make writes idempotent and independent before: start rows at arbitrary points in time after: align rows on 10m (then 1h) boundaries • Store more data per KeyValue Remember you pay for the key along each value in a row, so large keys are really expensive Don’ts • Use HTable / HTablePool in app servers asynchbase + Netty or Finagle = performance++ • Put variable-length fields in composite keys They’re hard to scan • Exceed a few hundred regions per RegionServer “Oversharding” introduces overhead and makes recovering from failures more expensive See detailed benchmark at Use asynchbase goo.gl/8at5V HTable asynchbase scan sequential read sequential write 50s 500s 200s 38s