LectureDBBenchmarking.pptVIP

Lecture 12: DB Benchmarking Why benchmarking? What does it measure? Why Benchmarks? The three most important aspects of a DBMS? functionality, price, and performance Performance is hard to figure out what to implement? How to implement? Performance is hard to compare response time? throughput? cost? ease of use? maintenance? Before the Wisconsin Benchmarks Vendors quote performance numbers for marketing, but None was published or verified Performance figures generally not comparable Big customers could afford benchmarking competition use real target applications but difficult and confusing without a standard procedure Vendors only as serious as necessary to make the sale “Vendors had little incentive to publish their performance because it was often embarrassing.” [TBH91] contribute little to move the state of the art forward When customers do their own “A customer will typically involve one or more db vendors and a hardware vendor in this process. These organizations will not encourage the customer to conduct more thorough and detailed tests because such tests take longer and are more likely to uncover problems that might kill the sale. The customer will be encouraged to hurry the testing process and make the selection.” [TBH91] “A [customer-defined] benchmark will present many opportunities for debate [over interpretation]. Both managers and technicians will be involved in rultings that require fundamental tradeoffs between realism, fairness, and expense. … A complex benchmark will leave managers with the feeling that Solomon had it easy.” [TBH91] When DB vendors do their own “They like to set up and perform preliminary testing in private, bring the customer in to witness the test, and then get the customer out quickly before anything can go wrong.” [TBH91] The Wisconsin Benchmarks The Wisconsin benchmarks changed all that around 1981 – 1983 The benchmark: a synthesized data set: the WISC database control various parameters: selectivity, # of