External Sorting - UH Department of Computer Science外部排序-嗯学院计算机科学系.pptVIP

External Sorting Why Sort? A classic problem in computer science! Data requested in sorted order e.g., find students in increasing gpa order Sorting is first step in bulk loading B+ tree index. Sorting useful for eliminating duplicate copies in a collection of records (Why?) Sort-merge join algorithm involves sorting. Problem: sort 1Gb of data with 1Mb of RAM. why not virtual memory? 2-Way Sort: Requires 3 Buffers Pass 1: Read a page, sort it, write it. only one buffer page is used Pass 2, 3, …, etc.: three buffer pages used. Two-Way External Merge Sort Each pass we read + write each page in file. N pages in the file = the number of passes So toal cost is: Idea: Divide and conquer: sort subfiles and merge General External Merge Sort To sort a file with N pages using B buffer pages: Pass 0: use B buffer pages. Produce sorted runs of B pages each. Pass 2, …, etc.: merge B-1 runs. Cost of External Merge Sort Number of passes: Cost = 2N * (# of passes) E.g., with 5 buffer pages, to sort 108 page file: Pass 0: = 22 sorted runs of 5 pages each (last run is only 3 pages) Pass 1: = 6 sorted runs of 20 pages each (last run is only 8 pages) Pass 2: 2 sorted runs, 80 pages and 28 pages Pass 3: Sorted file of 108 pages Number of Passes of External Sort I/O for External Merge Sort … longer runs often means fewer passes! Actually, do I/O a page at a time In fact, read a block of pages sequentially! Suggests we should make each buffer (input/output) be a block of pages. But this will reduce fan-out during merge passes! In practice, most files still sorted in 2-3 passes. Number of Passes of Optimized Sort Double Buffering To reduce wait time for I/O request to complete, can prefetch into `shadow block’. Potentially, more passes; in practice, most files still sorted in 2-3 passes. Sorting Records! Sorting has become a blood sport! Parallel sorting is the name of the game ... Datamation: Sort 1M records of size 100 bytes Ty