作业信息检索lecture.pptxVIP

Lecture 4: Index Construction · Last lecture : · Wildcards ○ ○ ○ ○ ○ ○ · Spell correction · Soundex $m mace → madden mo □〉 among amortize · This time : on abandon among · Index construction · Dictionary data structures a-hu h y-m n-z · Tolerant retrieval 0 0 0 Plan Index construction · How do we construct an index? · What strategies can we use with limited main memory? Hardware basics · Many design decisions in information retrieval a based on the characteristics of hardware · We begin by reviewing hardware basics Hardware basics · Access to data in memory ismuchfaster than acces to data on disk. · Disk seeks : No data is transferred from disk while disk head is being positioned. · Therefore : Transferring one large chunk of data f disk to memory is faster than transferring many small chunks. · Disk I/O is block-based : Reading and writing of en blocks (as opposed to smaller chunks). · Block sizes : 8KB to 256 KB. Hardware basics · Servers used in IR systems now typically have seve GB of main memory, sometimes tens of GB. · Available disk space is several (2 – 3) orders of magnitude larger. · Fault tolerance is very expensive : It ’ smuch che to use many regular machines rather than one fault tolerant machine. Hardware assumptions for this lectur · symbol statistic value · s average seek time 5 ms = 5 x 10-3 s · b transfer time per byte 0.02 μs = 2 x 10- s · processor ’ sclock rate 109 s-1 · p low-level operation 0.01 μs = 10-8 s (e.g., compare swap a word) · size of main memory several GB · size of disk space 1 TB or RCV1: Our collection for this lectur · Shakespeare ’ s collecte