Parallel and Distributed Systems Computer Science, FSU并行与分布式系统计算机科学FSU.pptVIP

Parallel and Distributed Systems Instructor: Xin Yuan Department of Computer Science Florida State University Parallel and distributed systems What is a parallel computer? A collection of processing elements that communicate and coorperate to solve large problems fast. What is a distributed system? A collection of independent computers that appear to its users as a single coherent system. A parallel computer is implicitly a distributed system. What are they? More than one processing element Elements communicate and cooperate Appear as one system PDS hardware is everywhere. diablo.cs.fsu.edu (2-CPU SMP) Multicore workstations/laptops linprog.cs.fsu.edu (4-node cluster, 2-CPU SMP, 8 cores per node) Hadoop clusters IBM Blue Gene/L at DOE/NNSA/LLNL (262992 processors). Data centers SETI@home. Why parallel and distributed computing? Moore’s law: the number of transistors double every 18 months. Why parallel and distributed computing? How to make good use of the increasing number of transistors on a chip? Increase the computation width (70’s and 80’s) 4bit-8bit-16bit-32bit-64bit-…. Instruction level parallelism (90’s) Pipeline, LIW, etc ILP to the extreme (early 00’s) Out of order execution, 6-way issues, etc Sequential program performance keeps increasing. The clock rate keeps increasing, clock cycles get smaller and smaller. Why parallel and distributed computing? The fundament limit of the speed of a sequential processor. Power wall (high frequency results in heat) Latency wall (speed of light does not change) Power wall: what will happen if we keep pushing ILP? Latency wall Speed of light = 300000000m/s One cycle at 4Ghz = 0.00000000025s The distance that the light can move at one cycle: 0.00000000025 * 300000000 = 7.5cm Intel chip dimension = 1.47 in x 1.47 in = 3.73cm x 3.73cm Why parallel and distributed computing? Power wall and latency wall indicate that the era of single thread performance improvement through Moore’s