Introduction SAND2005-1555P Unlimited Release.pdfVIP

SAND2005-1555P Unlimited Release March 2005 A Quick Look at InfiniBand: November 2004 Douglas Doerfler Sandia National Laboratories Introduction The InfiniBand technology (IB) development community, e.g. Mellanox, Voltaire, Topspin, InfiniCon, etc. are deploying their next generation NICs and switches [1, 2, 3, 4, 5]. The NICs are based on the PCI Express bus and the switches employ a 24 port switch chip developed by Mellanox. This new switch chip has led to the development of high port count (e.g. 144 and 288 ports) standalone switches. Sandia has been acquiring equipment using this technology for its next generation Linux clusters. The purpose of this white paper is to take a quick look at the latest generation InfiniBand technology and contrast it with experience obtained with some other HPC technology. This is an initial “Quick Look” based on the recent acquisition of a small development cluster, named Escher, which contains the latest generation IB components. This study looks at unidirectional and bidirectional characteristics between two nodes. Granted this is not representative of what an application may see, especially in a large job consisting of several nodes and several layers of switching. These characteristics will be examined in future studies. As the title implies, this is a quick look. This study also takes a look at cost of the components in systems of varying size. PCI Express PCI-Express is the latest in a long generation of internal interconnect I/O buses for printed circuit motherboards. It seems that PCI Express is going to be the technology that replaces the PCI-X bus for high-speed intra-node peripheral communications on computational processing boards. PCI and PCI-X will most likely remain to support legacy low performance peripherals, but all of the major HPC interconnect vendors will or will be providing PCI Express based NICs. The IB community is the first to deploy PCI Express based NICs.