eliminating the mid dleman strongpstrong eer -to-strongpstrong eer data#223;o w.pdfVIP

Eliminating The Middleman: Peer-to-Peer Dataflow Adam Barker Jon B. Weissman Jano van Hemert National e-Science Centre University of Minnesota, National e-Science Centre University of Edinburgh Minneapolis, MN, USA. University of Edinburgh a.d.barker@ed.ac.uk jon@cs . j.vanhemert@ed.ac.uk ABSTRACT 1. INTRODUCTION Eciently executing large-scale, data-intensive workflows such Eciently executing large-scale, data-intensive workflows as Montage must take into account the volume and pattern common to scientific applications must take into account of communication. When orchestrating data-centric work- the volume and pattern of communication. For example, in flows, centralised servers common to standard workflow sys- Montage [7] an all-sky mosaic computation can require be- tems can become a bottleneck to performance. However, tween 2–8 TB of data movement. Standard workflow tools standards-based workflow systems that rely on centralisa- based on a centralised enactment engine, such as Taverna tion, e.g., Web service based frameworks, have many other [19] and OMII BPEL Designer [18] can easily become a per- benefits such as a wide user base and sustained support. formance bottleneck for such applications, extra copies of This paper presents and evaluates a light-weight hybrid the data (intermediate data) are sent that consume network architecture which maintains the robustness and simplicity bandwidt