MapReduceintheCloudsforScience.docxVIP

MapReduce in the Clouds for Science CloudCom 2010 Nov 30 – Dec 3, 2010 Thilina Gunarathne, Tak-Lon Wu, Judy Qiu, Geoffrey Fox {tgunarat, taklwu, xqiu,gcf}@ Introduction Cloud computing combined with cloud infrastructure services A very viable alternative for scientists MapReduce Excellent fault tolerance features Scalability Ease of use. Several options for using MapReduce in cloud environments MapReduce as a service Setting up MapReduce cluster on cloud instances Specialized cloud MapReduce runtimes Take advantage of cloud infrastructure services. Introduction Analyze the performance and viability of performing 2 types of bioinformatics computations using MapReduce in cloud environments Sequence alignment Sequence assembly AzureMapReduce Leverages the high latency, eventually consistent, yet highly scalable Azure infrastructure services to provide an decentralized, on demand MapReduce framework Sustained performance of clouds Platforms Apache Hadoop On BareMetal On EC2 Amazon Web Services Elastic MapReduce Microsoft Azure AzureMapReduce Challenges for MapReduce in the clouds Data storage Reliability Master node Metadata storage Performance consistency Communication consistency and scalability Sustained performance Choosing suitable instance types Logging AzureMapReduce A solution to the void of parallel programming frameworks on Microsoft Azure Built on using Azure cloud services Distributed, highly scalable highly available services, backed by industrial strength data centers and technologies Minimal management / maintenance overhead Reduced footprint Decentralized control Ability to dynamically scale up/down AzureMapReduce Features Familiar programming model Fault Tolerance Co-exist with eventual consistency of cloud services Easy testing and deployment Combiner step Web based monitoring console AzureMapReduce Architecture AzureMapReduce Architecture AzureMapReduce Architecture AzureMapReduce Architecture AzureMapReduce Architecture AzureMapReduce Architec