作业感知的hadoop集群网络调度方法研究计算机应用技术专业论文.docxVIP

Job—aware Job—aware Network Scheduling for Hadoop Cluster Abstract Recently with the explosion of data growth,data center has become a COle infrastructure for big data processing．To analyze the data qmcl【ly and efficiently and extract value information,many distributed frameworks are proposed，such as Hadoop and Dryad．The frameworks split big data across the clusters of hundreds or thousands of computers，analyze each piece in parallel，then transfer the output of each piece and merge to the final result．To improve cluster utilization and to keep the duration of data processing jobs low are comnlon goals for data centers． In data center，big data framework like Hadoop transfer a mount of data between different computation stages，which has become a key bottleneck of application performance． Optimizing the scheduling of flows call improve big data job performance．Traditional techniques are mostly flow—based scheduling，without considering the flow correlations．In this paper，we use Hadoop as a concrete example to obtain flow information during Hadoop shuffle from application layer and propose job-aware priority scheduling for data stream based on its feature． First．we observe that rich traffic demand information exists in the intermediate file and log files．This observation motivate US to obtain traffic forecasting from application layer． And such information and CO-dependency carl be extracted through run-time file system monitoring and file analyzing． Then we propose job-aware priority scheduling to optimize shuffle transfer with global view．The key is to allocate identical priority to flows for the same job．With priority policy， flows with high priority give priority to allocation of network resources，which makes flows of a job complete together as soon as possible to avoid job delay for a flow’S long duration． We allocate network resources for jobs from two perspectives，path management and buffer queue management at switch．In Fat-Tree topology，we introduce flow-based