基于HDFS的海量视频数据重分布算法.doc

基于HDFS的海量视频数据重分布算法 郭建华 杨洪斌 陈圣波 （上海大学计算机科学与技术 上海 200444） 摘 要 基于视频数据的分布式计算与基于文本类型数据的分布式计算存在很大的差异。视频数据本身是非结构化的，并且同样大小的视频，内容不同会导致任务执行消耗的时间也不同。对于简单的结构化数据，HDFS其默认的负载均衡器能够解决负载均衡的问题。但是视频文件存在热点访问以及复杂度不一致问题。使用HDFS默认的数据分布机制不能很好地解决计算负载均衡问题。本文提出了一种基于HDFS的海量视频数据重分布算法。首先对视频文件的访问次数以及历史视频分析对视频文件的访问时间进行记录，然后将数据进行量化之后加权作为该视频文件的负载度，最后使用文件置换手段将负载高的视频与低的视频进行置换，直到每个节点的负载达到均衡为止。实验结果表明，使用本文提出的数据重分布算法，可以减少海量视频数据的处理时间。 关键字 HDFS、数据重分布、视频复杂度、视频热度 Weight distribution algorithm for massive video data based on HDFS Guo Jianhua , Yang Hongbin, Chen Shenbo (Department of Computer Science and Technology, Shanghai University, Shanghai 200444, China) Abstract There is a big difference between the distributed computing based on the video data and the distributed computing based on the text type data. The video data is unstructured, and the same size of the video that has different content will lead to different execution time. For simple structured data, the default load equalizer of HDFS can solve the problem of load balancing. But the video file has the problem of different access times and complexity inconsistency. Using the default data distribution mechanism of HDFS is not well to solve the load balancing problem. In this paper, a new algorithm for massive video data redistribution based on HDFS is proposed. Firstly, the access times and the history analysis time of the video file are recorded. Secondly, the data is quantified by weighted as the load of the video file. Lastly, this paper uses the means of file replacement to exchange high load video and low load video, until each node to achieve load balancing. Experimental results show that using the data redistribution algorithm proposed in this paper can reduce the processing time of massive video data. Keywords HDFS, Data redistribution, Video complexity, Video popularity1引言 云计算是一种基于互联网的计算方式,其包括虚拟化、大数据分布式计算、高性能并行计算、大规模数据与网络存储等技术。其主要原理是将一个大的任务划分为若干小任务，然后将这些小任务分配给多台机器进行并行执行，多台机器通过网络通信的方式协作处理，最后将所有的计算结果进行汇总。云计算在近几年得到了飞速的发展，其中Apache基金下的Hadoop在这